Not too long ago in a distant part of London (Elephant and Castle), I was trying desperately to make the organisation I worked for firewall co-operate with APNS and other Apple services. I was following the advice given in Apple’s own note here: https://support.apple.com/en-gb/HT203609
They basically say for APNS open up your firewall to the entire 17.0.0.0/8 address range because we own it.
The problem was it didn’t work … reliably. Used to work periodically but never figured out at the time quite where it was going wrong. Was it Jamf to Apple? Was it the clients to Apple? I ran out of time and eventually that role ran out of steam. Since then I’ve had adventures in both Saudi Arabia and currently Switzerland but the Apple traffic problem has bugged me since.
Ben Toms has suggested to me “Why don’t you just do *.apple.com instead?” and my response to him and others has been … if the networking kit supported using DNS names then I would but it doesn’t so I sadly can’t. And recently it’s been found that this advice isn’t 100% either.
I encountered this when researching my https://richard-purves.com/2016/09/10/apple-services/ blog post. I was finding lots of .apple.com addresses but they’d occasionally resolve to other places because they were just CNAMES. Bradley Chapman at JNUC 2017 finally helped put the pieces together for me with his talk on APNS. The relevant part is from 17:38 in.
Akamai. They’re using Akamai as a CDN. And they have sixteen different points of locality arranged around the globe. Below is a modified picture from Bradley’s talk to illustrate the problem that we face.
So if you’re in an environment that’s locked down, what can you do? If you’ve followed Apple’s own instructions for APNS you’ve just opened your firewall to 16,777,216 different addresses owned by Apple alone!
Now Neil Martin yesterday gave me this link about macOS Recovery and it’s networking needs. Ports 80 and 443 … ok. Two DNS address’ to whitelist: osrecovery.apple.com and oscdn.apple.com … ok. I’ve got 17.0.0.0/8 opened up on those ports anyway … and nothing.
So I do some digging. Literally. Opened up the terminal and threw some dig commands at it to find out what’s going on. First up, osrecovery.apple.com.
Doesn’t seem so bad. That’s resolving to a 17/8 address, but what about oscdn.apple.com ? This is the one doing the heavy downloading work.
Now that’s a problem. See the section in red? So we do a DNS look up on that address, we find it’s a CNAME that referrs you to an akadns server. That in turn refers you to an Akamai DNS service that is constantly rotating IP addresses before coming back with the final result … and THAT’s not in 17/8!
Software Update has the exact same issue. My own invesigations show that iTunes patches, OS updates and the like all stem from three separate Apple addresses and they are: mesu.apple.com swcdnlocator.apple.com and swscan.apple.com . To save time, the first two are in Apple’s range and swscan.apple.com where your downloads come from, is not.
Akamai is good at hosting. They’re good at load balancing. If you’re trying to allow these services through however this is AWFUL. No security team in the world is going to allow you to effectively whitelist half the internet’s IP range! (exaggeration for effect).
So what can we do about this? There isn’t an easy answer to that but let’s start by identifying where the problems actually are first. From there we can progress.
- Apple. Their infrastructure is so big, client base so wide that they need a massive backend to deliver security fixes and the like. The documentation on what to allow and where is also majorly lacking. Oh and other vendors like Microsoft are heading this direction too.
- Traffic. The emphasis today is security. APNS traffic in particular implements a lot of security and a lot of corporate practices do NOT enhance that security, in fact they make it worse. Every time I hear the term “SSL Inspection” I really do want to run away and scream.
- Outmoded security ideas: We’ve had the perimeter security model for years. It’s time for a change.
Perimeter security model? You’ve probably been doing it without realising. Here’s a picture I “borrowed” from the internet to help explain.
To use an analogy: The firewall is the Castle walls and moat. The VPN is the guard with the sharp sword allowing entry or not to recognised people. Inside, people can wander around in the areas they’re supposed to with the permission of the local King or Lord.
This worked when the internet was small but to my mind, it’s fast becoming outdated in the same way that castles in Europe became outdated when the weapons to knock down their walls got better. All it takes is a better cannon …
So what’s the alternative? Reorganise the entire structure, both in layout and in trust. Remove the concept of an internal “trusted” network or even network segments and move to an identity verified system with multi-factor authentication. Like this (with apologies for “borrowing” from Wikipedia):
What this means is your offices become “dumb pipes” to the internet and you cloud everything else. No access unless your MFA works. No more perimeters, no more firewalls (at least in the office, probably want some in the cloud).
Yes i’m advocating for cloud computing in it’s loosest sense and we’ve been heading that way for some time. As long as companies like Apple keep implementing management controls that rely on unrestricted internet access (because the security is implemented in the protocols rather than in expensive hardware), then perimeter based networks are going to have bigger and bigger issues consuming and deploying services.
Think of Apple as leading the charge on this one. I hate to use the phrase but it’s rapidly becoming an “internet of things” and who knows where things will go. I do know this for certain: organisations that have or are going this path are actually having a smoother ride right now that organisations that are not. At some point that will start costing more money than it would have, and managers out there … you should be thinking about this now. I’m already hitting this wall.