Close Encounters of the Proxy Kind

Apple provides two ways to cope with the ever present spectre of dealing with corporate proxy servers and access. The first is a configuration profile as delivered by a MDM solution or even manually via a non MDM solution.

We wont be examining this.

Its a very blunt tool: all or nothing and it only really works for things that work through Apple NSURL and other official APIs in macOS. Which turns out to be a pain if youre dealing with apps and other tools that don’t use those API’s and if you have a corporate proxy that requires authentication and is running in an explicit manner. (If it was transparent, we wouldn’t be in this mess at all!)

That leaves option 2: scripting! This turns out to be more complicated than it should be but thankfully that’s what this post is about.

So let’s assume you have to write a script to do this. Let’s also assume you only need to do this on your Ethernet connection because corporate policy. Finally we’ll assume that you only want to proxy the HTTP and HTTPS protocols, no SOCKS or PAC files or anything else that complicated. Just an address and a port number.

The reason for the last requirement is simple: APNS via the proxy. If you use any other setting, I’m assured (and my own experiments confirm) that APNS traffic will attempt to go direct which will fail horribly. However just using the HTTP and HTTPS settings causes all APNS traffic from the mac to go straight to your proxy. Interesting.

Apple, don’t you dare patch that. Especially if you want to be “enterprise friendly”.

Now let us begin.

The GUI Config

This is deceptively simple. Apple supplies a command called networksetup that does the work. It’s format is also deceptively simple and you would (on first glance) run these commands as root:

networksetup -setwebproxy Ethernet proxy.address portnumber
networksetup -setsecurewebproxy Ethernet proxy.address portnumber

The issue here is a big one: changing Ethernet names. If you just had an iMac or Mac Mini with a built in ethernet port then you could simply use the commands above. What if you have a laptop? Now you have USB Ethernet or Thunderbolt Ethernet . What if you have a Mac Pro with dual ethernet ports? Now it’s Ethernet 1 and Ethernet 2 . Thunderbolt Display? Now Thunderbolt Ethernet .

And the list goes on and on. The USB-C adaptors are apparently different again. So to make this a little more universal, we need to think about how to detect the various Ethernet adaptors potentially installed. That will require a little more scripting but networksetup actually comes to our rescue again.

networksetup -listallnetworkservices | grep Ethernet

That will nicely grab a list of all Ethernet services present on the device. However since bash doesn’t like spaces in the names, you’ll get unwanted line splitting. The best technique I know for dealing with this is to backup and reset the Internal Field Separator variable. Now you end up with code that looks like this:

OIFS=$IFS
IFS=$'\n'
   ethport=($( networksetup -listallnetworkservices | grep Ethernet ))
IFS=$OIFS

We back up the original IFS setting to the OIFS variable. Then we set the IFS to only split on a newline character (the \n ). Now we use the ethport variable to hold the output of the networksetup command we used earlier. You may have noticed the command is encapsulated in a $( ) string: that’s so we can capture the output of the command. That is encapsulated in another set of ( ) brackets as we want that output to be saved as an array.

Lastly we set IFS back to it’s default by reading it from it’s backup.

Now we need to process the resulting output. For that, we’re going to use a bash loop.

address="proxy.address"
port="portnumber"

for (( i=0; i<${#ethport[@]}; i++ ));
do
   networksetup -setwebproxy "${ethport[$i]}" "$address" "$port"
   networksetup -setsecurewebproxy "${ethport[$i]}" "$address" "$port"
done

I’m putting the address and port we need into a pair of variables for easy changing later. The command doesn’t need the protocol in front thankfully, so we can reuse the same address in this particular case. Your environment may differ.

Start using the i variable as a counter, starting at 0. We set the maximum loop size to just under the total number of entries in the ethport array and finally tell the for loop to increase i every loop with the double +.

The networksetup commands are present, and they read out the correct Ethernet service name per loop and apply the config in the variables.

If you only have one ethernet, it’ll run once. If you have many ethernet then it’ll run multiple times.

Put that all together and you end up with this:

OIFS=$IFS
IFS=$'\n'
ethport=($( networksetup -listallnetworkservices | grep Ethernet ))
IFS=$OIFS

httpaddress="http://proxy.address"
httpsaddress="https://proxy.address"
port="proxyport"

for (( i=0; i<${#ethport[@]}; i++ ));
do
   echo "Configuring ${ethport[$i]} to use $address:$port" networksetup -setwebproxy "${ethport[$i]}" "$httpaddress" "$port" networksetup -setsecurewebproxy "${ethport[$i]}" "$httpsaddress" "$port"
done

Zen and the Art of Command Line

This is all well and good but there are plenty of command line tools that don’t use Apple’s API’s for internet traffic. Biggest example being the unix curl command. However a lot of unix tools (but not all!) do respect a couple of environment variables. These are:

http_proxy
https_proxy

The issue here is that in typical unix fashion, this isn’t uniformly respected either. Sometimes command line apps will only respect the following:

HTTP_PROXY
HTTPS_PROXY

So that means we have to set all four to be on the safe side. Now assuming the same variables as in the previous example we can do this:

export http_proxy=http://$address:$port/
export HTTP_PROXY=http://$address:$port/
export https_proxy=https://$address:$port/
export HTTPS_PROXY=https://$address:$port/

Look simple so far? What we need is a way to load these automatically when a user loads a terminal session and just for good measure, ANY user not just a specific one. Thankfully the BSD underpinnings of macOS make this easy because we have access to /etc/profiles .

/etc/profiles is used to set system wide environment variables at the time a user logs into terminal. So we can append a few lines to the bottom of that file with the management tool and script of our choice and a standard user will be proxied as best we can. So we modify the code above to look like this:

echo "export http_proxy=http://$address:$port/" >> /etc/profile
echo "export HTTP_PROXY=http://$address:$port/" >> /etc/profile
echo "export https_proxy=https://$address:$port/" >> /etc/profile
echo "export HTTPS_PROXY=https://$address:$port/" >> /etc/profile

Now every time you run this, it will append those lines to the bottom of the /etc/profile file. That’s a little problematic as that file will balloon over time as you keep running it. Ideally we would like some detection code to see if the file has been modified or not. Add the lines if not, modify them if they do. Thankfully this can be done with some test commands and a little sed.

proxysettest=$( cat /etc/profile | grep http_proxy | wc -l | awk '{ print $1 }' )
proxyconfigtest=$( cat /etc/profile | grep http_proxy=http://$address:$port/ | wc -l | awk '{ print $1 }' )

if [ "$proxysettest" = "0" ];
then
   echo "export http_proxy=http://$address:$port/" >> /etc/profile
   echo "export HTTP_PROXY=http://$address:$port/" >> /etc/profile
   echo "export https_proxy=https://$address:$port/" >> /etc/profile
   echo "export HTTPS_PROXY=https://$address:$port/" >> /etc/profile
fi

if [ "$proxyconfigtest" = "0" ];
then
   sed -i -e 's,http://.*,http://'"$address"':'"$port/"',g' /etc/profile
   sed -i -e 's,https://.*,https://'"$address"':'"$port/"',g' /etc/profile
fi

We test to see if there are any lines inside the file starting http_proxy and make a count. We do another test to see if there are any http_proxy entries that have the current address and port set.

If we don’t detect anything, simply append the lines to the end of the file using the echo command.

If we don’t detect the right config, then we use sed to replace the existing settings with the correct ones.

Excellent. We have enough checking that we can run this code as much as we like and not be afraid of messing things up.

Command Line: the weirdness begins

We’ve dealt with /etc/profile , but we should also consider /etc/bashrc as well. That way we can cover both interactive (aka user) shells as well as non-interactive (aka process) shells running commands. The good news is that the code to deal with that eventuality is almost exactly the same as the stuff above. So, quick section. Woohoo!

proxysettest=$( cat /etc/bashrc | grep http_proxy | wc -l | awk '{ print $1 }' )
proxyconfigtest=$( cat /etc/bashrc | grep http_proxy=http://$address:$port/ | wc -l | awk '{ print $1 }' )

if [ "$proxysettest" = "0" ];
then
   echo "export http_proxy=http://$address:$port/" >> /etc/bashrc
   echo "export HTTP_PROXY=http://$address:$port/" >> /etc/bashrc
   echo "export https_proxy=https://$address:$port/" >> /etc/bashrc
   echo "export HTTPS_PROXY=https://$address:$port/" >> /etc/bashrc
fi

if [ "$proxyconfigtest" = "0" ];
then
   sed -i -e 's,http://.*,http://'"$address"':'"$port/"',g' /etc/bashrc
   sed -i -e 's,https://.*,https://'"$address"':'"$port/"',g' /etc/bashrc
fi

Pretty much we change the path to the file.

Command Line: The sudo redux

Now we run into an interesting problem. If you open up a terminal window, you’ll be proxied. If you sudo your way to root, you are not. This is because you probably didn’t run as a new login shell and frankly unless you read man pages on su and sudo you probably didn’t know this was a problem.

Now I have good news and bad news. Good news is this is doable for all situations. Bad news it involves getting dirty with sudoers.

Anyone who’s done anything with sudoers knows of the dire warnings that are in the man page. Specifically the use of the visudo command to check your syntax before committing to the file. This starts to make life complicated especially if you want to modify such an essential file that can break serious stuff if messed with.

Thankfully there’s a directory next to the sudoers file in /etc called sudoers.d . Aha! We don’t have to get our hands too dirty! We can just write the stuff we need into a file in that folder, set owner and permissions and job done! Right? Not quite.

Thanks to Rich Trouton for finding this: https://stackoverflow.com/questions/21640770/file-in-etc-sudoers-d-file-not-being-read-by-sudo

Short version is you can’t just call the file anything you want. However I settled on “1-configureproxy” and that works just fine. Now lets talk some code.

echo "Defaults env_keep += "'"http_proxy"'" " >> /etc/sudoers.d/1-configproxy
echo "Defaults env_keep += "'"https_proxy"'" " >> /etc/sudoers.d/1-configproxy
echo "Defaults env_keep += "'"HTTP_PROXY"'" " >> /etc/sudoers.d/1-configproxy
echo "Defaults env_keep+= "'"HTTPS_PROXY"'" " >> /etc/sudoers.d/1-configproxy
chown root:wheel /etc/sudoers.d/1-configproxy
chmod 0644 /etc/sudoers.d/1-configproxy

So similar to previous code but also similar issues. First of all this will create a file and just keep appending those lines every time it’s run, so we should do something about that. Also you may have noticed the strange quoting in those echo commands. That’s because putting quotes inside quotes in bash can be tricky to handle, so we escape them out. The idea is to get a line that looks something like this:

Defaults env_keep += "http_proxy"

That’s why there’s multiple quotes around the http_proxy part of the line.

What these lines are actually doing is telling a function called “env_keep” to keep any existing variables with a specified name. That allows the variables we set above in the previous sections to be passed through to the sudo shell someone has invoked. We have to do this four times as multiple variables per line can be unreliable.

Again this is basic code and we are appending to a file. Potentially over and over. Thankfully in bash, file tests are pretty easy with the if command. You end up with this:

if [ ! -f "/etc/sudoers.d/1-configproxy" ];
then
   touch /etc/sudoers.d/1-configproxy
   echo "# This should preserve the proxy variables if we sudo" >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"http_proxy"'" " >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"https_proxy"'" " >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"HTTP_PROXY"'" " >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"HTTPS_PROXY"'" " >> /etc/sudoers.d/1-configproxy
   chown <a href="http://root:wheel" target="_blank" rel="noopener">root:wheel</a> /etc/sudoers.d/1-configproxy
   chmod 0644 /etc/sudoers.d/1-configproxy
fi

So if the file doesn’t exist, create it. If it does exist, do nothing because we only need to set this once. Not bad.

Putting it all together …

We have all the pieces, combining it all will provide a script you can run as many times as you want. I have some extra logic in my production script that’s environment specific but for most of you this will work nicely. Enjoy. Github post coming later.

#!/bin/bash

# Script to deal with proxy settings.
# Set if missing. Reconfigure if necessary.
# Author: richard at richard - purves dot com

address=“proxy.address”
port=“portnumber"

# Have we created a file in /etc/sudoers.d ? Create/Replace here.
if [ ! -f "/etc/sudoers.d/1-configproxy" ];
then
   touch /etc/sudoers.d/1-configproxy
   echo "# This should preserve the proxy variables if we sudo" >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"http_proxy"'" " >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"https_proxy"'" " >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"HTTP_PROXY"'" " >> /etc/sudoers.d/1-configproxy
   echo "Defaults env_keep += "'"HTTPS_PROXY"'" " >> /etc/sudoers.d/1-configproxy
   chown root:wheel /etc/sudoers.d/1-configproxy
   chmod 0644 /etc/sudoers.d/1-configproxy
fi

# Check if /etc/profile has been configured. Configure if not. Change if needed.
proxysettest=$( cat /etc/profile | grep http_proxy | wc -l | awk '{ print $1 }' )
proxyconfigtest=$( cat /etc/profile | grep http_proxy=http://$address:$port/ | wc -l | awk '{ print $1 }' )

if [ "$proxysettest" = "0" ];
then
   echo "export http_proxy=http://$address:$port/" >> /etc/profile
   echo "export HTTP_PROXY=http://$address:$port/" >> /etc/profile
   echo "export https_proxy=https://$address:$port/" >> /etc/profile
   echo "export HTTPS_PROXY=https://$address:$port/" >> /etc/profile
fi

if [ "$proxyconfigtest" = "0" ];
then
   sed -i -e 's,http://.*,http://'"$address"':'"$port/"',g' /etc/profile
   sed -i -e 's,https://.*,https://'"$address"':'"$port/"',g' /etc/profile
fi

# Check if /etc/bashrc has been configured. Configure if not. Change if needed.
proxysettest=$( cat /etc/bashrc | grep http_proxy | wc -l | awk '{ print $1 }' )
proxyconfigtest=$( cat /etc/bashrc | grep http_proxy=http://$address:$port/ | wc -l | awk '{ print $1 }' )

if [ "$proxysettest" = "0" ];
then
   echo "export http_proxy=http://$address:$port/" >> /etc/bashrc
   echo "export HTTP_PROXY=http://$address:$port/" >> /etc/bashrc
   echo "export https_proxy=https://$address:$port/" >> /etc/bashrc
   echo "export HTTPS_PROXY=https://$address:$port/" >> /etc/bashrc
fi

if [ "$proxyconfigtest" = "0" ];
then
   sed -i -e 's,http://.*,http://'"$address"':'"$port/"',g' /etc/bashrc
   sed -i -e 's,https://.*,https://'"$address"':'"$port/"',g' /etc/bashrc
fi

# Find the current Ethernet device(s) and read into an array for later processing.
OIFS=$IFS
IFS=$'\n'
   ethport=($( networksetup -listallnetworkservices | grep Ethernet ))
IFS=$OIFS

# Set the current Ethernet device(s)
for (( i=0; i<${#ethport[@]}; i++ ));
do
   echo "Configuring ${ethport[$i]} to use $address:$port"
   networksetup -setwebproxy "${ethport[$i]}" "$address" "$port"
   networksetup -setsecurewebproxy "${ethport[$i]}" "$address" "$port"
done

Richard Purves

Recovering Filmmaker. Recovering Apple Admin. Recovering Brit.