I recently completed reading the pf FAQ on the OpenBSD website. I have been doing quite a bit of playing around and thought I would document the testing I performed on creating a redundant firewall configuration using CARP and pfsync.
I decided I must do a little experimenting based on two quotes from the FAQ. The first quote describes the use of the Common Address Redundancy Protocol (CARP):
One common use for CARP is to create a group of redundant firewalls. The virtual IP that is assigned to the redundancy group is configured on client machines as the default gateway. In the event that the master firewall suffers a failure or is taken offline, the IP will move to one of the backup firewalls and service will continue unaffected.
The second quote is from the section discussing the value of using pfsync with CARP:
pfsync: Synchronizes the state table amongst all the firewalls. In the event of a failover, traffic can flow uninterrupted through the new master firewall.
I read this as follows, “If I have a redundant firewall configuration using both CARP and pfsync, I can fail one firewall and have no interruption in traffic flow.”
That just sounds cool, and I’m going to test it out.
My experiment will be done in a virtual lab environment. I will use Oracle VM VirtualBox for the virtualization platform. The firewalls will be vanilla OpenBSD 5.0 installs. The test hosts will be two other virtual machines I already have set up.
The goal: Simulate the failure of the primary firewall during operations involving Web and SSH traffic traveling from hosts on the LAN side to hosts on the WAN side.
Lets set it up and see.
Architecture Review
First – below is an ASCII diagram of the topology, consistent with the example in the pf FAQ:
+----| WAN/Internet |----+ | | em0| |em0 +-----+ +-----+ | fw1 |-em2----------em2-| fw2 | +-----+ +-----+ em1| |em1 | | ---+-------Shared LAN-------+--- Interface FW1 FW2 em0 192.168.1.21 | 192.168.1.22 em1 172.16.0.21 | 172.16.0.22 em2 10.10.10.21 | 10.10.10.22 WAN Shared 192.168.1.100 LAN Shared 172.16.0.100 WAN Gateway 192.168.1.1
The em2 interfaces will be used to send pfsync updates. Even in a lab environment, I want to heed the warning from the FAQ:
When pfsync(4) is set up to send and receive updates on the network, the default behavior is to multicast updates out on the local network. All updates are sent without authentication. Best common practice is either:
- Connect the two nodes that will be exchanging updates back-to-back using a crossover cable and use that interface as the syncdev (see below).
- Use the ifconfig(8) syncpeer option (see below) so that updates are unicast directly to the peer, then configure ipsec(4) between the hosts to secure the pfsync(4) traffic.
To simulate the crossover cable, we will set up the separate em2 interface. We could also easily configure IPsec on the em2 interfaces. Either way…
CARP will be configured both on the WAN and LAN side to support hosting services on both sides.
For the purposes of this experiment, minimal firewall rules will be in place. This is not recommended for production :)
Firewall Installation
I created a new virtual machine and installed OpenBSD 5.0 following the install guidance. It took about 20 minutes on a well oiled machine. Both firewalls will be identical, so once the first machine was installed and I verified it boots and I could log in, I shut it down and used the VirtualBox “Clone” feature to create a copy of the VM.
From the clean installs of OpenBSD 5.0, the following configurations established the environment.
Virtual Networking Setup
For both firewalls, the VirtualBox network settings are the same…
Adapter 1: Enabled, Bridged, Advanced->Promisc: Allow All
Adapter 2: Enabled, Internal, sharedlan, Advanced -> Promisc: Allow All
Adapter 3: Enabled, Internal, pfsync
Host Network Setup
Firewall 1
Edit: /etc/hostname.em0
inet 192.168.1.21 255.255.255.0 NONE
Edit: /etc/hostname.em1
inet 172.16.0.21 255.255.255.0 NONE
Edit: /etc/hostname.em2
inet 10.10.10.21 255.255.255.0 NONE
Setup gateway:
# echo 192.168.1.1 > /etc/mygate
Setup DNS – Edit: /etc/resolv.conf
lookup file bind
nameserver 208.67.222.222
nameserver 208.67.220.220
Then restart network:
# sh /etc/netstart
Firewall 2
Edit: /etc/hostname.em0
inet 192.168.1.22 255.255.255.0 NONE
Edit: /etc/hostname.em1
inet 172.16.0.22 255.255.255.0 NONE
Edit: /etc/hostname.em2
inet 10.10.10.22 255.255.255.0 NONE
Setup gateway:
# echo 192.168.1.1 > /etc/mygate
Setup DNS – Edit: /etc/resolv.conf
lookup file bind
nameserver 208.67.222.222
nameserver 208.67.220.220
Then restart network:
# sh /etc/netstart
Test Network Setup
Firewall 1
$ for h in "192.168.1.22" "10.10.10.22" "172.16.0.22"; do echo; ping -c1 $h; done;
Firewall 2
$ for h in "192.168.1.21" "10.10.10.21" "172.16.0.21"; do echo; ping -c1 $h; done;
Set Up CARP, pfsync, and Enable Forwarding (NAT)
Firewall 1
Edit: /etc/sysctl.conf
net.inet.carp.preempt=1
net.inet.ip.forwarding=1
This will set preemption and enable forwarding upon next reboot.
So I don’t have to reboot now, we’ll manually set the sysctl variable:
# sysctl -w net.inet.carp.preempt=1
# sysctl -w net.inet.ip.forwarding=1
Create: /etc/hostname.pfsync0
up syncdev em2
Create: /etc/hostname.carp0
inet 192.168.1.100 255.255.255.0 192.168.1.255 vhid 1 \
carpdev em0 pass wan_interface_passwd
Create: /etc/hostname.carp1
inet 172.16.0.100 255.255.255.0 172.16.0.255 vhid 2 \
carpdev em1 pass lan_interface_passwd
Restart network:
# sh /etc/netstart
Note, OpenBSD will fix perms on the hostname.carp files, however, we’ll fix them on hostname.pfsync0:
# chmod 640 /etc/hostname.pfsync0
Firewall 2
Edit: /etc/sysctl.conf
net.inet.carp.preempt=1
net.inet.ip.forwarding=1
So I don’t have to reboot now:
# sysctl -w net.inet.carp.preempt=1
# sysctl -w net.inet.ip.forwarding=1
Create: /etc/hostname.pfsync0
up syncdev em2
Create: /etc/hostname.carp0
inet 192.168.1.100 255.255.255.0 192.168.1.255 vhid 1 \
carpdev em0 pass wan_interface_passwd advskew 128
Create: /etc/hostname.carp1
inet 172.16.0.100 255.255.255.0 172.16.0.255 vhid 2 \
carpdev em1 pass lan_interface_passwd advskew 128
Restart network:
# sh /etc/netstart
# chmod 640 /etc/hostname.pfsync0
Set Up Firewall Fules
The rules on both firewalls will be identical. So, I configured to taste on Firewall 1, and then scp’s the /etc/pf.conf file over to the other. Based on the topology above, the following firewall rules are in place. These are NOT recommended for production!
#----------------------------------------------------------------------
# -=< F I R E W A L L R U L E S >=-
#----------------------------------------------------------------------
#----------------------------------------------------------------------
# M A C R O S A N D L I S T S
#----------------------------------------------------------------------
if_ext = em0 # external if, external if CARP (carp0)
if_int = em1 # internal if, internal if CARP (carp1)
if_pfs = em2 # dedicated if for pfsync
approved_dns = "{ 208.67.222.222 208.67.220.220 }"
#----------------------------------------------------------------------
# O P T I O N S
#----------------------------------------------------------------------
set skip on lo
# Nat outgoing
match out on $if_ext from !$if_ext to any nat-to ($if_ext)
#----------------------------------------------------------------------
# F I L T E R S
#----------------------------------------------------------------------
#Default policy: block
block return in
block return out
block in quick from urpf-failed
# Scrub incoming packets
match in all scrub (no-df)
# Support CARP, pfsync
pass quick on $if_pfs proto pfsync keep state (no-sync)
pass quick on { $if_ext $if_int } proto carp keep state (no-sync)
# Allow outbound DNS to approved DNS resolvers
pass in log (all) on $if_int inet proto { tcp udp } \
from $if_int:network to $approved_dns port domain
pass out log (all) on $if_ext inet proto { tcp udp } \
from $if_ext to $approved_dns port domain
# Allow all outbound Web traffic
pass in log (all) on $if_int inet proto tcp \
from $if_int:network to any port { www, https }
pass out log (all) on $if_ext inet proto tcp \
from $if_ext to any port { www, https } modulate state
# Allow all outbound SSH traffic
pass in log (all) on $if_int inet proto { tcp udp } \
from $if_int:network to any port ssh
pass out log (all) on $if_ext inet proto { tcp udp } \
from $if_ext to any port ssh modulate state
# By default, do not permit remote connections to X11
block in on ! lo0 proto tcp to port 6000:6010
Test CARP Fail Over
Now that the interfaces have been set up, CARP should be running on both the WAN and LAN interfaces. Firewall 1 should be the CARP master on both carp0 (WAN side) and carp1 (LAN side).
This can be seen by running the following commands on both firewalls:
# ifconfig carp0
# ifconfig carp1
The output can be seen in the following screenshot:
Image 1: CARP Configuration on Firewall 1 and Firewall2
We can simulate the failure of the Firewall 1 CARP 1 interface by tweaking the advskew. To see the CARP traffic, we can run the following command on both Firewalls:
# tcpdump -i em1
Then, to fail the interface, run the following on Firewall 1:
# ifconfig carp1 advskew 130
The tcpdump output will show that Firewall 2 promotes to master. This can be seen in the following screenshot:
Image 2: CARP Master Fail Over
We can easily fail back to Firewall 1 by setting the advskew value on the carp1 interface on Firewall 1 to a number lower than the advskew value on the carp1 interface on Firewall 2 (128).
Set Up Hosts
Ultimately we want to have a host on the LAN network (our simulated user’s workstation) and a host on the WAN network (our SSH server) talking, and then fail over the interfaces. The Hosts will be running BackTrack Linux, though any operating system should work fine, provided you can configure a static IP address and default gateway.
Virtual Network Setup
For Host 1 (LAN side), set
Adapter 1: Enabled, Internal, sharedlan
For Host 2 (WAN side), set
Adapter 1: Enabled, Bridged
Host Network Setup
Host 1
Edit: /etc/network/interfaces
FROM:
auto eth0
iface eth0 inet dhcp
TO:
auto eth0
iface eth0 inet static
address 172.16.0.10
netmask 255.255.255.0
gateway 172.16.0.100
Then restart network:
# /etc/init.d/networking restart
Host 2
Edit: /etc/network/interfaces
FROM:
auto eth0
iface eth0 inet dhcp
TO:
auto eth0
iface eth0 inet static
address 192.168.1.30
netmask 255.255.255.0
gateway 192.168.1.1
Note: Host 2 here (on the WAN side) has the gateway set to the external network gateway, not the WAN facing interface on the Firewalls (em0)
Then restart network:
# /etc/init.d/networking restart
DNS Resolver Setup
We will use the OpenDNS resolvers. For both Hosts, the configuration is the same.
Edit: /etc/resolv.conf
208.67.222.222
208.67.220.220
Perform Testing
To perform the test, simply ssh from Host 1 (LAN) to Host 2 (WAN), and then use a web browser on Host 1 (LAN) to browse the web. Optionally have Wireshark or TCPDump running on Host 1 to see what’s going through the interface.
Then, fail over Firewall 1 by setting the advskew value on the CARP interfaces to a value higher than the values on Firewall 2. I have a script (provided below) that can be modified and used to support quickly changing the advskew values.
Results
Test results were very positive. Web based traffic failed over immediately when the master was failed. The SSH session, however, did not move over to the promoted firewall. I did quite a bit of investigating principally by checking TCP dumps and comparing MAC addresses of packets coming and going. Once the master was failed (by setting the advskew to a number higher than the backup firewall), the SSH outbound traffic continued to be sent to the CARP IP address, but response traffic was sent from the MAC address of the failed master firewall. If I shut down the interface ($ifconfig em1 down), then SSH traffic would simply halt.
I turned to the web. I was a little frustrated to see that the CARP protocol is not fully documented (that I could find), though dozens of articles provided adequate troubleshooting information. I found this nugget in the NetBSD Guide chapter on CARP, “services that require a constant connection to the server (such as SSH or IRC) will not be transparently transferred to the other system–though in this case, CARP can help with minimizing downtime.” I suspect that the carp handler on a promoted master host will not take TCP ACK packets for a connection that was NOT initiated through that same host via the TCP 3-way handshake. I could not confirm this. With that in mind, however, I was happy that things worked as expected (SSH traffic behaved in accordance with documented behavior). I wonder if it would be prudent to try to configure a failed master to send TCP RSTs on active connections…
I also noticed a peculiar behavior that I was not able to resolve, but I did find a work around. Under certain conditions, after failing over between FW1 and FW2 several times, network traffic would simply stop. DNS would work, TCP sessions would get created, and the HTTP GET would get sent, but no replies would ever come back. I did not investigate to resolution, however I did find that if I would boot FW2 (the backup firewall) first and then boot FW1 (which would immediately preempt FW2), this problem would NOT materialize. I speculate that the firewall configuration, or some peculiarity in the virtual environment may have been causing this. Again – the work around was reliable, and if I was moving this into production, I’d do more research…
Final Thoughts
I am quite impressed with the simplicity of setup and the effectiveness in implementation of this reliable redundant configuration. I am also impressed that the CARP advertisements are (with the use of the CARP password parameter) encrypted with a SHA1 HMAC. Additionally, pfsync traffic can be exchanged using crossover cables or secured using ipsec. I did note, however, that CARP, even encrypted, is not invulnerable to replay attacks. I wonder if the carp traffic could be encapsulated in an ipsec tunnel between hosts to mitigate this vulnerability.
Useful Scripts and Commands
I created several files and used several tcpdump commands when testing and debugging. Below are those files that, when used with the configuration documented in this post, will make testing quite easy. Additionally, tcpdump commands useful for seeing what was moving through the physical interfaces or the pf log are provided…
BASH Script Files
status.sh – Show current status of CARP interfaces
#!/bin/sh echo "Carp0:" ifconfig carp0 | grep advskew echo "Carp1:" ifconfig carp1 | grep advskew
set_carp_200.sh – Change the advskew value on CARP interfaces to 200. Can be easily modified.
echo "Setting carp0 advskew 200..." ifconfig carp0 advskew 200 echo "Setting carp1 advskew 200..." ifconfig carp1 advskew 200 sleep 7 sh status.sh
Useful TCPDump Commands
em0 – See what’s hitting the em0 interface (here, on FW1) ignoring the CARP advertisements.
tcpdump -i em0 src or dst 192.168.1.21 and !dst 224.0.0.18
em1 – See what’s flowing on the Shared LAN side.
tcpdump -i em1 !dst 224.0.0.18
em2 – Watch the pfsync updates.
tcpdump -i em2
pflog – See what’s being processed by pf and logged to pflog0 in accordance with pf.conf.
tcpdump -n -e -ttt -i pflog0
Video
Below is a short video that demonstrates the fail over of the master, then failing back again. The video was shot using host configured in accordance with the specifications above. Perhaps this will add value. It will be best viewed full-screen…


When most people talk about SCADA, they are generally including a whole lot of stuff that is not SCADA.


















Image 1: Simple Netcat Session
Image 2: Partial Pivot
Image 3: Two-way Pivot
Image 3: Two-Way Pivot Using Named Pipe


