Detecting Bufferbloat

Bufferbloat is topic which has been gaining broader attention, but is still not widely understood. This post will walk you through the basics of bufferbloat and how to determine if you are the victim of bufferbloat.

A Brief Synopsis of the Bufferbloat Problem

The topic of bufferbloat has been explained wide and far, but I'll add to the conversation too, focusing on brevity. This summary is based on the highly informative and technical talk Bufferbloat: Dark Buffers in the Internet, a Google Tech Talk by Jim Gettys. There is an assumption in the design of TCP that if there is network congestion, there will be timely packet loss. This packet loss triggers well designed TCP flow control mechanisms which can manage the congestion. Unfortunately, engineers designing consumer grade routers and modems (as well as all sorts of other equipment) misunderstood or ignored this assumption and in an effort to prevent packet loss added large FIFO (first-in-first-out) buffers. If users congest a network chokepoint, typically an outgoing WAN link, the device's large buffers are filled with packets by TCP and held instead of being dropped. This "bufferbloat" prevents TCP from controlling flow and instead results in significant latency.

Detecting Bufferbloat

All that's required to experience bufferbloat is to saturate a segment of your network which has one of these large FIFO buffers. Again, the outgoing WAN link is usually the easiest to do, but can also happen on low-speed WiFi links. I experienced this myself when installing Google's Music Manager, which proceed to upload my entire iTunes library in the background, at startup, using all available bandwidth. (Thanks Google!) I detected the latency using mtr. Windows and OS X does not offer such a fancy tool, so you can simply just ping your WAN gateway and see the lag.

Music Manager enabled, bufferbloat, slow ping to WAN gateway

Music Manager paused, no bufferbloat, fast ping to WAN gateway

Managing Bufferbloat

Unfortunately, there are no easy answers out there right now for many users. Often we cannot control the amount of bandwidth a piece of software will try to use or the equipment given to us by an ISP. If you are looking for a partial solution to the problem, checkout Cerowrt, a fork of the OpenWrt firmware for routers. It makes use of the best available technologies used to combat bufferbloat. Additionally, be on the look out for any software that might saturate a network segment, such as Bittorrent, Netflix streaming, or large file transfers over weak WiFi links.

IPv6 Tunnels with Debian/Ubuntu behind NAT

As part of End Point's preparation for World IPv6 Launch Day, I was asked to get my IPv6 certification from Hurricane Electric. It's a fun little game-based learning program which had me setup a IPv6 tunnel. IPv6 tunnels are used to provide IPv6 for those whose folks whose ISP or hosting provider don't currently support IPv6, by "tunneling" it over IPv4. The process for creating a tunnel is straight forward enough, but there were a few configuration steps I felt could be better explained.

After creating a tunnel, Hurricane Electric kindly provides a summary of your configuration and offers example configurations for several different operating systems and routers. Below is my configuration summary and the example generated by Hurricane Electric.

However, entering these commands change won't survive a restart. For Debian/Ubuntu users an update in /etc/network/interfaces does the trick.

1 2 3 4 5 6 7 8 9

#/etc/network/interfaces auto he-ipv6 iface he-ipv6 inet6 v4tunnel   address 2001:470:4:9ae::2   netmask 64   endpoint 209.51.161.58   local 204.8.67.188   ttl 225   gateway 2001:470:4:9ae::1

Firewall Configuration

If you're running UFW the updates to /etc/default/ufw are very straightforward. Simply change the IPV6 directive to yes. Restart the firewall and your network interfaces and you should be able to ping6 ipv6.google.com. I also recommend hitting http://test-ipv6.com/ for a detailed configuration test.

Behind NAT

If you're behind a NAT, the configuration needs to be tweaked a bit. First, you'll want to setup a static IP address behind your router. If you're router supports configuration of forwarding more than just TCP/UDP, you'll want to forward protocol 41 (aka IPv6) (NOT PORT 41), which is responsible for IPv6 tunneling over IPv4, to your static address. If you've got a consumer grade router that doesn't support this, you'll just have to put your machine in the DMZ, thus putting your computer "in front" of your router's firewall. Please make sure you are running a local software firewall if you chose this option.

After handling the routing of protocol 41, there is one small configuration change to /etc/network/interfaces. You must change your tunnel's local address from your public IP address, to your private NATed address. Here is an example configuration including both the static IP configuration and the updated tunnel configuration.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

#/etc/network/interfaces auto eth0 iface eth0 inet static   address 192.168.0.50   netmask 255.255.255.0   gateway 192.168.0.1   auto he-ipv6 iface he-ipv6 inet6 v4tunnel   address 2001:470:4:9ae::2   netmask 64   endpoint 209.51.161.58   <b>local 192.168.0.50</b>   ttl 225   gateway 2001:470:4:9ae::1

Don't forget to restart your networking interfaces after these changes. I found a good ol' restart was helpful as well, but of course, we don't have this luxury in production, so be careful!

Checking IPv6

If you're reading this article, you're probably responsible for several hosts. For a gentle reminder which of your sites you've not yet setup IPv6, I recommend checking out IPvFoo for Chrome or 4or6 for Firefox. These tools make it easy for you to see which of your sites are ready for World IPv6 Launch Day!

Getting Help

Hurricane Electric provides really great support for their IPv6 tunnel services (which is completely free). Simply email ipv6@he.net and provide them with some useful information such as:

1 2 3 4

cat /etc/network/interfaces cat netstat -nrA inet6  (these are your IPv6 routing tables) cat /etc/default/ufw relevant router configurations

I was very impressed to get a response from a competent person in 15 minutes! Sadly, there is one downside to using this tunnel; IRC is not an allowed.

Due to an increase in IRC abuse, new non-BGP tunnels now have IRC blocked by default. If you are a Sage, you can re-enable IRC by visiting the tunnel details page for that specific tunnel and selecting the 'Unblock IRC' option. Existing tunnels have not been filtered.

I guess ya gotta earn it to use IRC over your tunnel. Good luck!