Eternal Media

I’ll just leave a note here for those of you who have seemingly random kernel panics at conferences or other places with IPv6 while you’re connected to your employer’s network using Cisco’s VPN client.

Cisco VPN-client, VMware, IPv6 and Mac OS X equals kernel panic

Here’s the magic recipe for cooking up a kernel panic.

• Mac OS X Leopard, any version will do (Tiger might be affected too)
• VMware or Parallels installed (a VM does not need to be running)
• IPv6 enabled on an interface attached to a network where IPv6 is actively spoken
• Connect to your corporate network with Cisco’s VPN client (any version will do)
• Do some work, preferrably important, at your computer for less than 10 minutes

Before you know it Cisco’s com.cisco.nke.ipsec kernel extension decides it’s a good time crash Mac OS X.

After the kernel panic you’ll have a brand new file in /Library/Logs/PanicReporter. It will look something like this:

Wed Sep 2 00:35:04 2009
panic(cpu 1 caller 0×001AB0FE): Kernel trap at 0×4860f618, type 14=page fault, registers:
CR0: 0×8001003b, CR2: 0×0eae60b1, CR3: 0×01088000, CR4: 0×00000660
EAX: 0×00000004, EBX: 0×0eae60b1, ECX: 0×00000003, EDX: 0×0eae60b1
CR2: 0×0eae60b1, EBP: 0×47bc7868, ESI: 0×0eae5cb1, EDI: 0×0eae5c50
EFL: 0×00010206, EIP: 0×4860f618, CS: 0×00000008, DS: 0×0e5b0010
Error code: 0×00000000

Backtrace (CPU 1), Frame : Return Address (4 potential args on stack)
0×47bc7678 : 0×12b4c6 (0×45f91c 0×47bc76ac 0×13355c 0×0)
0×47bc76c8 : 0×1ab0fe (0×469a98 0×4860f618 0xe 0×469248)
0×47bc77a8 : 0×1a1713 (0×47bc77c0 0×0 0×47bc7868 0×4860f618)
0×47bc77b8 : 0×4860f618 (0xe 0×530048 0×47bc0010 0xe5b0010)
0×47bc7868 : 0×4860fe72 (0xeae60b1 0×601a8c0 0xd0511ac 0×12fdc2)
0×47bc78b8 : 0×48610bc0 (0xd0511ac 0xeae5c30 0×6 0×379d2b)
0×47bc7938 : 0×48609f8b (0xfb12804 0xfd7b80c 0xeae5c0e 0xf3)
0×47bc79c8 : 0×4860a912 (0xfb12804 0xfd7b80c 0xeae5c0e 0xf3)
0×47bc7a88 : 0×4860675e (0×47bc7adc 0×47bc7ae0 0×47bc7ae4 0×47bc7ad4)
0×47bc7b08 : 0×218126 (0×0 0×5897004 0×2 0×47bc7b6c)
0×47bc7b88 : 0×22af42 (0×5897004 0×2 0×43599900 0xab2bb50)
0×47bc7bb8 : 0×245220 (0×5897004 0×2 0×43599900 0xab2bb50)
0×47bc7d78 : 0×24bcd3 (0×43599900 0×1 0×0 0xd07da00)
0×47bc7de8 : 0×24d4c2 (0×1 0×0 0×0 0×1)
0×47bc7f18 : 0×24fd6f (0xd07da68 0×206 0×47bc7f58 0×23271f)
0×47bc7f58 : 0×2503a2 (0xd07da68 0×0 0×1 0×1a436f)
Backtrace continues…
Kernel loadable modules in backtrace (with dependencies):
com.cisco.nke.ipsec(2.0.1)@0×48604000->0×48672fff

BSD process name corresponding to current thread: kernel_task

Mac OS version:
9L30

Kernel version:
Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386
System model name: MacBookPro2,2 (Mac-F42187C8)

A known bug

This is a known bug. Here’s an excerpt I got after trying to convince Cisco for months that their kernel extension is buggy for their VPN client is broken on Mac OS X.

Cisco VPN Client for Mac OS X is not supported if Fusion, Parallels or Crossover is installed on the same workstation. This hasn’t been documented yet in the usual way but there is a bug in the bug toolkit for that:

CSCsj38286 Bug Details
================
unity mac fails with parallels fusion and crossover

Symptoms
================
Parallels, Fusion, and CrossOver prevent the VPN Client from working properly.

Conditions
================
The VPN Client does not support the use of virtual environments.

Workarounds
================
Cisco VPN with Parallels:

* Install the VPN Client SW on MacOS.
* Configure Paralllels Networking. You’ll probably want to use “shared networking” in Parallels. This causes Parallels to share a single MacOSX-side IP address by using NAT on all network traffic to/from the guest virtual machines/OS’s. Some applications (IPTV) are NAT-intolerant and won’t work in this case.

Alternately, instead of running Cisco VPN Client on MacOS, it can be run on the Windows side. Of course, then MacOSX would not be ‘inside’ the VPN but Windows would be.

It will not be fixed even though the bug is easily reproducible.

The workaround

Remember the key ingredients previously listed? Cisco VPN (connected to the VPN), VMware/Parallels and IPv6. Take one out and the problem goes away!

You obviously need Cisco VPN to be able to work so it’s not a good candidate for removal. VMware/Parallels could be removed but it’s quite convenient to have the ability to run VMs on your computer. That leaves us with IPv6, which happens to be easy to turn off an on. Just go to System Preferences, Network, Advanced, the TCP/IP tab and set “Configure IPv6″ to “Off”. Now you’ll be able to work connected to a VPN while still being able to keep VMware/Parallels on the same computer.

Yet another IPv6 issue with Cisco VPN client

VPN concentrators and the like can force connecting clients to block access to the LAN they’re on. This feature is called “Allow local LAN access” and is usually turned off (i.e, block local LAN access). The obvious use for this is to prevent potential attackers (or worms, viruses, ..) from accessing the corporate network through a client connected to it via a VPN connection.

This blocking feature on the client does not work with IPv6 (or anything else than IPv4). So, if you’ve got a client with IPv4 and IPv6, connecting to your corporate network with Cisco’s VPN client it’s possible for a remote attacker to gain access to your network by first accessing the client over IPv6 and then walking straight into to your network over the VPN connection. Lame.

Snow Leopard has native support for Cisco IPSec VPN

Go to System Preferences, select Network and add a new network interface of type VPN (Cisco IPSec).

A couple of weeks ago I listened in on a stream from the IETF 75 meeting in Stockholm where Gunnar Kreitz was having a speech on behalf of Spotify. Slides are available from spotify-gunnar-kreitz.pdf and there’s more information available here and here.

At the end of his speech someone brought up the question about IPv6. Needless to say they did not currently support IPv6 and had the usual “not-at-this-time” plans for implementing it. However, they claimed to have built their protocol with support for IPv6 in mind.

One of the usual arguments for ignoring IPv6 is that there’s no demand for it, something which is part of the chicken-and-egg problem where no demand creates no need to support it and vice versa.

At least in Sweden there’s a lack of useful content (for Joe Average) available over IPv6 that could aid in creating demand for IPv6. Only a small number of companies provides useful stuff over IPv6.

Spotify would be a great candidate for making even more useful content available over IPv6. Spotify have the advantages of being both popular and being in control over their software and the protocol used for communication. Since Spotify does P2P it also needs to do NAT traversal and requires at least one of the clients to be able to accept incoming connections by means of no NAT or UPNP/NAT-PMP. Supporting IPv6 could potentially lift some of the restrictions imposed by NAT and hence decrease the load on their content distribution network.

When the Spotify client wants to establish a session with Spotify’s server (aka Access Point) it starts off by looking up the service in DNS.

$ dig +short _spotify-client._tcp.spotify.com srv
10 100 4070 georgia.lon.spotify.com.
10 100 4070 larissa.lon.spotify.com.
10 100 4070 charlotte.lon.spotify.com.
10 100 4070 diana.lon.spotify.com.
10 100 4070 grace.lon.spotify.com.
10 100 4070 greta.lon.spotify.com.
10 100 4070 denise.lon.spotify.com.
10 100 4070 esther.lon.spotify.com.

These are SRV records where each record describe the service in terms of priority, weight, port and hostname for the servers. Right now none of the listed servers have AAAA records, i.e, there are no IPv6 addresses associated with the hostname.

What happens next is that the server list is sorted by priority and weight before the client tries to connect to the first server in the list. If it fails, it will go on and try the next one. If all attempts to connect to the servers on port 4070 fail, the client will revert to try some commonly open ports such as 80 (HTTP) and 443 (HTTPS) - because everyone loves music.

Once the session to the access point is established, most of the data the client needs to exchange with the server is handled inside that connection. Metadata such as playlist-, track-, artist- and album details, including album arts aswell as music is multiplexed inside this single connection. Aside from client updates, the only traffic that happens outside this connection is related to content distribution via P2P and what seems to be a relatively new HTTP based CDN.

The client gracefully handles errors that occur, such as connections problems or playlists or tracks being temporarily unavailable. It also seems to keep track download speeds from the access point, P2P and the CDN.

It also regularly reports latencies of different operations (such as time between a double-click on a track to when it actually starts to play) to the server.

It’s clear there’s already infrastructure available to measure delays or other issues that could be imposed by clients accessing their service over IPv6 aswell as a general design philosophy of handling any network or service errors gracefully.

Hooking up one or more of the access points with IPv6 should - in essence - be a rather simple operation. Since the client already deals with unreliable wireless networks with latency and bandwidth fluctations none of the complaints against ISP IPv6 networks should apply either.

Judging from symbols in the binaries for Spotify and libspotify it seems as if it’s using obsolete IPv4 interfaces such as gethostbyname(3) for resolving hostnames. It’s simple to replace with more modern facilities such as getaddrinfo(3) as detailed in Address-familiy independent socket programming for IPv6.

Need beta testers? Sign me up and I’ll help finding more people like myself.

So who’s using IPv6? Pretty much everyone that didn’t turn off Teredo in Windows Vista. Most that didn’t turn off the 6to4 tunnel option in older Apple Airports. Those that did turn it on. People and companies that just thinks IPv6 is plain cool.

Most OSes (Windows XP SP3, Vista, Mac OS X, Linux) today have production quality support for IPv6 out of the box. Windows Vista comes with Teredo turned on by default.

Update: Spotify’s CTO Andreas Ehn responded to a tweet about this blog post:

Agreed. We’re looking into it. Don’t hold your breath, though.

I recently installed Microsoft Visual C++ 2008 Express to hack on Openspotify. It’s been a few years since I last used Visual Studio so I was a bit curious on what improvments was available in the 2008 release.

I was baffled to discover that some of the annoyances are still there. For example, adding search paths for the preprocessor, or libraries to link with, is still very much !userfriendly. And the GUI elements looks like the year just struck 1995 again.

Quite unexpected in a 2008 release.. It makes me wonder if it’s intentionally kept like this. Maybe it’s some kind of inside joke?

The other day a server got overloaded and crashed after falling victim to a cascading effect due to poorly written code. Which interestingly also caused troubles for other servers depending on the first one. Yikes!

The recipe for ruining someone’s Friday night

When two hosts tries to communicate with eachother there is a whole bunch of things that cango wrong. Most of the time though, the network connecting two hosts (and related services) work just fine and there’s nothing to worry about. And while writing code it’s sometimes appealing to not bother dealing with the unexpected.

Imagine a small PHP script like the one below, crontab’d to run every minut:

<?php

$c = curl_init("http://example.net/feed.xml");

curl_setopt($c, CURLOPT_RETURNTRANSFER, TRUE);

$xml = curl_exec($c);

/* Do something useful and make the world go round here */

?>

What could possibly go wrong? Tons of stuff, but for now we’ll just concentrate on network related issues.

ARP resolution failures

For two hosts in the same subnet to communicate over IP they need to know eachothers MAC addresses. MAC addresses are link-local addresses in the form of 00:17:fa:d1:19:0b, unique (though not globally) for each ethernet network card.

The protocol that provides resolution of an IP address to a MAC address is called ARP (address resolution protocol) and is defined in RFC826.

Initially a the ARP table cache is empty and before a remote host on the same subnet (or the local router) can be contacted the local host needs to figure out its MAC address. This is done by sending an ARP packet to the ethernet broadcast address (ff:ff:ff:ff:ff:ff) that basically asks, who has IP address 1.2.3.4? All hosts on the subnet will see this packet but only the host who has IP address 1.2.3.4 will respond to the query with an ARP response packet that says “1.2.3.4 is at 00:11:22:33:44:55″.

By default Linux 2.6 will send 3 ARP “who has IP x.x.x.x” packets, one each second before giving up. After giving up the software that tried to contact the remote host on the same subnet will be notified by the means of a “No route to host” error.

Computers and devices in IP networks maintains an ARP table cache where IP addresses are mapped to MAC addresses. In Linux 2.6 the default maximum age for a cached entry in the ARP table is 60 seconds and can be modfied by changing the sysctl variable net.ipv4.neigh.default.gc_stale_time. (On many Cisco switches and routers the default mx age is 4 hours).

In the event of replacing a server with a given IP address by another one, with a different network card and hence a different MAC address, another problem could occur. If hosts (including the router) on the local subnet have previously been talking to that IP adress they might still have its MAC address cached in the ARP table cache.

That means that communicating with the replaced host will be subject to upper protocol timeouts until the cached ARP entry expires.

DNS failures

When software tries to resolve the hostname of a server to its IP address it contacts configured resolvers on the local machine. In most UNIX like systems this is usually /etc/resolve.conf which contains a list of one or more nameservers.

On Linux DNS resolution is handled by GLIBC’s resolver library. It will try to contact each of the nameservers listed in the order they’re listed. Each listed server will be given 5 seconds to get back with an answer before the resolver attempts to contact the next server listed. That means software depending on working DNS resolution (handled by calls such as getaddrinfo(3) or gethostbyname(3)) could potentially be blocked for 2×5 seconds if the first two listed nameservers does not respond at all.

One reason why this might happen is obviously because the configured resolver is down.

Another reason might be because of DNS fuckups in the zone which holds the hostname being resolved. In its simple form, when a nameserver tries to resolve foo.example.net it will first contact one of the DNS root servers and query it for a list of responsible nameservers for the .net domain. Next, it will then go to one of those nameservers and query it about what nameservers are responsible for the domain example.net. Finally it will query the nameservers for example.net about foo.example.net.

Now, the nameservers listed for example.net might just be plain wrong and/or not responding and can also cause the local resolver to timeout while waiting for an answer from the upstream resolver.

TCP failures

When establishing a TCP connection to a service on a remote host the local hosts send TCP packets with the SYN flag set to inform the remote host it wants to establish a TCP session to the service identified by the port specified.

If the remote host is reachable and responding to packets, but no program is listening on port, the contacting host will be notified immediately by TCP packet with the RST flag set, causing a “Connection refused” error to be delievered to the software that tried to connect. That usually happens as fast as packets can be transmitted (provided that they’re not lost on the way).

On the other hand, if the remote host is not responding, because of it being down, because of a firewall filtering packets, or general packet loss, the software trying to contact a remote host will be subject to retries and timeouts defined by the TCP/IP stack.

In Linux, the number of times the TCP/IP stack will retry a connection attempt (i.e, retransmit SYN packets) is controlled by the sysctl variable net.ipv4.tcp_syn_retries. It’s set to 5 by default.

In terms of delay before the software gets a “Connection timed out” error, this means:

• Send initial SYN packet and wait for a response for 3 seconds
• Retransmit SYN packet and wait up to 6 seconds (retry #1)
• Retransmit SYN packet and wait up to 12 seconds (retry #2)
• Retransmit SYN packet and wait up to 24 seconds (retry #3)
• Retransmit SYN packet and wait up to 48 seconds (retry #4)
• Retransmit SYN packet and wait up to 96 seconds (retry #5)

3*(2^6) adds up to 192 seconds(!)

Once the software has successfully established a TCP connection there’s of course no guarantee that the remote host will keep on being reachable for the life of the session. It can happen because of changes in the network, because of packet loss, etc.

Here’s an excerpt from the tcp(7) man page in Linux:

      tcp_retries1 (integer; default: 3)

The  number of times TCP will attempt to retransmit a packet on

an established connection normally, without the extra effort of

getting  the network layers involved.  Once we exceed this num

ber of retransmits, we first have the network layer update  the

route  if  possible before each new retransmit.  The default is

the RFC specified minimum of 3.

       tcp_retries2 (integer; default: 15)

The maximum number of times a TCP packet  is  retransmitted  in

established  state  before giving up.  The default value is 15,

which corresponds to a duration of approximately between 13  to

30  minutes,  depending  on  the  retransmission  timeout.  The

RFC 1122 specified minimum limit of 100  seconds  is  typically

deemed too short.

There are of course some mitigating factors such as:

• The packet loss that stalled the connection might just be temporary
• If the remote host rebooted it will likely come up within a few minutes and will respond with a TCP packet with the RST flag set, effectively killing the session on the other host
• The software that was talking to the remote host might maintain its own timeouts and tear down the connection when it detects it stalled

Fixing the script

Going back to the previously presented PHP script, it turns out the default timeout for libcurl is… to never timeout(!)

It’s easy to fix by introducing the following line of code before the call to curl_exec()

curl_setopt($c, CURLOPT_TIMEOUT, 10);

That will tell libcurl to never wait more than 10 seconds for anything, including DNS resolution, TCP connects, HTTP responses and data transfers.

Software defined timeouts in Apache and Varnish

In Apache there are basically two software defined timeouts of interests. The first is directive Timeout which controls how long Apache will sit around waiting for I/O in various situations. It defaults to 300.

The second timeout of interest is the directive ProxyTimeout which defines how long Apache will wait for a backend webserver configured with the Proxy directive. It defaults to the value of Timeout, i.e. 300 seconds.

In Varnish 2.0.4 there’s three interesting timeouts. The first is the parameter connect_timeout which controls how long Varnish will wait before giving up connecting to a backend web server. It defaults to 0.4 seconds.

Next up is the parameter first_byte_timeout which controls how long to wait for a response from a backend server. It defaults to 60 seconds.

Also related is the parameter, between_bytes_timeout, which controls how long to wait for more data after the last read. It also defaults to 60 seconds.

Being away from home it’s nice to be able to hook up to the Internet by means of a wireless network such as 802.11b/g/n or 3G.

Many of these network providers require you to pay ridiculous amounts of money for getting an hour or day worth of internet access.

They usually implement this by having a transparent proxy between the network you connect to and the internet. While not yet being logged into the network as a paying customer, the transparent proxy intercepts outgoing HTTP connections and redirect you to their portal where you can buy access to the internet.

Everything else is usually blocked except for DNS. Allowing recursive DNS is  usability feature that allows you to type in pretty much anything in the browser’s address bar so you can be redirected to their portal, without poisoning the DNS cache held by the resolver.

Abusing DNS for fun and profit

Having the ability to do recursive DNS lookups allows you to tunnel IP traffic over DNS. In 2002 I wrote something called FLSToS, FLS Theft of Service, which did just that. It was useful for avoiding having to pay for internet access when connected to Telia Homerun (wifi) or while not being logged in on a ComHem cable network.

Recently I also discovered Iodine which also does IP-over-DNS. It looks a lot more cleaner than the code I wrote for FLSToS and still seems to be under active development. Cool!

Accidently finding a reason to write code

Going back to that transparent proxy thing, I had an interesting experience with one large wireless networks I recently hooked up to.

Once connected to the network, I was curious and figured it could be interesting to see what the HTTP redirect response from the transparent proxy looked like. I fired up telnet (die netcat, die!) to connect to a wellknown web server and typed in a HTTP request manually.

Being tired and all I accidently forgot to type in the Host: header required by the HTTP 1.1 specification in my request. I did get a response, but not the one I expected.

Interestingly, what was returned was a custom error message I had configured myself on that well known web server I connected to. Which essentially meant, I was talking to real server and not to the network’s portal.

Retrying the operation, now including the Host: header got me a HTTP redirect from the transparent gateway. WTF?

After playing around some more I found out you could speak anything except HTTP 1.1 over port 80 and end up talking to the real server. According to people in the know, the device handling these redirects is loaded with custom firmware to do the job and is unlikely to be fixed.

That’s cool, but it would indeed be more useful if you could somehow configure your computer to multiplex TCP over that single, open port.

OpenSSH + DynamicForward

OpenSSH has support for something called DynamicForward which essentially turns ssh(1) into a SOCKS4/5 server that forwards connections over the SSH connection, and terminates them on the SSH server you connected to.

While it worked great in terms of performance it unfortunately was a hazzle to reconfigure all sorts of applications to use a SOCKS server. Neither Spotify nor Mozilla Firefox adhere to the system-wide settings for proxies to be used in networked applications, just to name a few.

Going back to the 90ies with SLIRP

Next up was running super old-school SLIRP over the SSH connection, to be able to route IP over the ppp0 interface provided by slirp. After having tons of weird issues (hello Linux TTY issues!)  it turned out having awful performance, providing 15-20 times worse throughput than native access.

SLIRP is dead old and does TCP/IP in userland by emulates outgoing connections with POSIX connect(2) and shuffles data with read(2) and write(2). So I figured, having a super old TCP/IP stack in combination with already running slirp over SSH and TCP/IP over a somewhat laggy connection is not a recipe for performance. Besides, it was annoying to have to compile up a non-default software such as slirp on every computer you would like to do this dance on.

Trying a more raw PPP solution

Pppd is available on every Mac OS X and Linux installation and also provides a ppp0 interface you could route IP traffic over. It seemed like a better candidate. After having set up a pppd to be run when logging in with SSH on the remote server and configuring pppd locally on my Mac it too turned out to suck performance wise.

OpenVPN can provide an interface too

In a hope to try to get rid with some of the overhead introduced by SSH and pppd I setup an OpenVPN server on port 80 on the remote host. Annoyingly performance still sucked, in fact it was even worse than SSH+pppd. TCP/IP over TCP/IP justs sucks.

An interesting firewall hack

I found an interesting piece of software called Transocks, Transparent SOCKSifying Proxy. It takes advantage of the redirect feature present with iptables and redirects outgoing connections to a local port. Transocks then listens on that port, accepting incoming connections and figure out where the connection is supposed to go by using a Linux specific ioctl() to grab the original destination address and port. Using this information it then connected to a pre-configured SOCKS server and tells it to connect to the attempted destination. From that point on it just shuffles data between the intercepted connection and the SOCKS server.

However, there were a number of issues with transocks. For once it wanted to link with an external SOCKS library which requires more non-default stuff to be installed. Secondly, the iptables rules provided as an example by its docs didn’t work out of the box on Ubuntu due to missing iptables modules, old unsupported usage etc. And finally it was Linux specific. I have a Mac

I suffer from the NIH syndrome, but it’s OK!

I decided to write my own version and implement SOCKS directly (it’s easy!) to get rid of the dependence of an external library. And of course it needed to run on Mac OS X too.

Enter fw-socks5-fwd.c. It does what transock was supposed to do, but actually works, has cleaner code and works on Mac OS X too.

$ ./fw-socks5-fwd
Transparent TCP port-forwarding over SOCKS5 using firewall next-hop rewrites
– noah@hack.se, 2009

Usage: ./fw-socks5-fwd <-d | -f> [-p port] [-s port] [-P /path/to/pid-file]
-d Run in the background (usually preferred)
-f Run in the foreground (useful for debugging)
-p port Port to accept redirected connections on (default is 1211)
-s port Port of SOCKS5 server (default is 1080)
-P file Where to write the process ID (default /var/run/fw-socks5-fwd.pid)
NOTE: This program assumes the SOCKS5 server is listening on 127.0.0.1

$ ./fw-socks5-fwd -d

Transparent TCP port-forwarding over SOCKS5 using firewall next-hop rewrites

– noah@hack.se, 2009

* Listening for redirected connections on local port 1211

* Connections will be forwarded to SOCKS5 server at 127.0.0.1:1080

* Now, make ipfw redirect outgoing connections to this program using

# ipfw add check-state

# ipfw add allow tcp from me to 127.0.0.0/8 setup keep-state

# ipfw add fwd 127.0.0.1,1211 tcp from me to any setup keep-state