I’m working on a project that involves communication between two Rails apps, one as an API and the other as a backend for web app. The web app communicates with the API via the excellent Typhoeus ruby gem, and everything was going swell until I got to implementing updates via HTTP PUTs for a couple resources.

The problem was that they were way slower than any of the other requests, regardless of payload size–slower than GETs, DELETEs, even POSTs. I tried a PATCH and that was as fast as the rest.

Since at the time I was originally debugging this I had no Internet access, I began throwing logging into the internals of Typhoeus everywhere, and eventually narrowed it down to the Ethon gem, which wasn’t too much help since it is basically just an interface to the CURL library. But when I did a plain ol’ CURL PUT to the same endpoint, everything worked just fine.

I tried using tcpump to diagnose the problem, suspecting that it lay in a difference in the way commandline CURL and the Typhoeus use were encoding the request, but my tcpdump-fu was weak: I figured the traffic I was interested would not be going over any of the usual interfaces listed by `ifconfig -a` but didn’t know which one it was, and it was around this time that I temporarily gave up, figuring some quick Googling would solve these problems when I got back online.

It turns out that this traffic goes over the “loopback” interface, which is in fact listed in the output of `ifconfig -a` as the first one on OSX: `lo0`, so I could have run `tcpdump -i lo0`. But now that I had the Internets back, I installed Wireshark, which for some reason I didn’t already have, and fired it up, setting a display filter for http traffic only.

After making a command line CURL PUT request, which was speedy, and a slow ruby Typhoeus request from the rails console, I had a look at the relevant captures in Wireshark and quickly noticed this difference:

You’ll notice that the second, slow one is composed of two separate TCP frames.

Examining the headers further, I noticed this:

Turns out that despite all the times I’ve looked at the list of HTTP codes I’ve always just skipped over the 1XXs. Wikipedia has a more detailed breakdown, but the gist is that clients may send an `Expect: Continue-100` header so that the server can check if a request is appropriate based solely on the headers, which can be useful for large requests. It turns out that the libcurl will send this header by default, though you can disable it with the `-H` command line flag, which is what my CURL commands were doing by specifying various content-type and authorization headers.

It also turns out that CURL will just send the second request anyway after one second if it doesn’t receive a 100 code, which explains why my API client was consistently slow but not broken.

Now, how to work around this. Some Googling revealed some people monkeypatching the Rails default Webrick server, which I didn’t want to do. I could also have hacked Typhoeus or Ethon, but that didn’t seem like the way to go either. The best way, it seemed, was to take advantage of this feature and have my application return a 100 code if the user was indeed authorized to make the request. (Alternatively, you can just specify an Expect header with an empty value, rather than “100-continue”.)

The sensible place to put this code, it seemed to me, was in Rack middleware. (It should be noted at this point that this does not seem to work with Webrick or Thin.) A barebones implementation could look like so:

class Preauthenticator
  def initialize app
    @app = app
  end

  def call env
    if env["HTTP_EXPECT"] =~ /100-continue/
      return [100, {}, [""]]
    else
      @app.call(env)
    end
  end
end

(For an example from the Unicorn source code, see here.)

The only tricky part here was figuring out how exactly to get Rails to use it. Middleware generally goes in your `config/application.rb` file (make sure to require your file with the middleware class in it at the top of that file), but including this like so: `config.middleware.use Preauthenticator` resulted in rack complaining about a “app error: deadlock; recursive locking (ThreadError)”. Running `rake middleware` will list all the middleware used by your Rails app; it seemed to me that my Preauthenticator should run before any of this, so I tried `config.middleware.insert_before Rack::Sendfile, Preauthenticator`, Rack::Sendfile being the first middleware listed, and that worked like a charm.

Now my HTTP PUTs were as speedy as the rest of my requests.

As of Dec 14, 2014, the middleware here will not work with Webrick 1.3.1, Thin 1.6.3, or Puma 2.10.0 ruby servers.

TL;DR

Typhoeus and CURL will make some PUTs and POSTs slow by sending an “Expect: 100-Continue” header. Add rack middleware that returns this header to make your PUTs and POSTs faster.