Space Vatican

Quite a few people have written about the performance failings of Net::HTTP, but until recently, to be honest, I never really cared a lot. Most of my http request needs have been fairly meagre, often not much more than hitting a url and checking the result code.

I've been playing with couchdb recently, and so my app does a fair amount of http requests. I've been using RelaxDB which uses net/http, so Net::HTTP's performance has started to matter.

Net::HTTP is not the only game in town. I spent some time recently playing with rfuzz, eventmachine and taf2-curb and came to largely the same conclusion as Paul Dix.

Leaning on a mature library such as libcurl gives taf2-curb a huge advantage. While eventmachine was on par speed wise, neither of the 2 http clients it includes are a complete implementation of the HTTP protocol. For example HttpClient will tell the remote server that it speaks HTTP/1.1, yet it does not support chunked encoding (mandatory part of the spec). HttpClient2 does understand chunked encoding, but doesn't let you set headers or a body to the request. Fine for just pinging a url, but not up to the task of working with couchdb. Something to do with couchdb's chunk encoded also seemed to confuse rfuzz.

taf2-curb does the job very nicely. On my dumb benchmark, 1000 requests for a static html page hosted on the same machine (ie we're pretty much only testing overhead) the numbers are:

Benchmark.bmbm(5) do |x|
  x.report 'net/http' do
    u = URI.parse('http://docs.local/')
    1000.times {Net::HTTP.get u}
  end

  x.report 'curb' do
    1000.times do
      c = Curl::Easy.new 'http://docs.local/'
      c.perform
    end
  end
end

               user     system      total        real
net/http   0.560000   0.270000   0.830000 (  1.065960)
curb       0.310000   0.170000   0.480000 (  0.696188)

On the other extreme, these numbers corresponds to ~1 meg of data pulled from couchdb (benchmark code the same apart from the urls, and I did 100 iterations rather than 1000).

               user     system      total        real
net/http  17.400000   8.900000   2.630000 (  32.067821)
curb       0.700000   1.300000   2.000000 (  29.586022)

curb comes up squarely on top. Another thing of note during this test is cpu usage (as you might expect from the difference in user time). With Net::HTTP the ruby process running this was taking up 60-70% (on a 2.4GHz core duo), with curb it used around 5% of cpu.

The commit to switch RelaxDB from net/http to taf2-curb is here for those interested - really very straightforward stuff. There may well be more to be had by fiddling with libcurl options, I haven't tried yet.