Space Vatican - Home

Slides from my RoR exchange talk

2009-12-04T11:08:00Z

Get them here. I had a great time there - well worth going!

A sneaky session gotcha in rails 2.3

2009-11-21T09:07:00Z

A colleague and I recently updated an app to rails 2.3.4 and found ourselves with a slightly sticky performance problem - slow app response times, with time disappearing seemingly nowhere. New Relic also showed that requests were queueing inside mongrels which wasn't shouldn't have been a problem since there were plenty of mongrels for the traffic being received at the time

Our friendly Engineyard support engineer pointed us to a ticket on the Rails lighthouse describing a rather similar situation encountered during the RC phase of rails 2.3: Active record based sessions causing trouble because they weren't returning their connection to the connection pool [1]. Connection pooling isn't new in 2.3 but the move to rack changed the order in which things like sessions, query caching, connection pool are put up.

This ticket was mark as resolved, however looking at the fix it became clear why we were still experiencing the problem:

if ActionController::Base.session_store == ActiveRecord::SessionStore
  configuration.middleware.insert_before :"ActiveRecord::SessionStore", ActiveRecord::ConnectionAdapters::ConnectionManagement
   configuration.middleware.insert_before :"ActiveRecord::SessionStore", ActiveRecord::QueryCache
else
  configuration.middleware.use ActiveRecord::ConnectionAdapters::ConnectionManagement
  configuration.middleware.use ActiveRecord::QueryCache
end

(Source here)

Since our session store isn't ActiveRecord::SessionStore, the code to put the database related middleware in the right place in the chain isn't executed. I'm not entirely sure what to do about this - the rails initializer cannot be expected to know the details of what the session store is doing (unless part of the session store interface was a uses_active_record? flag) but as it is this very sneaky and would have taken a while to find if we hadn't been pointed in the right direction.

[1] As of Rails 2.2, Active Record has a concept of a connection pool: a given rails instance won't use more than a certain number of connections (5 by default) even if that rails instance was a jruby based instance with hundreds or threads. If you need a connection from the pool and they are all in use then you have to wait until one becomes available. Normally you are blissfully unaware of this - rails marks your connection as no longer used when your action has finished processing.

Error launching remote program: security policy error

2009-11-07T17:44:00Z

I created a new iPhone project today and got a rather vague error message from Xcode when I tried to run it on a device:

Error launching remote program: security policy error

After some digging it turns out that the problem was that I had two provisioning profiles on the device:

a wildcarded one (which had expired) that I use for quick hacks that don't warrant going through the hassle of setting up new provision profiles
one created specifically for this application

The wildcarded profile had expired, and for some reason the iPhone was trying to launch the app with the expired profile even though it had a valid profile that it could have used. Deleting the expired profile from the device did the trick. If only that error message had been a bit more explicit ...

My first Iphone app

2009-10-12T10:51:00Z

After much toiling kgb's first iPhone app is on sale! Get it now. Only in the US unfortunately.

For the record, we submitted two versions of the app: an earlier version that we didn't expect to release (we wanted to see what the approval process was like and if apple was OK with the basic idea of our app) and the one that is on sale today. Both were approved in about 10 days - no approval horror stories here!

Unit testing Core Data iphone apps

2009-09-10T14:34:00Z

This is probably obvious, but in the interest of saving someone else the few minutes I spent scratching my head on this one...

Your app probably has some code that looks like

managedObjectModel = [[NSManagedObjectModel mergedModelFromBundles:nil] retain]

If you pass nil then the main bundle is search. When your an iphone app, that is the app itself - no surprises there. But when you are a unit test bundle, you are no longer the main bundle (the test rig application is). You need to tell CoreData to look inside your unit test bundle, with something along the lines of

managedObjectModel = [[NSManagedObjectModel mergedModelFromBundles:
[NSArray arrayWithObject:[NSBundle bundleWithIdentifier:@"com.yourcompany.unittests"]]] retain];

For this to work you also need your data model to be present in your unit test bundle. For some reason, unlike your .m files the inspector doesn't show you a list of targets the model should be included in - you need to drag the data model file into the compile source build phase of your unit test target.

ruby openssl and client side certificates

2009-07-11T16:48:00Z

I recently needed to deal with ssl connection using client side certificates. The ruby openssl bindings are fairly impenetrable, here's what worked for me (at least in part as a note for myself in the future)

ctx = OpenSSL::SSL::SSLContext.new
ctx.cert = OpenSSL::X509::Certificate.new("mycert.cer")
ctx.key = OpenSSL::PKey::RSA.new("mykey.pem")
ssl = OpenSSL::SSL::SSLSocket.new(sock, ctx)
ssl.connect

If the key you've got is a .p12 file (which is what the key chain utility on the mac exports) then you'll need to convert it like so

openssl pkcs12 -in key.p12  -nocerts -nodes -out key.pem

Fun with ruby http clients

2009-04-13T09:11:00Z

Quite a few people have written about the performance failings of Net::HTTP, but until recently, to be honest, I never really cared a lot. Most of my http request needs have been fairly meagre, often not much more than hitting a url and checking the result code.

I've been playing with couchdb recently, and so my app does a fair amount of http requests. I've been using RelaxDB which uses net/http, so Net::HTTP's performance has started to matter.

Net::HTTP is not the only game in town. I spent some time recently playing with rfuzz, eventmachine and taf2-curb and came to largely the same conclusion as Paul Dix.

Leaning on a mature library such as libcurl gives taf2-curb a huge advantage. While eventmachine was on par speed wise, neither of the 2 http clients it includes are a complete implementation of the HTTP protocol. For example HttpClient will tell the remote server that it speaks HTTP/1.1, yet it does not support chunked encoding (mandatory part of the spec). HttpClient2 does understand chunked encoding, but doesn't let you set headers or a body to the request. Fine for just pinging a url, but not up to the task of working with couchdb. Something to do with couchdb's chunk encoded also seemed to confuse rfuzz.

taf2-curb does the job very nicely. On my dumb benchmark, 1000 requests for a static html page hosted on the same machine (ie we're pretty much only testing overhead) the numbers are:

Benchmark.bmbm(5) do |x|
  x.report 'net/http' do
    u = URI.parse('http://docs.local/')
    1000.times {Net::HTTP.get u}
  end

  x.report 'curb' do
    1000.times do
      c = Curl::Easy.new 'http://docs.local/'
      c.perform
    end
  end
end

               user     system      total        real
net/http   0.560000   0.270000   0.830000 (  1.065960)
curb       0.310000   0.170000   0.480000 (  0.696188)

On the other extreme, these numbers corresponds to ~1 meg of data pulled from couchdb (benchmark code the same apart from the urls, and I did 100 iterations rather than 1000).

               user     system      total        real
net/http  17.400000   8.900000   2.630000 (  32.067821)
curb       0.700000   1.300000   2.000000 (  29.586022)

curb comes up squarely on top. Another thing of note during this test is cpu usage (as you might expect from the difference in user time). With Net::HTTP the ruby process running this was taking up 60-70% (on a 2.4GHz core duo), with curb it used around 5% of cpu.

The commit to switch RelaxDB from net/http to taf2-curb is here for those interested - really very straightforward stuff. There may well be more to be had by fiddling with libcurl options, I haven't tried yet.

A Teeny Tiny Plugin for working with others

2009-04-03T18:01:00Z

If you work with designers or getting to grip with a new codebase you may find yourself frequently answering the question 'What template generates this bit of html'. This smidgen of code adds comments to your html with the path to the templates used. For example the template

Hello
 <%= render :partial => 'some_partial' -%>

might generate

<!-- TEMPLATE: views/dummy/index.html.erb -->
Hello
<!-- TEMPLATE: views/dummy/_some_partial.erb -->
Here is a partial
<!-- ENDTEMPLATE: views/dummy/_some_partial.erb -->
<!-- ENDTEMPLATE: views/dummy/index.html.erb -->

You can grab the code from github. Original idea from this thread on rails-talk

Make your tests run a little quicker for free

2009-01-25T16:30:00Z

It's a good thing when tests run quickly. You get a shorter feedback cycle, you're unlikely to move on to something else while your tests are running. Another good thing is ruby-debug: a fast debugger for ruby, worth its weight in gold when you need it. As of Rails 2.1 ruby-debug is always loaded and active during tests.

Having the debugger there does have some overhead. Not much, but it's still there. You're incurring that overhead whether your tests need it or not. On your CI server, it's pure wasted time for example. Even running locally it's frequently unecessary - most of the time when I run the tests I'm just running the tests, not using the debugger to try to understand a test failure.

So, without further ado I present this:

unless ENV['DEBUGGER']
  Debugger.stop if defined?(Debugger)
end

Stick it in your test_helper.rb, just below where it says

require 'test_help'

Run your tests again, and hey presto, faster! It does very from app to app, on some of the smaller apps I have the difference is pretty marginal, on the bigger apps I've got gains of up to 20-30%. I expect that on the smaller ones loading the Rails environment, setting up fixtures etc... dominates actual test runtime. And the day you do need to run the tests under the debugger it's as easy as

rake DEBUGGER=true

if you're using rake to run your tests or

DEBUGGER=true ruby -Itest test/unit/some_test.rb

if you're running an individual test (this might change according to what shell you use). In fact most of the time you won't even need to do this, as calling debugger calls Debugger.start for you.

When cache_classes gets you down

2008-12-28T14:28:00Z

As I've mentioned before, in Rails 2.2 if config.cache_classes is true (which it is in production) then all of your models, controllers etc are loaded up as part of application initialization. This is great if you're firing up a server that should actually be handling requests as it avoids all sorts of nasty race conditions to do with requiring files and defining classes in ruby. It's not so great if you're running a small script, cron job or something like a migration.

The reasons are two-fold. First off, it makes startup slower. This is irrelevant when starting up a web server because it is just getting out of the way stuff you would do anyway but it is time wasted if you've just got a single threaded script that doesn't care about dependency race conditions (and probably only touches one or two models anyway), especially if the script only takes a second or two to run once the environment is loaded. Secondly it can break stuff if models or controllers try and look at the database when the class is loaded. One example of this is ActiveScaffold. For those of you not familiar with it, ActiveScaffold helps you create adminy CRUD style interfaces. A very basic controller might look like this:

class ProductsController < ApplicationController
  active_scaffold
end

When the controller is loaded ActiveScaffold sees that it's working on the ProductsController, infers that the corresponding model is Product and goes off to find out what columns that model has. Imagine you've just added that model and controller and it's time to deploy on your production machine. At this point the products table doesn't exist (since the migration hasn't run yet). Since you're running in your production environment when the rake task causes rails to be loaded it will load your application's classes, including ProductsController which will try and introspect the products table. Fail.

There are two workarounds I can think of. One is a separate rails environment: define an environment with cache_classes set to false but that still uses the production database. Instead of running your migrations in the production environment run them in the production_without_cache_classes environment. This works but can be a bit annoying, especially for adhoc scripts and stuff like that and is even more of a pain if you have multiple productiony environments (eg a staging environment).

Another is to edit your production.rb so that it reads

if defined?(DONT_CACHE_CLASSES) && DONT_CACHE_CLASSES
  config.cache_classes = false
else
  config.cache_classes = true
end

Then for any script which you want to run without class caching turned on stick DONT_CACHE_CLASSES=true at the top (before you require config/environment). If you want to extend this to your rake files then edit the Rakefile at the top level of the app. This feels slightly neater to me as I don't have to remember to fiddle with the environment when running the script.

Another variation upon this is to use an environment variable instead of a constant which would allow you to do things such as rake db:migrate CACHE_CLASSES=false or similar (although of the default tasks I can't of one you'd run in production where you would want class caching to be on).

This is far from beautiful but appears to get the job done for now.

Dates, params and you

2008-12-03T13:26:00Z

A not particularly nice area of Rails are the date and time helpers. 3 popups just isn't a very nice bit of user interface. It's a lot of clicks when you want to change dates and most people can't reason in their head about just the date. It's far easier to pick a date from a calendar type view. Still the helpers rails provides are fine for that quick and dirty date input.

Based on the questions on the mailing list about this, the thing that trips people up is that, unlike other attributes you might typically have, dates and times are not representable by a single input control. Instead you have several, one for each component (year, month, day etc...). So in particular, there is no single value in your params hash with your date or time. Exactly what is in your params hash depends on whether your using select_date or date_select (if you're entering a datetime, select_datetime or datetime_select).

These are to each other as text_field_tag is to text_field: date_select is expecting to hook up to an attribute of an instance variable (or if you use form_for or fields_for an attribute of the corresponding object) whereas select_date isn't. However unlike the other pairs of functions like textfield_tag/text_field, select/select_tag these two send very different parameters through to your controller.

select_date is perhaps the easiest to understand. It will result in a hash (by defaults it is named "date", but you can override this with the :prefix option) with keys like year, month, day. You can then put those together to get an instance of Date or Time. For example the following in your view

<%= select_date Date::today, :prefix => 'start' %>

will result in a params hash like this:

{:start => {:year => '2008', :month => '11', :day => '22'}}

As I said, there is nothing in the params hash that is the actual value. You have to put it together yourself, for example

my_date = Date::civil(params[:start][:year].to_i, params[:start][:month].to_i, 
                      params[:start][:day].to_i)

A bit more work than you average parameter, but there's nothing mysterious going on here. Under the hood, select_date is also quite boring: it's just calling select_year, select_month, select_day with appropriate options and concatenating the result. A consequence of that is that if you want some odd combination (eg just months and seconds) you can just do that concatenation work yourself. One interesting thing about those subhelpers is that the first parameter you give them can be one of two things:

an integer in which case the corresponding day/month/year is displayed (eg 3 for March)
something like a Date or DateTime in which case the relevant date component is extracted from it

date_select is where the fun is. Here the expectation is that there is a model object we will want to update and we want to be able to do

my_object.update_attributes params[:my_object]

However update_attributes just wants to set attributes. If you pass it {'foo' => 'bar'} it will try and call the method foo= passing bar as a parameter. For a date input that is made up of these multiple parameters this is clearly a problem. What solves this is something called multiparameter assignment. If there are parameters whose name is in a certain format, then instead of just trying to call the appropriate accessor Rails will gather the related parameters, feed them through a transformation function (for example Time.mktime or Date::new[1]) and then set the appropriate attribute.

The format used is as follows: all the related parameters start with the name of the attribute which lets Rails know they are related. Next Rails needs to know in what order to pass them to the transformation function and whether a typecast is needed. If your view contained

<%= date_select 'product', 'release_date' %>

Then your parameters hash would look like

{:product => 
        {'release_date(1i)' => '2008', 'release_date(2i)' => '11', 'release_date(3i)' => '22'}}

Rails can look at this and see that this is to do with the release_date attribute. It's a date column, so rails knows to use Date::civil. The suffixes tell rails that 2008 is the first parameter to Date::civil and is an integer, that 11 is the second parameter and so on. Rails constructs the value using Date::civil(2008,11,22) and assigns that to release_date.

If you don't intend to pass the parameters to update_attributes (or other functions with that syntax such as the new or create methods on an ActiveRecord class) there's not a lot of point in putting up with the scary parameter names althouh you can of course construct the date yourself with

Date::civil(params[:product]['release_date(1i)'].to_i,
 params[:product]['release_date(2i)'].to_i, 
params[:product]['release_date(3i)'].to_i)

You might as well just use select_date and have readable parameter names though.

So, to sum up use date_select or datetime_select when creating/updating ActiveRecord objects but select_date or select_datetime for just a general purpose date input. As a closing tip, with select_datetime you can use the :use_hidden option in which case hidden form inputs are generated instead of select boxes.[2]

[1] There's a bit more to this. For one the range of times representable by a Time object is limited on most platforms (since it's commonly a 32 bit number of seconds since an epoch). Rails has some conversion code that will try and create an instance of Time but if necessary will fall back and create a DateTime object. Secondly there's some cleverness to do with interpreting the user's input with respect to the correct time zone.

[2] This is (I think) a slight misuse. The intent of the use hidden is that it is the mechanism by which the :discard_day and so on work

with_options for fun and profit

2008-11-25T20:30:00Z

Active Support has a nifty little helper that can cut down on repetition. A lot of things in Rails like validations, associations, named_scopes, routes etc... take a hash of options as their final parameter. There are times where you use many of these with some common options, for example

class Customer < ActiveRecord::Base
  validates_presence_of :phone_number, :if => :extended_signup
  validates_presence_of :job_title, :if => :extended_signup
  validates_presence_of :job_industry, :if => :extended_signup
end

or maybe you have a bunch of associations which all share an association proxy extension module and some settings

class Customer < ActiveRecord::Base
  has_many :foos, :extend =>MyModule, :order => "updated_at desc", :conditions => {:active => true}
  has_many :bars, :extend =>MyModule, :order => "updated_at desc"
  has_many :things, :extend =>MyModule, :order => "updated_at desc"
end

Enter with_options!

class Customer < ActiveRecord::Base
  with_options :extend => MyModule, :order => "updated_at desc" do |options|
    options.has_many :foos, :conditions => {:active => true}
    options.has_many :bars
    options.has_many :things
  end
end

Any time you've got a bunch of method calls taking some common options, with_options can help.

So how does it work on the inside? All the with_options method actually does is yield a special object to its block - all the craftiness is in that object. What we want that object to do is forward method calls to the object we're really interested in (in this case the Customer class) adding the options before it does so.

As with many such proxy objects we undefine just about every method and just implement method_missing. The implementation of method_missing inspects the arguments and merges any options present with the common set defined by the call to with_options (so individual methods can take extra options or override the common ones) before passing them onto the "real" object.

Originally a limitation was that the last argument just had to be a hash, so for example if you had a procedural named_scope then with_options couldn't work. Luckily a recent commit rectifies this: if the thing you're trying to merge with is a proc, then with_options will replace it with a new proc that merges the common options with the result of the call to the original proc. If you're not on edge you'll have to wait for 2.3 in order to get this.

While both the examples I gave showed using with_options on what is essentially model class configuration it is by no means limited to that. You could use it for that sort of configuration on your own classes or just inside a regular method - anytime you are making several method calls on the same object with a hash of options at the end.

environment.rb and requiring dependencies

2008-11-21T22:39:00Z

In the before time, the bottom of most of my apps' environment.rb was an unholy mess. Inflector rules, requiring of various libraries or gems, various bits of app specific configuration etc... all jumbled together. Rails 2.0 introduced initializers: any file in config/initializers is run at an appropriate time during the initialisation process. You get to split that mess into a handful of well organised, single purposed little files (and rails 2.1 simplified the case of requiring gems with the config.gems mechanism).

You might still have a few stragglers though, that one require that you didn't bother moving into an initializer because it hardly seemed worth creating a whole file just for that one line. With the imminent release of Rails 2.2 it's high time you made that change.

Living Thread Dangerously

Unless you've been living under an internet-proof rock you've probably heard about Rails' new threadsafeness. There's a bunch of hard work across the framework that's gone into making this possible, but one particular area is to do with loading code. Ruby's require mechanism isn't threadsafe (or as it has been put to me, it's thread dangerous) nor is the automatic loading stuff Rails' uses. For example say two threads both hit a constant called Foo that has yet to be loaded. Thread 1 starts loading foo.rb and gets as far as

class Foo < ActiveRecord

At this point Thread 2 hits Foo. However at this point the constant Foo now exists and so Thread 2 doesn't load foo.rb. However since Thread 1 hasn't yet processed the rest of foo.rb the Foo class will be missing all its instance methods, validations etc... If both threads end up loading foo.rb at the same time then weird things can happen like validations being added twice and so on. It can also cause the dependencies system to spuriously claim it couldn't find a constant. It's a small world of pain you don't want to get involved in. Making require threadsafe is fundamentally hard (and is something the ruby-core and jruby folks have been worrying about).

What Rails 2.2 does in production mode is load all of your models, controllers and so on as part of the initialization process instead of loading them as they are needed. No more loads from different threads when your app is actually running, no more pain.

The Bad Thing

So, how does this connect with the statement I made above about moving things into initializers? Your average environment.rb file looks a little like this

#set some constants like RAILS_GEM_VERSION
require File.join(File.dirname(__FILE__), 'boot')
Rails::Initializer.run do |config|
  #set some config settings
end
#if you're old school, app configuration here
require 'some_dependency'

The bulk of initialization happens when you call run. This yields to the block to allow you to set the various settings (and also reads the appropriate environment file and so on) but the key thing is that by the time that function has returned, all of the initialization has happened.

In particular, Rails will try to load all of your application classes before the stuff at the bottom of environment.rb has been executed. If a model depends on some_dependency.rb being loaded (for example if that file added a validation that it uses) then your app will die before it even finished initialising.

If however you're a good person and move these things into initializers (i.e. files in config/initializers) then they will run at an appropriate time in the boot process (i.e. before Rails loads up all your application classes) and you won't get an unpleasant surprise when you try and deploy your app.

ASAP

2008-11-20T14:24:00Z

Things most likely to make me not reply to a message on a mailing list:

This is urgent

Plz reply ASAP

First, foremost and [0]

2008-11-15T17:58:00Z

This doesn't work (the something field will not be updated):

post.comments.first.something = true
post.comments.first.save

but this does

post.comments[0].something = true
post.comments[0].save

as does

post.comments.each {|c| puts c.id}
post.comments.first.something = true
post.comments.first.save

Both of these would have worked in rails 2.0.x and previous versions. So what changed?

Quacks like a duck but breathes fire

As you may know, post.comments looks an awful lot like an array but isn't an array. It's an association proxy. It has methods defined on it for things like finding objects from the database, the count method that does an sql count and things like that. When you ask it to do something that can only really be done by having the ruby objects in memory it will load the objects from the database into an actual ruby array and pass methods onto that (this all happens via method_missing). So far business as usual, rails has been like this for a long long time. In particular were you to call the first or the last methods on an association then the array would be loaded and first would be called on that array.

This sort of depends on your problem domain, but a lot of the time loading the entire array just to look at the first or last element of it is wasteful. You've always been able to do some_association.find :first (and as of 2.1 some_association.find :last) but that flows a little less easily off the tongue and of course doesn't play nicely if you pass your association to some code that thinks it's just working with an array. So a few months ago changes were made to make first and last load just that one item from the database[1]. Of course if the target array is already loaded then it just returns the first item from that array.

At the end of the day what that ends up meaning is that in the first example I gave, each call to post.comments.first returns a different object, ie the one that we call save on is not the same as the one we made the change to. The second and third examples are ok purely because they force the array to be loaded which in turn means that calls to first no longer hit the database in that way[2].

Of course if you're doing things right your unit tests would catch this sort of thing, but it's still likely to leave you scratching your head a little (I certainly recall spending a few minutes looking at code very similar to the first example and wondering why it no longer worked). Slightly more subtle are performance problems, for example if you were iterating over various attributes then you'd be hitting the database each time to load somethings.first.

I'm not sure what to think about this sort of thing. There is a perfectly sound rationale for doing this but it introduces little ifs, buts and maybes into the illusion that association proxies behave like Arrays. As far as performance goes the implications vary. For big associations it can be a huge win, other times loading 1 object instead of 3 will make little to no difference. In other places I do genuinely want to load the whole array but I'd rather write first than [0] if I'm accessing the first element. Maybe the example I gave is a little artificial, maybe not, but at the end of the day, first no longer being a synonym for [0] is a habit that is hard to break.

[1] I'm simplifying things quite considerably here - there are a number of edge cases which that code has to tread around quite carefully, including unsaved parent objects, unsaved children objects, custom finder sql etc... [2] While I've concentrated on first, everything I've said here applies to last too. In a way it's slightly worse in that (at least for me) the difference in comfort between writing last and [-1] is greater than the difference between first and [0]