Jul 14 2012

Once upon a time, I was building my First Serious Rails App. I was drawn to Rails in the first place because of automated testing and ActiveRecord; I felt the pain of not using an ORM and spending about a week on every deploy making sure that things were still okay in production. So of course, I tried to write a pretty reasonable suite of tests for the app.

To gloss over some details to protect the innocent, this app was a marketplace: some people owned Things, and some people wanted to place Orders. Only certain Things could fulfill an Order, so of course, there was also a ThingType table that handled different types of Things. Of course, some Types came in multiple Sizes, so there also needed to be a Size Table and a ThingTypeSize table so that a User could own a Thing of a certain Type and a certain Size.

Stating that creating my objects for tests was difficult would be an understatement.

Then I read a blog post about FactoryGirl. Holy crap! This would basically save me. With one simple Factory(:thing) I could get it to automatically build a valid list of all that other crap that I needed!

So of course, I had to write my spec for a thing:

describe Order do
  it "generates quotes only from Things that are of the right size" do
    order = Factory(:order)
    thing = Factory(:thing, :size => order.size)
    thing = Factory(:thing)
    order.quote!
    order.quote.thing.should == thing
  end
end

This test worked. It also generated around 15 objects, saved them in the database, and queried them back out. I don’t have the code running anymore, but it was like 30-40 queries, and took a second or two to run.

That was one test. I was trying to test a lot, even though I wasn’t good at test first yet, so my suite got to be pretty big. Also, sometimes my factories weren’t the best, so I’d spend a day wondering why certain things would start failing. Turns out I’d defined them slightly wrong, validations started to fail, etc.

How did we get here?

This story is one of Ruby groupthink gone awry, basically. Of course, we know that fixtures get complicated. They get complicated because we have these crazy ActiveRecord models, don’t use plain Ruby classes when appropriate, and validations make us make extra objects just to get tests to pass. Then fixtures get out of date. So let’s introduce a pattern!

Of course, since we know that Factories are really useful when things get complicated, let’s make sure to use them from the start, so we don’t have to worry about them later. Everyone started doing this. Here’s how new Rails apps get started:

steve at thoth in ~/tmp
$ rails new my_app
      create  
      create  README.rdoc
<snip>
      create  vendor/plugins/.gitkeep
         run  bundle install
Fetching gem metadata from https://rubygems.org/.........
<snip>
Using uglifier (1.2.6) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

steve at thoth in ~/tmp
$ cd my_app 

steve at thoth in ~/tmp/my_app
$ cat >> Gemfile
gem "rspec-rails"
gem "factory_girl_rails"
^D

steve at thoth in ~/tmp/my_app
$ bundle
Fetching gem metadata from https://rubygems.org/........
<snip>
steve at thoth in ~/tmp/my_app
steve at thoth in ~/tmp/my_app
$ rails g rspec:install                                                   
      create  .rspec
       exist  spec
      create  spec/spec_helper.rb

$ rails g resource foo
      invoke  active_record
      create    db/migrate/20120714180554_create_foos.rb
      create    app/models/foo.rb
      invoke    rspec
      create      spec/models/foo_spec.rb
      invoke      factory_girl
      create        spec/factories/foos.rb
      invoke  controller
      create    app/controllers/foos_controller.rb
      invoke    erb
      create      app/views/foos
      invoke    rspec
      create      spec/controllers/foos_controller_spec.rb
      invoke    helper
      create      app/helpers/foos_helper.rb
      invoke      rspec
      create        spec/helpers/foos_helper_spec.rb
      invoke    assets
      invoke      coffee
      create        app/assets/javascripts/foos.js.coffee
      invoke      scss
      create        app/assets/stylesheets/foos.css.scss
      invoke  resource_route
       route    resources :foos

steve at thoth in ~/tmp/my_app
$ rake db:migrate                                                         
==  CreateFoos: migrating =====================================================
-- create_table(:foos)
   -> 0.0065s
==  CreateFoos: migrated (0.0066s) ============================================

$ cat > spec/models/foo_spec.rb
require 'spec_helper'

describe Foo do
  it "does something" do
    foo = Factory(:foo)
    foo.something!
  end
end
^D

$ bundle exec rake spec       
/Users/steve/.rvm/rubies/ruby-1.9.3-p194/bin/ruby -S rspec ./spec/controllers/foos_controller_spec.rb ./spec/helpers/foos_helper_spec.rb ./spec/models/foo_spec.rb
*DEPRECATION WARNING: Factory(:name) is deprecated; use FactoryGirl.create(:name) instead. (called from block (2 levels) in <top (required)> at /Users/steve/tmp/my_app/spec/models/foo_spec.rb:5)
F

Pending:
  FoosHelper add some examples to (or delete) /Users/steve/tmp/my_app/spec/helpers/foos_helper_spec.rb
    # No reason given
    # ./spec/helpers/foos_helper_spec.rb:14

Failures:

  1) Foo does something
     Failure/Error: foo.something!
     NoMethodError:
       undefined method `something!' for #<Foo:0x007f82c70c07a0>
     # ./spec/models/foo_spec.rb:6:in `block (2 levels) in <top (required)>'

Finished in 0.01879 seconds
2 examples, 1 failure, 1 pending

Failed examples:

rspec ./spec/models/foo_spec.rb:4 # Foo does something

Randomized with seed 27300

rake aborted!
/Users/steve/.rvm/rubies/ruby-1.9.3-p194/bin/ruby -S rspec ./spec/controllers/foos_controller_spec.rb ./spec/helpers/foos_helper_spec.rb ./spec/models/foo_spec.rb failed

Tasks: TOP => spec
(See full trace by running task with --trace)

Woo! Failing test! Super easy. But what about that test time?

Finished in 0.01879 seconds

Now, test time isn’t everything, but a hundredth of a second. Once we hit a hundred tests, we’ll be taking almost two full seconds to run our tests.

What if we just Foo.new.something!?

Finished in 0.00862 seconds

A whole hundredth of a second faster. A hundred tests now take one second rather than two.

Of course, once you add more complicated stuff to your factories, your test time goes up. Add a validation that requires an associated model? Now that test runs twice as slow. You didn’t change the test at all! But it got more expensive.

Now, a few years in, we have these massive, gross, huge, long-running test suites.

What to do?

Now, I don’t think that test times are the end-all-be-all of everything. I really enjoyed this post that floated through the web the other day. I think that the ‘fast tests movement’ or whatever (which I am/was a part of) was a branding mistake. The real point wasn’t about fast tests. Fast tests are really nice! But that’s not a strong enough argument alone.

The point is that we forgot what testing is supposed to help in the first place.

Back to basics: TDD

A big feature of tests is to give you feedback on your code. Tests and code have a symbiotic relationship. Your tests inform your code. If a test is complicated, your code is complicated. Ultimately, because tests are a client of your code, you can see how easy or hard your code’s interfaces are.

So, we have a pain in our interface: our objects need several other ones to exist properly. How do we fix that pain?

The answer that the Rails community has taken for the last few years is ‘sweep it under the rug with factories!’ And that’s why we’re in the state we’re in. One of the reasons that this happened was that FactoryGirl is a pretty damn good implementation of Factories. I do use FactoryGirl (or sometimes Fabrication) in my request specs; once you’re spinning up the whole stack, factories can be really useful. But they’re not useful for actual unit tests. Which I guess is another way to state what I’m saying: we have abandoned unit tests, and now we’re paying the price.

So that’s what it really boils down to: the convenience of factories has set Rails testing strategies and software design back two years.

You live and learn.