The Devver Blog

Our very own Dan Mayer will be at Lone Star Ruby Conf Friday and Saturday! Be sure to talk to Dan about Devver, testing, or anything else!

Like most Rubyists, I write unit tests to verify the non-trivial parts of my code. I also try to use mocks and stubs to stub out interactions with systems external to my code, like network services.

For the most part, this works fine. But I've always struggled to find a good way to test interaction with the filesystem (which can often be non-trivial and therefore should be tested). On the one hand, the filesystem could be considered "external" and mocked out. But on the other hand, the filesystem is accessible when the tests run. In this way, the filesystem is sort of like a local database - it could be mocked out, but it doesn't have to be, and there are tradeoffs to both approaches.

Over the past year or so, I've tried out a few approaches for testing interactions with the filesystem, each of which I'll explain below. Since none of the approaches met my needs, Avdi and I built a new testing library, which I'll introduce below.

Mocking the file system.

Sometimes, it is simplest to just mock the interaction with the filesystem. This works well for single calls to methods like File#read or File#exist? (these examples use Mocha):

File.stubs(:read).returns("file contents")
File.stubs(:exist?).returns(true)

However, this approach breaks down when you want to test more complex code, which, of course, is the code you're more likely to want to test thoroughly. For instance, imagine trying to set up mocks/stubs for the following method (which atomically rewrites the contents of a file):

require 'tempfile'

class Rewriter

  def rewrite_file!(target_path)
    backup_path = target_path + '.bak'
    FileUtils.mv(target_path, backup_path)
    Tempfile.open(File.basename(target_path)) do |outfile|
      File.open(backup_path) do |infile|
        infile.each_line do |line|
          outfile.write(yield(line))
        end
      end
      outfile.close
      FileUtils.cp(outfile.path, target_path)
    end
  rescue Exception
    if File.exist?(backup_path)
      FileUtils.mv(backup_path, target_path)
    end
    raise
  end

end

Now imagine setting up those same mocks/stubs for each of the five or so tests you'd want to test that method. It gets messy.

Even more importantly, mocking/stubbing out methods ties your tests to a specific implementation. For instance, if you use the above stub (File.stubs(:read).returns("file contents")) in your test and then refactor your implementation to use, say, File.readlines, you'll have to update your tests. No good.

MockFS

MockFS is a library that mocks out the entire filesystem. It allows you write test code like this:

require 'test/unit'
require 'mockfs'

class TestMoveLog < Test::Unit::TestCase

  def test_move_log
    # Set MockFS to use the mock file system
    MockFS.mock = true

    # Fill certain directories
    MockFS.fill_path '/var/log/httpd/'
    MockFS.fill_path '/home/francis/logs/'

    # Create the access log
    MockFS.file.open( '/var/log/httpd/access_log', File::CREAT ) do |f|
      f.puts "line 1 of the access log"
    end

    # Run the method under test
    move_log

    # Test that it was moved, along with its contents
    assert( MockFS.file.exist?( '/home/francis/logs/access_log' ) )
    assert( !MockFS.file.exist?( '/var/log/httpd/access_log' ) )
    contents = MockFS.file.open( '/home/francis/logs/access_log' ) do |f|
      f.gets( nil )
    end
    assert_equal( "line 1 of the access log\n", contents )
  end
end

Although I suspect MockFS would be a great fit for some projects, I ended up running into issues.

First of all, it depends on a library (extensions) that can have strange monkey-patching conflicts with other libraries. For example, compare this:

require 'faker'
puts [].respond_to?(:shuffle) # true

to this:

require 'extensions/all'
require 'faker'
puts [].respond_to?(:shuffle) # false

Secondly, as you'll notice in the above example, using MockFS requires you to use methods like MockFS.file.exist? instead of just File.exist?. This works fine if you're only testing your own code. However, if your code calls any libraries that use filesystem methods, MockFS won't work.

(Note: There is a way to mock out the default filesystem methods, but it's experimental. From the MockFS documentation:

"Reading the testing example above, you may be struck by one thing: Using MockFS requires you to remember to reference it everywhere, making calls such as MockFS.file_utils.mv instead of just FileUtils.mv. As another option, you can use File, FileUtils, and Dir directly, and then in your tests, substitute them by including mockfs/override.rb. I'd recommend using these with caution; substituting these low-level classes can have unpredictable results. ")

All that said, MockFS is probably your best option if you're only testing your code and you want to mock out files that you can't actually interact with - for instance, if you need to test that a method reads/writes a file in /etc (although for the sake of testability, it's generally good to avoid hardcoding fully-qualified paths in your code).

FakeFS is another library that uses this approach. I haven't used it personally, but it looks quite nice.

Creating temp files and directories (with Construct)

Besides mocking the filesystem, another option is to have tests interact with actual files and directories on disk. The advantages are that the test code can be simpler to write and you don't have to use any special filesystem methods.

Of course, as always, you want the test itself to contain all the relevant setup and teardown - you don't want your tests to depend upon some set of files that have no explicit connection to the test itself (or create files that aren't cleaned up).

To make this easy, we created a new library called Construct. Construct makes test setup simple by providing helpers to create temporary files and directories. It takes care of the cleanup by automatically deleting the directories and files that are created within the test. And because it creates regular files and directories, you can use plain old Ruby filesystem methods in your code and tests.

To install Construct, simply run:

# gem install devver-construct --source http://gems.github.com

Using Construct, you can write code like this:

require 'construct'

class ExampleTest < Test::Unit::TestCase
  include Construct::Helpers

  def test_example
    within_construct do |construct|
      construct.directory 'alice/rabbithole' do |dir|
        dir.file 'white_rabbit.txt', "I'm late!"
        assert_equal "I'm late!", File.read('white_rabbit.txt')
      end
    end
  end

end

Let's look at each line in more detail.

    within_construct do |construct|

When you call within_construct, a temporary directory is created. All files and directories are, by default, created within that temporary directory and the temporary directory is always deleted before within_construct completes.

The block argument (construct) is a Pathname object with some additional methods (#directory and #file, which I'll explain below). You can use this object to get the path to the temporary directory created by Construct and easily create files and directories.

Note that, by default, the working directory is changed to the temp dir within the block provided to within_construct.

      construct.directory 'alice/rabbithole' do |dir|

Here we are using the construct object to create a new directory within the temp directory. As you can see, you can create nested directories like alice/rabbithole in one step. The block argument (dir) is again a Pathname object with the same added functionality noted above.

Just like before, the working directory is changed to the newly created directory (in this case, alice/rabbithole) within the block.

        dir.file 'white_rabbit.txt', "I'm late!"

Here we use the dir object to create a file. In this case, the file will be empty. However, it's easy to provide file contents using either an optional parameter or the return value of the supplied block:

within_construct do |construct|
  construct.file('foo.txt','Here is some content')
  construct.file('bar.txt') do
  <<-EOS
  The block will return this string, which will be used as the content.
  EOS
  end
end

As a more real-world example, here's how you could use Construct to start testing the #rewrite_file! method we looked at before:

require 'test/unit'
require 'construct'
require 'shoulda'

class RewriterTest < Test::Unit::TestCase
  include Construct::Helpers

  context "#rewrite_file!" do

    should "alter each line in file" do
      within_construct do |c|
        c.file('bar/foo.txt',"a\nb\nc\n")
        Rewriter.new.rewrite_file!('bar/foo.txt') do |line|
          line.upcase
        end
        assert_equal "A\nB\nC\n", File.read('bar/foo.txt')
      end
    end

    should "not alter file if exception is raised" do
      within_construct do |c|
        c.file('foo.txt', "1\n2\nX\n")
        assert_raises ArgumentError do
          Rewriter.new.rewrite_file!('foo.txt') do |line|
            Integer(line)*2
          end
        end
        assert_equal "1\n2\nX\n", File.read('foo.txt')
      end
    end

  end

end

You can learn more at the project page (both the README and the tests have more examples).

(As an aside, since Construct changes the working directory, it doesn't play nicely with ruby-debug. Specifically, if you place a breakpoint within a block, you'll see the message "No sourcefile available for test/unit/foo_test.rb" and you won't be able to view the source. If anyone knows an easy way to make Dir.chdir work with ruby-debug, I'd very much appreciate some help!)

Conclusion

We've been moving our filesystem tests over to using Construct and so far have found it to be very useful. How do you test interactions with the filesystem? Do you use one of the above approaches, or something else? Or do you skip testing the filesystem altogether?

We're very happy to announce that today we released our public beta!

This is a big step forward for us and we're extremely excited about this release. Please try it out and give us feedback on our support site!

I'm happy to announce that Devver is one of the sponsors of BizConf. The event is shaping up to be a great one, so I'm very excited about attending.

The presenter lineup looks great. Here are just a few of the talks that I'm looking forward to attending:

Corey Haines - Why Agile Will Probably Fail You

Jon "Lark" Larkowski - Getting Things Done

Ian McFarland - Rails Economics and the ARC Model

Check out the full list of speakers.

If you're interested in attending, the bad news is that early bird pricing is over. The good news is that if you use the code DEVVER when registering, you'll get $1000 off your ticket price.

If you do attend BizConf, please come find me to talk about getting started with Devver!

In order to show how easy it is to configure Devver for a project, we've made a short screencast to walk you through the steps. We've used DataMapper as an example application. As you can see, it only takes a few minutes to set up Devver and then the specs complete in a fraction of the time. In fact, the whole process - setup and Devver run - takes less time than running 'rake spec'.

In order to see the commands clearly, you'll want to enter full-screen mode. Or, if you prefer, you can download the high-quality version.

Our friend Andrew Hyde recently helped us film a short video of us announcing some of the recent developments here at Devver. Check it out!

Devver Announces Funding from Andrew on Vimeo.

We're looking for an awesome Ruby developer to join our team. Get more details at http://devver.net/jobs.

Dan usually goes to the Boulder CTO lunches, but he was out of town this month, which meant I had the pleasure of hanging out with some of Boulder's best and brightest.

This month's guest was Matt McAdams of TrackVia. TrackVia is an online database that is powerful yet simple enough to be used by people who are used to keeping data in spreadsheets (primarily business people). Matt gave a candid and often hilarious talk that touched on both both technical topics, and, luckily for me, a discussion of pricing and metrics, which are two topics that I'm currently very interested in.

On technology decisions:

Matt wasn't a database guy originally, but used his practical knowledge he gained working on a previous startup

Went with the simplest design that could work and it's continued to scale well

Smart technology decisions have allowed TrackVia to compete with a small, lean development team

On product development:

TrackVia started as a contract project for a single customer, but they saw the broader appeal

One of the earliest databases in TrackVia is the bug database (still around).  In other words, they've been dogfooding since day one.

They don't worry about the competition. Instead, they focus on building the features that get people to sign up and pay.

On pricing:

You've got try stuff and iterate. TrackVia has changed their pricing several times.

Customers on the old pricing models have always been grandfathered in.

Sometimes raising your price can actually gain customers because some people assume that a cheap product or service must be low-quality (even if it's actually very high quality).

If big customers really want feature X, it's OK to ask them to pay extra to accelerate the development of that feature (or to customize their experience).

On metrics:

Good metrics allow you to try different strategies and measure their effect.

You must measure, tweak, and iterate.

If you can iterate on a weekly basis and your competition can iterate on a quarterly basis, you'll win.

Metrics must continually be improved. TrackVia spends a lot of time tracking useful metrics, but even they know they need to add additional metrics in some key areas.

---

As usual, the CTO lunch was a great place to hear from other Boulder companies and I learned a lot. Thanks for everyone who attended and special thanks to Matt for leading our discussion.

Tonight we just launched the brand new version of Devver at http://devver.net. It's not perfect (yeah, yeah, we know the blog doesn't match - that should be fixed in the next week or so), but we're trying to "release early, release often." Let us know how you like the new look and how we can improve it!

Last week, we had Avdi, the newest addition to our team, join us in Boulder, CO. It was great to get some face-to-face time, since Avdi will primarily be working from his home in Pennsylvania while Dan and I continue to work in Boulder.

We are excited about the benefits of having a distributed team, but we're also aware that there are a number of challenges. As a result, one of the things we worked on last week was figuring out the tools and practices we'll be using to work effectively from across the country. Luckily, both Avdi and Dan have experience working remotely which we can draw upon.

We evaluated a number of options, but settled on the following tools and practices.

Practices

• Daily Standup. Every day at the same time, we all get on video chat. We cover what we did yesterday, what we're working on today, and whether or not we're blocked on anything. The goal is to keep this meeting at 15 min or less.
• Minimize interruptions. Whenever we need to communicate with each other, we try to do so on the channel that is the least disruptive (and disrupts the fewest team members). Of course, sometimes we need to be disruptive if an issue is pressing, if someone is blocked, or if we need to have high-bandwidth communication (information, especially cues like body language, don't come across very effectively on channels like email)
• Keep it simple. We want to use the smallest number of tools and channels that will allow us to work effectively.

Channels and Tools

Less disruptive More disruptive	Channel	Tool	Properties
	Passive Updates	Present.ly	Asynchronous Not required reading
	Email	Any email client (in practice, Gmail)	Asynchronous Required reading (usually) Sometimes time-sensitive, sometimes not
	IM	Skype	Semi-synchronous (but usually synchronous) Usually time-sensitive
	Voice/video chat	Skype	Synchronous High bandwidth* (especially video chat) Best for meetings

* By "high bandwidth", I don't mean that the tool itself requires a lot of TCP/IP traffic (although this is true, it doesn't really matter). What I mean is that we can communicate a lot of information between team members in a short amount of time.

Other Tools

• Lighthouse for issue tracking
• GitHub for source control and our project wiki
• RealVNC for screen sharing (essential for remote pair programming)

This is our first attempt at finding a good set of tools and practices for remote collaboration. As time goes on, we'll undoubtedly iterate and improve upon these.

For another perspective (with a slightly different set of tools), here is a presentation from 2008 about virtual teams.

What tools and practices have worked (and which have not worked) for your team?