The Devver Blog

Update: Devver now offers a hosted metrics service for Ruby developers which can give you useful feedback about your code. Check out Caliper, to get started with metrics for your project.

This is the second post in my series of Ruby tools articles. This time I am focused on Ruby test quality tools. Devver is always really interested in testing, and obviously the quality of a project's tests is important. We are always looking at ways to add even more value to the investment teams put in with testing. Simply knowing that you are writing higher quality tests helps increase the value returned on the time invested in testing. I haven't found many tools to help with test quality, but these tools are a great help to any Ruby tester.

Heckle

---

Heckle is an interesting tool to do mutation testing of your tests. Heckle currently supports Test:Unit and RSpec, but does have a number of issues. I had to run it on a few different files and methods before I got some useful output that helped me improve my testing. The first problem was it crashing when I passed it entire files (crashing the majority of the time). I then began passing it single methods I was curious about, which still occasionally caused Heckle to get into an infinite loop case. This is a noted problem in Heckle, but -T and providing a timeout should solve that issue. In my case it was actually not an infinite loop timing error, but an error when attempting to rewrite the code, which lead to a continual failure loop that wouldn't time out. When I found a class and method that Heckle could test I got some good results. I found one badly written test case, and one case that was never tested. Lets run through a simple Heckle example.

#install heckle
dmayer$ sudo gem install heckle

#example of the infinite loop Error Heckle run
heckle Syncer should_be_excluded? --tests test/unit/client/syncer_test.rb -v
Setting timeout at 5 seconds.
Initial tests pass. Let's rumble.

**********************************************************************
***  Syncer#should_be_excluded? loaded with 13 possible mutations
**********************************************************************
...
2 mutations remaining...
Replacing Syncer#should_be_excluded? with:

2 mutations remaining...
Replacing Syncer#should_be_excluded? with:
... loops forever ...

#Heckle run against our Client class and the process method
dmayer$ heckle Client process --tests test/unit/client/client_test.rb
Initial tests pass. Let's rumble.

**********************************************************************
***  Client#process loaded with 9 possible mutations
**********************************************************************

9 mutations remaining...
8 mutations remaining...
7 mutations remaining...
6 mutations remaining...
5 mutations remaining...
4 mutations remaining...
3 mutations remaining...
2 mutations remaining...
1 mutations remaining...

The following mutations didn't cause test failures:

--- original
+++ mutation

 def process(command)

   case command
   when @buffer.Ready then
     process_ready
-  when @buffer.SetID then
+  when nil then
     process_set_id(command)
   when @buffer.InitProject then
     process_init_project
   when @buffer.Result then
     process_result(command)
   when @buffer.Goodbye then
     kill_event_loop
   when @buffer.Done then
     process_done
   when @buffer.Error then
     process_error(command)
   else
     @log.error("client ignoring invalid command #{command}") if @log
   end
 end

--- original
+++ mutation
 def process(command)
   case command
   when @buffer.Ready then
     process_ready
   when @buffer.SetID then
     process_set_id(command)
   when @buffer.InitProject then
     process_init_project
   when @buffer.Result then
     process_result(command)
   when @buffer.Goodbye then
     kill_event_loop
   when @buffer.Done then
     process_done
   when @buffer.Error then
     process_error(command)
   else
-    @log.error("client ignoring invalid command #{command}") if @log
+    nil if @log
   end
 end

Heckle Results:

Passed    :   0
Failed    :   1
Thick Skin:   0

Improve the tests and try again.

#Tests added / changed to improve Heckle results
  def test_process_process_loop__random_result
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    client.log.expects(:error).with("client ignoring invalid command this is random")
    client.process("this is random")
  end

  def test_process_process_loop__set_id
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    cmd = DataBuffer.new.create_set_ids_msg("4")
    client.expects(:process_set_id).with(cmd)
    client.process(cmd)
  end

#A final Heckle run, showing successful results

dmayer$ heckle Client process --tests test/unit/client/client_test.rb

Initial tests pass. Let's rumble.
**********************************************************************

***  Client#process loaded with 9 possible mutations

**********************************************************************
9 mutations remaining...

8 mutations remaining...

7 mutations remaining...

6 mutations remaining...

5 mutations remaining...

4 mutations remaining...

3 mutations remaining...

2 mutations remaining...

1 mutations remaining...

No mutants survived. Cool!
Heckle Results:
Passed    :   1

Failed    :   0

Thick Skin:   0
All heckling was thwarted! YAY!!!

rcov

---

rcov is a code coverage tool for Ruby. If you are doing testing you should probably be monitoring your coverage with a code coverage tool. I don't know of a better tool for code coverage than rcov. It is simple to use and generates beautiful, easy-to-read HTML charts showing the current coverage broken down by file. An easy way to make you project more stable is to occasionally spend some time increasing the coverage you have on your project. I have always found it a great way to get back into a project if you have been off of it for awhile. You just need to find some weak coverage points and get to work.

rcov screenshot