The Devver Blog

We're extremely happy to announce that Caliper is now generating metrics for all gems pushed to Gemcutter!

By taking advantage of the new webhooks feature in Gemcutter, Caliper is alerted whenever a new version of a gem is pushed. At that point, we automatically generate metrics. In addition, every gem page on Gemcutter now features a 'Metrics' link to the Caliper metrics.

We're thrilled to be generating metrics for so many great Ruby projects. As always, please send us feedback about how we can improve Caliper!

For more details on webhooks, check out the Changelog screencast by Nick Quaranto.

Many, many thanks to Nick Quaranto, Chad Fowler, and all the other supremely awesome Gemcutter contributors!

If you have tried Caliper, it would help us immensely if you filled out this short, eight-question survey at Survey.io. Thanks!

For the past several weeks, you may have noticed that not much has changed on Caliper. The reason for this is that we've been working hard to allow you to use Caliper on your private GitHub repositories (we're currently in private beta for that feature. If you are interested, you can sign up for a beta invitation).

However, we realize that we have much to improve on Caliper. We've been talking with our users to identify the biggest problems and most-requested features. In addition, we're getting help from the folks at Viget Labs to make Caliper easier to understand, more intuitive to use, and all around better.

In our first batch of work, we'll be focusing on three improvements.

First, we'll be developing an improved project dashboard. Our current dashboard isn't terribly useful. We'd like to better represent the state of the project, important trends, and areas that need the most help. We're working now to come up with a concrete design that addresses these issues. In the meantime, what data would you find most helpful on the dashboard?

We'll also be changing the way you navigate to specific tools. Users have told us that the specific tool names ('Reek', 'Flog') are not, by themselves, very helpful for navigation. As a result, we'll be increasingly highlighting their function. For example, even if you aren't familiar with Flog, we'll make it obvious that it is a tool for measuring complexity.

We're considering eventually merging tools that have a similar function. So instead of going to two different pages for Reek and Roodi results, we might just have a 'code smells' report. Similarly, Flog and Saikuro might be presented within the same page. Would this be helpful?

To be clear, it isn't our intention to obscure the fact that Caliper is built on great tools like metric_fu, Reek, Flog, etc. We will still make it easy to understand where the data is coming from and highlight the work being done by the dedicated open-source developers who write them. However, we think that grouping by function will make Caliper easier to use and understand.

Did you know that you can set up Caliper to automatically generate metrics using GitHub post-receive hooks? Did you know that, by default, up to 15 of your latest commits are automatically run through Caliper when you set up a project? Or that with one click, you can generate metrics for up to 40 commits throughout the history of your project?

Many users don't know this, because our UI for these features is currently pretty terrible. The last task in the current batch of work will be to make these features more visible and easier to use. Our goal is to make it simple to see trends in your project over time.

(Incidentally, many users also don't know that you can view a metrics diff for two commits to see how a commit has changed the metrics, but making this more noticeable may or may not get tackled in the first set of changes).

What's next for Caliper after we complete these changes? We're currently talking to users about what features and improvements matter most to them. One possibility is to display some or all of our metric data directly on a view of source code itself, so you don't have to open up an editor (or click through to GitHub) to see the code. How important is this feature to you?

As always, feedback from our users is hugely helpful. If you have ideas about features that you'd like to see, please let us know in the comments or start a discussion at our support site. Thanks!

After much deliberation, we've decided to shut down our Test Accelerator product on January 22nd, 2010. (the Accelerator was previously just known as "Devver", but to clarify between our old product and Caliper, we're referring to it as the "Accelerator" or the "Test Accelerator").

This has been a difficult decision for us. Understandably, some people have been confused and disappointed by this news, so I wanted to take some time to explain our decision.

For over a year, we worked to build our vision of accelerated Ruby testing in the cloud. We made our share of mistakes and learned a lot in the process. Ultimately, we found the demand was too small to warrant overcoming all the technical challenges that were harder than we anticipated.

We had many different technical challenges, but it boils down to this: we underestimated the difficulty of making it simple to run Ruby/Rails tests in the cloud.

When we began working on the Accelerator, we knew we would need many features: automatic installation of gems, automatic setup of common databases, and support for Test:Unit, Shoulda, and RSpec. However, other features caught us by surprise. For instance, many projects required running daemons, such as memcached or Solr to be running when the tests executed. Also, it was common for projects to work fine on a user's machine, but fail in the cloud due to undeclared RubyGem dependencies (or dependencies that lacked sufficient version information). Some problems were extra difficult because, due to costs, we needed to be able to run many projects on a single EC2 instance (and it's currently not possible to create virtual machines within EC2 Xen instances).

We worked hard to overcome these challenges, but it was still too common for users to discover that running tests in the cloud required additional work on their part (or, in some cases, that their project was not supported). And even worse, users had to jump through these hoops before they could assess whether or not the Accelerator would speed up their tests enough to make the process worth it. Because the potential benefit and possible configuration time were both unknown, many users understandably gave up before they experienced the time savings of running tests in parallel. And unfortunately, the projects that were most likely to need acceleration (those with large test suites), were also the most likely to have complex dependencies and require the most setup work.

While we may have been able to solve these problems with a lot of additional engineering, we also found that only a small percentage of teams really needed test acceleration. Some teams loved our solution, because they had test suites that took 10, 20, or even 45 minutes to run. However, most projects had smaller test suites. We found that many teams didn't mind waiting for suites that took 3-5 minutes (or, more accurately, didn't mind enough to justify the considerable time investment of setting up tests on the Accelerator).

So, after much discussion, we decided that we needed to create a product that would help a larger set of Ruby teams and require significantly less setup to provide value. Those discussions led to Caliper, a tool to identify risky code and technical debt in Ruby code.

Focus is key in a startup, especially a small one like ours. While we considered keeping the Accelerator running in maintenance mode, we realized we could not provide the level of support our users deserve while executing effectively on Caliper. As a result, we've decided to shut down the Accelerator indefinitely.

We deeply appreciate each and every person who helped us with the Accelerator, either by using it, or giving us ideas, feedback, and encouragement. We could not have gotten as far as we did without the help from users, friends, and mentors.

It pains us to shut down the service and we apologize for any inconvenience this causes our users. We hope that you understand our decision and we look forward to helping you improve your code with Caliper.

(Accelerator users: if you need to speed up your tests, here are some possible alternatives to Devver: multitest, parallel_specs, Testjour, and DeepTest. I don't have experience with these tools, so I can't comment on how well they work myself, but you may want to check them out.)

According to the Sinatra FAQ, source reloading was taken out of Sinatra in version 0.9.2 due to "excess complexity" (in my opinion, that's a great idea, because it's not a feature that needs to be in minimal a web framework like Sinatra). Also, according to the FAQ, Rack::Reloader (included in Rack) can be added to a Sinatra application to do source reloading, so I decided to try it out.

Setting up Rack::Reloader is easy:

require 'sinatra'
require 'rack'

configure :development do
  use Rack::Reloader
end

get "/hello" do
  "hi!"
end

$ ruby hello.rb
== Sinatra/0.9.4 has taken the stage on 4567 for development with backup from Thin
>> Thin web server (v1.2.4 codename Flaming Astroboy)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:4567, CTRL+C to stop
[on another terminal]
$ curl http://localhost:4567/hello
hi!

If you add another route, you can access it without restarting Sinatra:

get "/goodbye" do
  "bye!"
end

$ curl http://localhost:4567/goodbye
bye!

But what happens when you change the contents of a route?

get "/hello" do
  "greetings!"
end

$ curl http://localhost:4567/hello
hi!

You still get the old value! What is going on here?

Rack::Reloader simply looks at all files that have been required and, if they have changed on disk, re-requires them. So each Sinatra route is re-evaluated when a reload happens.

However, identical Sinatra routes do NOT override each other. Rather, the first route that is evaluated is used (more precisely, all routes appended to a list and the first matching one is used, so additional identical routes are never run).

We can see this with a simple example:

require 'sinatra'

get "/foo" do
 "foo"
end

get "/foo" do
 "bar"
end

$ curl http://localhost:4567/foo
foo   # The result is 'foo', not 'bar'

Clearly, Rack::Reloader is not very useful if you can't change the contents of any route. The solution is to throw away the old routes when the file is reloaded using Sinatra::Application.reset!, like so:

configure :development do
  Sinatra::Application.reset!
  use Rack::Reloader
end

$ curl http://localhost:4567/hello
greetings!

Success!

A word of caution: you MUST call reset! very early in your file - before you add any middleware, do any other configuration, or add any routes.

This method has worked well enough for our Sinatra application. However, code reloading is always tricky and is bound to occasionally produce some weird results. If you want to significantly reduce the chances for strange bugs (at the expense of code loading time), try Shotgun or Rerun. Happy reloading!

We released the first version of Caliper just under two months ago. Since then, we've fixed bugs, made improvements, and rolled out new features. The feedback we've gotten from the community has been encouraging and your suggestions have helped up improve Caliper each week.

Initially, we simply wanted to take an existing tool (namely, metric_fu) and make it dead simple to set up, so that all open-source projects could get access to metric data. However, our larger goal is to improve the usability, features, and data presentation in Caliper so it becomes an indispensable part of your regular Ruby development process.

To that end, I'm happy to announce that support for private projects is on the road map. We're still working out the details, but we plan to release this feature within the next thirty days.

At the same time, we'll work with users to figure out how we can make metrics useful enough to be a regular part of your process (by "regular", we mean that we hope you use metrics at least once a week).

We recently put out a short survey (results here) to try to assess what features are most important for our users. By looking at the results (somewhat limited, likely due to overly wordy questions on my part) and based on some conversations I had at RubyConf, we see a few possible scenarios for where to take Caliper next.

"Code exploration"
In this scenario, you'd primarily use Caliper to jump into an existing code base, in order to help you better understand where the problematic areas are. This might be used by consultants for a project-level code review, for understanding how to start contributing to an open-source project, or even to periodically do a survey of problems of your own project. The primary features for this scenario would be our Hot Spots feature (which list the worst methods, classes, and files in the project). Important new features include a display that shows the source code with the problems displayed alongside the source.

"Quality monitoring"
In this scenario, the primary usage for Caliper is to set and maintain certain metrics thresholds within your code base. You'd set levels for the various tools and then get alerts when those thresholds are reached. Key features would include an ability to customize thresholds for each tool, a dashboard that displays offending classes and methods, and customizable notifications.

"Refactoring assistant"
In this scenario, a single developer uses Caliper to assist them in refactoring up his/her code, perhaps after reviewing a commit that will be pushed, or because a specific section of code is known to be overly complex. Caliper would both find the biggest problems and provide feedback on how refactorings change the metrics. Key features would include an improved commit-based view of metric changes as well as the ability to push code directly to Caliper before merge that code into a shared 'master' branch.

Do any of these scenarios resonate with you? Would you prefer one over another? Is there another scenario that would be more useful? If you are currently using metrics in your project, how do you use them and what features do you want? If you don't currently use metrics, what features would make you change your mind? Please make your voice heard in the comments, so we can make Caliper a valuable addition to your toolbox.

For the past few months, we've been building Caliper to help you easily generate code metrics for your Ruby projects. We've recently added another dimension of metrics information: community statistics for all the Ruby projects that are currently in Caliper.

The idea of community statistics is two-fold. From a practical perspective, you can now compare your project's metrics to the community. For example, Flog measures the complexity of methods. Many people wonder exactly defines a good Flog score for an individual method. In Jake Scruggs' opinion, a score of 0-10 is "Awesome", while a score of 11-20 is "Good enough". That sounds correct, but with Caliper's community metrics, we can also compare the average Flog scores for entire projects to see what defines a good average score.

To do so, we calculate the average Flog method score for each project and plot those averages on a histogram, like so:

Looking at the data, we see that a lot of projects have an average Flog score between 6 and 12 (the mean is 10.3 and the max is is 21.3).

If your project's average Flog score is 9, does that mean it has only "Awesome" methods, Flog-wise? Well, remember that we're looking at the average score for each project. I suspect that in most projects, lots of tiny methods are pulling down the average, but there are still plenty of big, nasty methods. It would be interesting to look at the community statistics for maximum Flog score per project or see a histogram of the Flog scores for all methods across all projects (watch this space!).

Since several of the metrics (like Reek, which detects code smells) have scores that grow in proportion to the number of lines of code, we divide the raw score by each project's lines of code. As a result, we can sensibly compare your project to other projects, no matter what the difference in size.

The second reason we're calculating community statistics is so we can discover trends across the Ruby community. For example, we can compare the ratio of lines of application code to test code. It's interesting to note that a significant portion of projects in Caliper have no tests, but that, for the projects that do have tests, most of them have a code:test ratio in the neighborhood of 2:1.

Other interesting observations from our initial analysis:
* A lot of projects (mostly small ones) have no Flay duplications.
* Many smaller projects have no Reek smells, but the average project has about 1 smell per 9 lines of code.

Want to do your own analysis? We've built a scatter plotter so you can see if any two metrics have any correlation. For instance, note the correlation between code complexity and code smells.

Here's a scatter plot of that data (zoomed in):

Over the coming weeks, we'll improve the graphs we have and add new graphs that expose interesting trends. But we need your help! Please let us know if you spot problems, have ideas for new graphs, or have any questions. Additionally, please add your project to Caliper so it can be included in our community statistics. Finally, feel free to grab the raw stats from our alpha API* and play around yourself!

* Quick summary: curl http://api.devver.net/metrics for JSON, curl -H 'Accept:text/x-yaml' http://api.devver.net/metrics for YAML. More details. API is under development, so please send us feedback!

Devver is great if you have a large test or spec suite that benefits from massive parallelism. But what if you have a project with fast tests and you just want to run them regularly? In this case, our existing service may be more than you need.

To further our vision of enabling Ruby developers to easily and quickly run their tests in the cloud, we've recently released the very first version of a HTTP-based Devver API. Right now, it offers a single service: a GitHub compatible web hook which will cause Devver to do a continuous integration-style build of your project, and email you the results.

The API is still in it's infancy but we already have some cool features:

We're releasing very early, so there are some rough spots (notably, the output is too verbose and the setup is too complicated). But we're also releasing often, so please let us know what you think and which features you'd like to see added.

If you have a project on GitHub, please try out the API and send us feedback. We'd really appreciate it!

We've just released version 2.4.2 of our new client. This release fix some bugs that you may have run into if you're just getting started on Devver.

Important Changes

The biggest fixes are in the default install_dependencies hook (located in .devver/hooks/install_dependencies). The old default hook failed to install Rails correctly, so you may have repeatedly seen an error like this:

Missing the Rails 2.3.3 gem. Please `gem install -v=2.3.3 rails`, update your RAILS_GEM_VERSION setting in config/environment.rb for the Rails version you do have installed, or comment out RAILS_GEM_VERSION to use the latest version installed.

In addition, the default hook now runs rake gems:install RAILS_ENV=test instead of rake gems:install to ensure that any gems declared in config/environments/test.rb get installed.

Updating your hook

If you have not altered your install_dependencies hook, you can just do the following to get the new behavior:

rm .devver/hooks/install_dependencies
devver --init

If you have altered this file, you'll need to do the following to get the new behavior:

mv .devver/hooks/install_dependencies .devver/hooks/install_dependencies.mine
devver --init

Then port your project-specific changes from install_dependencies.mine to install_dependencies.

As always, if you have any problems, please don't hesitate to let us know by emailing support@devver.net.

If you're trying to use Devver and you see this error:

/Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/em/connection.rb:302:in `start_tls': undefined method `set_tls_parms' for EventMachine:Module (NoMethodError)
from /Library/Ruby/Gems/1.8/gems/devver-2.4.1/lib/client/mod_client.rb:32:in `post_init'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/em/connection.rb:43:in `new'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/em/connection.rb:36:in `instance_eval'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/em/connection.rb:36:in `new'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:716:in `bind_connect'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:723:in `connect'
from /Library/Ruby/Gems/1.8/gems/devver-2.4.1/lib/client/client.rb:125:in `push_tests'
from /Library/Ruby/Gems/1.8/gems/devver-2.4.1/bin/devver:145
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:1503:in `call'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:1503:in `event_callback'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/pr_eventmachine.rb:342:in `run_timers'
from (eval):44:in `each'
from (eval):44:in `each'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/pr_eventmachine.rb:339:in `run_timers'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/pr_eventmachine.rb:322:in `run'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/pr_eventmachine.rb:318:in `loop'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/pr_eventmachine.rb:318:in `run'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/pr_eventmachine.rb:64:in `run_machine'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.8/lib/eventmachine.rb:242:in `run'
from /Library/Ruby/Gems/1.8/gems/devver-2.4.1/bin/devver:144
from /usr/bin/devver:19:in `load'
from /usr/bin/devver:19

... it's because EventMachine (and other compiled gems) may act a little funky after installing Apple's latest OS.

In this post, I'll go through some steps that should help you resolve this error (... assuming you use MacPorts. If you don't, please tailor these directions accordingly). But first:

• Warning: The directions below may leave your system in an unknown state. Please, please, please backup everything before going further. Time Machine is your friend!
• Disclaimer: These steps worked for me, but I can't guarantee they'll work for you. If you have any problems, please let me know in the comments or email support@devver.net.
• Note: If you already installed Devver before upgrading to Snow Leopard, things will likely continue to work fine for you for now - but these directions may be helpful when you upgrade Devver clients.

OK, now that we've got that out of the way ...

1. Upgrade MacPorts

Try running something like port list installed. If you get an error like this:

dlopen(/opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib, 10): no suitable image found.  Did find:
/opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib: no matching architecture in universal wrapper
while executing
"load /opt/local/share/macports/Tcl/pextlib1.0/Pextlib.dylib"
("package ifneeded Pextlib 1.0" script)
invoked from within
"package require Pextlib 1.0"
(file "/opt/local/bin/port" line 40)

You need to install the latest version of MacPorts. Go here and download the 'Snow Leopard' dmg and install.

2. Reinstall openssl and ruby

Now that MacPorts is installed, do this:

sudo port sync
sudo port upgrade openssl
sudo port upgrade ruby

You could also try sudo port upgrade ruby186 if you want Ruby 1.8.6, but I haven't tried this myself.

3. Reinstall EventMachine

sudo gem uninstall eventmachine
sudo gem install eventmachine

And you're done!

Now you should be able to run Devver normally. If that didn't work, please let me know!

Bonus: Reinstall other gems

Matt Aimonetti has written a handy script (Update: here is my fork, with a new gem path an an important bug fix on line eight) that will find the gems you should reinstall once you've done the above steps. The only change you'll need to make is to replace the gem path:

#Dir.glob('/Library/Ruby/Gems/**/*.bundle').map do |path|
Dir.glob('/opt/local/lib/ruby/gems/**/*.bundle').map do |path|

For more info, you can find a great guide on resolving Ruby-related problems when upgrading to Snow Leopard at rubyonrails.org