• Single-file Sinatra apps with specs baked-in

    It's so easy to create little single-file apps in Sinatra that it almost seems a shame to start a second file just for tests.  The other day Dan and I decided to see if we could create a Sinatra app with everything - including the tests - baked right in.  Here's what we came up with.

    The code switches modes on the name of the executable used to run the file. If we run it with the spec command, we get a test run:

    $ spec -fs sinatra-tests-baked-in.rb

    Example App
    - should serve a greeting
    - should serve content as text/plain

    Finished in 0.007221 seconds

    2 examples, 0 failures

    Otherwise, if we call it as a Ruby program, it runs the Sinatra server as we would expect:

    $ ruby sinatra-tests-baked-in.rb
    == Sinatra/0.9.1.1 has taken the stage on 4567 for development with backup from Thin
    >> Thin web server (v1.0.0 codename That's What She Said)
    >> Maximum connections set to 1024
    >> Listening on 0.0.0.0:4567, CTRL+C to stop

    And there you have it: a true single-file application, specs and all.

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on May 13th, 2009 by Avdi in Development, Hacking, Ruby and tagged , , .

  • Managing Amazon EC2 with your iPhone

    I wanted a quick way when out and about to easily manage our AWS EC2 instances while out and about. It hasn't happened often, but occasionally I am away from the computer and I need to reboot the instances. Perhaps I remember our developer cluster isn't being used and want to shut it down to save some money.

    I didn't find anything simple and free with a quick Google search, so in a about an hour I wrote a nice little Sinatra app that will let me view our instances, shutdown, or reboot any specific instance or all of them. The tiny framework actually turned out to be even more useful as I now have options that let us tail error logs, reboot Apache, reboot mongrel clusters, or execute any common system administration task.

    I won't be going into detail on how to build a iPhone webapp using Sinatra and iUI, because Ben already created an excellent post detailing all of those steps. In fact I used his old project as the template when I created this project. I can't begin to explain how amazingly simple it is to build an iPhone webapp using Sinatra, so if you have been thinking of a quick project I highly recommend it.

    Here are some screen shots showing the final app. (screenshot courtesy of iPhoney):

    ec2 manager home view

    ec2 manager home view.

    ec2 manager describe view

    ec2 manager describe instances view.

    ec2 manager instance view.

    ec2 manager instance view.

    This app uses the Amazon EC2 API Tools to do all the heavy lifting. So this app assumes that you already have the tools installed and working on the machine you want this app to run on. This normally involves installing the tools and setting up some environment variables like EC2_HOME, so make sure you can run ec2-describe-instances from the machine. After that you should just have to change EC2_HOME in the Sinatra app to match the path where you installed the EC2 tools.

    Let me know if you have any issues, it is quick and dirty, but I have already found it useful.

    To run the app:
    cmd> ruby -rubygems ./ec2_manager.rb

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on March 5th, 2009 by Dan in Amazon Web Services, Development, Hacking, Ruby, Tips & Tricks, Tools and tagged , , , , .

  • Using Ruby to Send Update Emails to Our Mentors

    At Devver.net, we send out weekly email updates to an awesome set of mentors. We do this for a number of reasons. First and foremost, we get valuable feedback and advice from our mentors on a variety of issues. But it's also an easy and effective way to keep us on track and even maximize our chances of success. As Paul Graham says in How Not To Die (he was talking directly to YC teams, but you'll get the idea):

    "For us the main indication of impending doom is when we don't hear from you. When we haven't heard from, or about, a startup for a couple months, that's a bad sign.

    ...

    Maybe if you can arrange that we keep hearing from you, you won't die.

    That may not be so naive as it sounds. ... [The] mere constraint of staying in regular contact with us will push you to make things happen, because otherwise you'll be embarrassed to tell us that you haven't done anything new since the last time we talked."

    Foodzie started emailing their mentors early in the summer. We actually borrowed (stole) their email format and best practices.

    One thing we've tried to not do is send out a completely generic email to all our mentors. Depending on the content and the interaction we've had with a specific mentor, we'll adjust his email accordingly. We begin each email with their name and send it directly to them (in other words, we don't put a huge list of addresses in the To, CC, or BCC fields). We do this because we can tailor it and it helps elicit individual responses from each mentor (it's easier to ignore a question if it's sent to a group).

    But, of course, sometimes the emails to a few mentors can be identical. In this case, my not-so-well-kept secret is that I just use a simple Ruby script to send out a duplicate email that appears to be hand-crafted (or at least copied and pasted).

    I've been told that Outlook can perform this functionality easily, but I don't know of any way to do this within Gmail. If there is, let me know so I can feel a little silly (in any case, the Ruby code was fun to write).

    To run this code, you'll need to install the highline gem. You'll also need to add your Gmail account, recipients, subject message, etc. Finally, you'll want to put your message inside a separate file within project directory. That way, you can easily modify, spellcheck, and format to your heart's content before sending.

    You can get the entire gmailr source code (all two files!) at Github. Please use this script for good, not evil - no one likes a spammer. Enjoy!

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on January 20th, 2009 by Ben in Development, Devver, Hacking, Ruby.

  • Installing and running git-svn on Mac OSX 10.4 Tiger

    I am shocked at how much time it took me to get git-svn working on my mac. I use MacPorts, which works well most of the time. Sometimes it has problems which makes me really wish for apt-get on OS X. apt-get normally has worked much nicer for me, but can have its issues too. I even occasionally wish for Windows and a simple install.exe which works 95% of the time out of the box. Really I wish Apple would throw some engineer support to MacPorts and make the service rock solid.

    I have had git installed and working for awhile, but preparing to switch our main project from Subversion (svn) to git, I thought I should start using git-svn. It seemed smart to use git-svn for awhile to get used to git, before a full switch so I could fall back on svn in a crunch. I decided to start using git-svn, but the first run of the git-svn command caused this error, and I had no idea how much of my night was about to be wasted...

    Can't locate SVN/Core.pm in @INC

    Searching led to a couple of webpages, but the most useful was getting git to work on OS X Tiger. It had a quick fix that might work or the long route fix. For some lucky people it is just a path problem. I checked if that was the case for me, by the following command

    PATH=/opt/local/bin:$PATH; git svn

    unfortunately for me I got the same error, OK I need to reinstall SVN with additional bindings...

    > sudo port uninstall -f subversion-perlbindings
    > sudo port install -f subversion-perlbindings

    leading to this error:

    ---> Building serf with target all
    Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_www_serf/work/serf-0.2.0" && make all " returned error 2
    Command output: /opt/local/share/apr-1/build/libtool --silent --mode=compile /usr/bin/gcc-4.0 -O2 -I/opt/local/include -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I. -I/opt/local/include/apr-1 -I/opt/local/include/apr-1 -c -o buckets/aggregate_buckets.lo buckets/aggregate_buckets.c && touch buckets/aggregate_buckets.lo
    libtool: compile: unable to infer tagged configuration
    libtool: compile: specify a tag with `--tag'
    make: *** [buckets/aggregate_buckets.lo] Error 1

    I spent some time searching and eventually I find the solution to the serf error. I couldn't read the blog because it wasn't in English, but I could read enough to solve my MacPorts serf install problem. I followed these few lines from the blog

    cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_www_serf/work/serf-0.2.0
    $ sudo ./configure --prefix=/opt/local --with-apr=/opt/local --with-apr-util=/opt/local
    $ sudo make all
    $ sudo port install serf

    Awesome, I have serf. Now what is next? Back to building svn with perl bindings, that works. Now, let's build git again since svn with perl bindings is finally installed.

    sudo port install git-core +svn

    Which fails because of p5-svn-simple

    dyld: lazy symbol binding failed: Symbol not found: _Perl_Gthr_key_ptr
    Referenced from: /usr/local/lib/libsvn_swig_perl-1.0.dylib
    Expected in: flat namespace

    dyld: Symbol not found: _Perl_Gthr_key_ptr
    Referenced from: /usr/local/lib/libsvn_swig_perl-1.0.dylib
    Expected in: flat namespace

    Error: Status 1 encountered during processing.

    OK, I need to get p5-svn-simple working. Searching leads to this thread MacPort errors related to git. Here you will find the amazingly useful comment by Orestis:

    "As mentioned move your libsvn_swig_perl* out of /usr/local/lib AND out of /usr/lib into temporary folders.

    Uninstall and reinstall subversion-perlbindings

    Install p5-svn-simple (and git-core +svn which is what lead me here)

    Move the libsvn_swig_perl files back in /usr/lib and /usr/local/lib (or else git svn won't work). 

    > cd /usr/local
    > mv ./lib/libsvn_swig_perl* ./bak/
    > sudo port install p5-svn-simple

    Sweet that works now

    > sudo port install git-core +svn
    > cd /usr/local
    > mv ./bak/libsvn_swig_perl* ./lib/

    Finally I try to run git-svn, only to see the same ERROR I had from the very beginning! I am about to lose it but decide that I should try the quick fix again to see if it is the path issue...

    PATH=/opt/local/bin:$PATH; git svn

    It works! Alright now it is just a path problem. So I open up my .bash_profile, and notice I already have that path included

    # Setting the path for MacPorts.
    export PATH=/opt/local/bin:/opt/local/sbin:/Applications/MzScheme\ v352/bin:$PATH

    But I also have an additional path added from when I originally built git from source, and it looks like I was running my old broken version of git-svn. So I just had to remove this one line from my .bash_profile

    export PATH=~/projects/git-1.5.6.1:$PATH

    and hours later and with a ton of frustration I have a fully functioning git-svn.

    Now that it is working, you can move on to learning git-svn in 5 minutes.

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on December 9th, 2008 by Dan in Development, Hacking, Misc, Tips & Tricks, Tools.

  • Revisiting additional Ruby Tools

    I have heard about new Ruby tools since I did my Ruby Tools Roundup. I am always interested in tools that can help improve our code, so I had to check some of them out. Similar to my last tools post, I will be trying out a tool and writing my general impressions along with the basic usage.

    reek


    I have to start with reek, since it has been the most requested and searched on our site since I originally wrote about tools. reek will help identify code smells, allowing you to fix up your code. Instead of looking at cyclomatic complexity or other metrics, reek looks at patterns to warn you about bad code. Reek currently detects a few code smells (Long Method, Large Class, Feature Envy, Uncommunicative Name, Long Parameter List, Utility Function, Nested Iterators, Control Couple, Duplication) but more are on the way.

    I think this project is useful but would need to be more customized before a nightly run would yield very useful results. The biggest problem I have is the signal to noise ratio seemed pretty high. Reek was warning me about "long methods" that were only 7 statements long, which just isn't something I am concerned about. The warnings on duplicate methods calls can be useful, after running reek on a few files I found a couple places where duplicate method calls were wasting time. Many of the other smells are interesting like 'Feature Envy', and 'Utility Function'. I will need to use reek more before I know if these smells are good indicators or often false positives.

    Below reek finds a utility function next_tick which is definitely a helper function that actually exists in two of our files, which probably should be moved into a helper mixin.

    def next_tick
        if(EM.reactor_running?)
          EM.next_tick do
            yield
          end
        else
          yield
        end
    end

    I am really looking forward to see how the tool progresses. If the project allows for a simple config customization to change the thresholds as well as ignore some files/smells, this could become a very useful tool to help keep a team maintain a high expectation of code quality. It would be useful to get nightly reports about any code that might not meet expectations, so a quick group code review could decide if it is an exception (which can be quickly added to the config) or if the code should be refactored and cleaned up.

    dmayer$ sudo gem install reek
    dmayer$ reek ./lib/client/client.rb
    [Utility Function] Client#next_tick doesn't depend on instance state
    [Long Method] Client#process_done has approx 7 statements
    [Duplication] Client#process_ready calls @buffer.create_reload_msg more than once
    [Long Method] Client#process_ready has approx 10 statements
    [Duplication] Client#report_system_message calls result.msg more than once
    [Feature Envy] Client#report_system_message refers to result more than self
    [Duplication] Client#send_tests calls Time.now more than once
    [Long Method] Client#send_tests has approx 24 statements
    [Feature Envy] Client#send_tests refers to tests more than self
    #check a whole directory
    dmayer$ reek ./lib/client/*

    Towelie


    Towelie helps discover duplication in Ruby code, it will help keep your code DRY. It doesn't have a nice interface at the moment and it is pretty young code. That being said, it can still be a really useful tool to help guide refactoring and code cleanup.

    ~/projects dmayer$ git clone git://github.com/gilesbowkett/towelie.git
    dmayer$ cd ~/projects/devver/
    dmayer$ irb -r ~/projects/towelie/lib/towelie.rb
    irb(main):001:0> @t = Towelie.new
    => #, @model=#>
    irb(main):002:0> @t.parse "lib/client"
    (string):24: warning: useless use of a variable in void context
    => nil
    irb(main):003:0> puts @t.duplicates
    found in:
    lib/client/test_unit_reporter.rb
    lib/client/rspec_reporter.rb
    
    def nl
    report_nl
    end
    
    ... 2 more dupes in the reporters ...
    
    found in:
    lib/client/test_unit_reporter.rb
    lib/client/rspec_reporter.rb
    
    def report(str)
    print(str.to_s)
    end
    
    found in:
    lib/client/sync_client.rb
    lib/client/rev_sync_client.rb
    lib/client/rev_client.rb
    lib/client/client.rb
    
    def quit
    send(@buffer.create_quit_msg)
    end
    
    found in:
    lib/client/sync_client.rb
    lib/client/rev_sync_client.rb
    lib/client/rev_client.rb
    lib/client/client.rb
    
    def send_quit
    send(@buffer.create_quit_msg)
    end
    
    => nil
    irb(main):004:0>

    There are currently many duplications because we are maintaining two clients while deciding what route to eventually take. We have also moved a lot of our shared client code into a mixin, and Towelie finds some methods that really should be moved there as well such as the methods "quit" and "send_quit", which is currently duped in 4 files. Towelie also points to the fact that we should refactor our reporters because they both duplicate code.

    I have always been annoyed with copied and pasted functions accidentally working its way in code, this could be a useful nightly run to keep a team DRY. Sometimes two team members implement the same functionality without even knowing a solution already exists in the code base. If you want to go a bit more in depth, check out Giles Bowkett's (creator of Towelie) How to use Towelie

    Flay


    Flay is another great tool by Ryan Davis who also works on Heckle and Flog which I covered in the past. Flay, like Towelie, helps keep your code DRY, it detects exact and similar code throughout a project. It seems to be more powerful than Towelie, as seen in this Towelie and Flay comparison. My biggest complaint is the current release has some pretty basic output that you see below. The output I got from Towelie was immediately more recognizable and useful, while Flay currently requires you to dig in a bit deeper on your own into its suggestions. An improvement is already being worked on and a verbose output mode should be in the release soon. Once better output is included I think Flay will be immediately useful out of the box even with small amounts of developer effort.

    I like that Flay has weight system, which should make it easy to set some threshold to ignore, high level weights are more likely to be worth your time and attention. One piece of code Flay tagged with a low weight was code that rescued and logged different errors thrown, which while similar actually served a purpose.

    rescue Errno::EISDIR => ed
          @stderr.puts "Error: #{ed.message}" if @stderr
          @stderr.puts "You can't pass a directory to devver only test files. Quitting." if @stderr
          send_quit
        rescue LoadError => le
          @stderr.puts "Error: #{le.message}" if @stderr
          @stderr.puts "Not all of the files can be found. Quitting." if @stderr
          send_quit
        rescue SyntaxError, NameError => se
          @stderr.puts "Error: #{se.message}" if @stderr
          @stderr.puts "This file doesn't appear to be a valid Ruby file. Quitting." if @stderr
          send_quit
    end

    Digging into the Flay results turned up some duplicate code that Towelie had missed. Since Towelie also caught a method that was duped in 4 client files that Flay missed (I was expecting Towelie's results to be a subset of what Flay found), perhaps there is room for both of the tools and learning to work with both a little bit is worth the time. After a little bit of work perhaps one of the projects will become a clearly better option. Until then I will be following both of these projects.

    sudo gem install flay
    dmayer$ flay lib/client/*.rb
    Processing lib/client/client.rb...
    Processing lib/client/mod_client.rb...
    ...
    Processing lib/client/syncer.rb...
    
    Matches found in :defn (mass = 84)
    lib/client/mod_client.rb:86
    lib/client/mod_rev_client.rb:124
    
    Matches found in :block (mass = 57)
    lib/client/client.rb:201
    lib/client/client.rb:205
    lib/client/client.rb:209
    
    ... 6 more results ...
    
    Matches found in :if (mass = 34)
    lib/client/mod_client.rb:63
    lib/client/mod_rev_client.rb:111
    
    Matches found in :defn (mass = 32)
    lib/client/mod_rev_client.rb:36
    lib/client/mod_rev_client.rb:50


    Conclusions


    That should cover it for this Ruby tools post, but I am really enjoying checking out the tools showing up in the Ruby scene. So as always let me know if I missed something, or if there is a tool you would like to see a full write up on. After some of the tools mature a little bit I will have to revisit a few of the tools which are currently in the early stages. I hope the Ruby tools scene keeps as active as it has been lately because there are some interesting projects being worked on.

    honorable mentions (things I didn't think really needed a full write up)


    • metric-fu a great gem to give quick access to a bunch of tools and metrics about your code (RCov, Saikuro, Flog, SCM Churn, and Rails Stats)
    • CruiseControl.rb when you start using all of these tools, continuous integration starts to become more important (or doing nightly runs). CruiseControl.rb is dead simple continuous integration.
    • Simian another code duplication tool, which is mentioned in 3 tools for drying your Ruby code (free for OSS, $99 for a license)
    • Ruby Tidy a tool for cleaning up HTML (I haven't used this in Ruby, but loved the Java version in my Java days)
    • Watir is an open-source library for automating web browsers. It allows you to write tests that are easy to read and maintain. It is simple and flexible.
    • Autotest, if you haven't heard of autotest, check it out, continuously run your tests every time you save a file in your project.
    • Rufus a tool that checks if code you are about to load is safe. Allows you to look for custom patterns that you don't want to run.
    • I wrote about a couple benchmarking tools last time and here is a great article / tutorial on Ruby benchmarking

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on December 3rd, 2008 by Dan in Development, Hacking, Misc, Ruby, Tools.

  • Building a iPhone web app in under 50 lines with Sinatra and iUI

    One awesome thing about the iPhone is that it can display documents very nicely, including Word, Excel, and PDF files. However, the other day I was complaining that it's not very easy to view the documents you store on your computer on your iPhone. Sure, you could email them to yourself, but then you have to search through your mail on your iPhone to find your documents. And I'm sure there is a snazzy iPhone app from the App Store to do this as well. But instead, let's build a quick web app using Sinatra and iUI.

    Here's what we'll be building (screenshot courtesy of iPhoney, which rules, by the way):

    A screenshot of the butler iPhone web app

    It doesn't look like much, but hey, it's less than 50 lines of code

    When you click on git-tutorial.pdf, you'll see the full document

    When you click on git-tutorial.pdf, you'll see this

    Sinatra is a really awesome minimalist web framework. It lets you build web applications with just a few lines of code. iUI is a collection of JavaScript, CSS, and images that lets you easily make your web sites look great on the iPhone. Using these two tools, it's really easy to build simple iPhone apps.

    To begin, install the Sinatra gem:


    > gem install sinatra

    Now, let's start with a simplest version of our app, which we'll call 'butler'. Let's make a directory for butler.


    > mkdir butler
    > cd butler
    > touch butler.rb

    Open up butler.rb in your favorite editor and type:

    Now start butler on your command line: > ruby -rubygems ./butler.rb and point your browser to http://localhost:4567 (you can use your computer's browser or the one on your iPhone - it doesn't matter. I find it's better to use the one on my computer while building the app, since it's easier to read Sinatra's debugging messages if something goes wrong). You should see a page that just says "Your files, sir." Congrats! You've made your first Sinatra app. Wasn't that easy?

    OK, let's make butler a little more useful. Sinatra will serve up any files in a subdirectory named public. Since we'll eventually be using this public directory for holding other JavaScript and CSS files as well, we'll actually put our files in ./public/files. We'll also make a link for convenience. Finally, while we're at it, let's put a few test files in there.

    > mkdir -p public/files
    > ln -s public/files files
    > echo "foo" > public/files/foo.txt
    > echo "bar" > public/files/bar.txt

    We want butler to link to each file, so let's build a little helper for that. In Sinatra, you can include helpers within a helper block. We'll also try out our helper for one file.

    Go refresh your browser to see the changes. There's no need to restart your application, because Sinatra automatically reloads changes (very cool!). You should see a link to foo.txt. Click on it, and you'll see the contents.

    Clearly, we don't want to hardcode this for just one file. Let's alter butler to look for every file within the ./files directory.

    OK, refresh your browser and you should see both foo.txt and bar.txt. This is looking pretty good, but we're not really creating valid HTML right now. We're missing html, head, and, body tags at the very least. We could add this all within our "get" handler, but that would clutter up the code.

    Instead, let's put this code into a view. Sinatra actually lets you put the view right after your other code, so you can build an entire application in one file. For simplicity, I'm going to do that for this tutorial. However, if this approach bothers you (or just messes up syntax highlighting in your editor), rest assured you can place the view code in a views directory and it would work the same way.

    Let's add the view to the end of our file, and use it in our handler. Notice that I name the view 'index' by beginning my declaration with @@ index - if I wanted a separate file, I would just put it in ./views/index.erb (you can also use Haml, if that's your cup of tea). Note I assign @links in the handler and it automatically is available in the view.

    Refreshing the browser now isn't really that exciting, since things look the same, but if you wanted, you could easily play around with the view to make things look different.

    One glaring problem is that this page isn't very usable on the iPhone itself. That's where iUI comes in. Start by downloading it (URL is in instructions below) to your butler directory, unzipping it, and copying the necessary files into your public directory.


    > mkdir iui
    > cd iui
    > wget http://iui.googlecode.com/files/iui-0.13.tar.gz
    > tar -xzvf iui-0.13.tar.gz
    > cd ..
    > mkdir public/images
    > cp iui/iui/*.png public/images
    > cp iui/iui/*.gif public/images
    > mkdir public/javascripts
    > cp iui/iui/*.js public/javascripts
    > mkdir public/stylesheets
    > cp iui/iui/*.css public/stylesheets

    To use iUI, you'll need to include the JavaScript and CSS in your view. You'll also need to add some elements to the body of your view. When you're done, the view will look like this:

    This html is probably a bit confusing, but don't worry. There are a few examples in ./iui/samples/ to learn from (and good iUI tutorials on the web). Finally, you'll want to alter the file_link helper to print out iUI code, like so:

    Note that target='_self' code. You need that to get iUI to open a link in a normal way. If you leave it off, it will use an AJAX call to load the file within the current page, which looks really funny when you try to open a binary file like a PDF.

    The final code looks like this:

    And there you have it - an iPhone web app in less than 50 lines of code, thanks to Sinatra and iUI. Now, whenever you want to view some files on your iPhone, either copy the file:

    > cp path/to/my_file ./files

    or if you prefer, link it:

    ln -s path/to/my_file ./files

    ... and then run butler

    ruby -rubygems ./butler.rb

    Figure out the IP address of your computer and simply point your iPhone browser to http://<ip>:4567

    I use butler primarily within my home network, but if you want to be able to view your files on the go, you'll need to poke a hole in your firewall. That's a bit outside the scope of this tutorial, but a quick Google search should give you some good results.

    Enjoy!

    Update: Removed an unused parameter from the code after pmccann called it to my attention.
    Update: Added -z option to tar after Peter pointed out the omission. The tar command without -z worked for me on OS X 10.5, but this is definitely more correct.
    Update: Added -rubygems option to ruby command. If you'd prefer to not use this option, check the comments below for ways to use RubyGems in a Ruby script.

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on November 25th, 2008 by Ben in Hacking, Tools.

  • Ruby Beanstalkd distributed worker intermediate lessons

    This post is a follow up to Ruby beanstalkd basics, I will try to make the example code little more interesting and useful. I am calling this is a Ruby beanstalkd intermediate write up, it sets up a few workers and distributes and receives results simultaneously. In this example the code resembles real code a bit more (using a queue cache and block passing). If there is enough interest in the Ruby/beanstalkd community, I will follow up with beanstalkd advanced lessons, and go into how we deal with failure cases such as worker dying during jobs, random jobs failing, processing multiple 'projects' at one time, using job priority settings, and using TTR/timeouts.

    So in this example we are making an estimate of PI. Yes I know that there are far better approximations out there than my simple results, but this was what I came up with for an incredibly simple distributed computing problem. I based my example on the PI Calculation problem from an Introduction to Parallel Computing. The basic idea is that you can calculate pi by guessing random points in a square and then seeing how many points are inside a circle that fits inside the square (PI= 4 * points_in_circle/total_points).

    I made a bunch of comments in the code that should help you follow but there are a few key sections worth pointing out.

    In the Ruby beanstalkd Basics, both the Server and the Clients only used one queue at a time. Now since we are sending on one queue while also listening on another we need access to both queues at once. We simply have a helper function with a queue_cache to make getting and reusing multiple queues incredibly easy.

    def get_queue(queue_name)
        @queue_cache ||= {}
        if @queue_cache.has_key?(queue_name)
          return @queue_cache[queue_name]
        else
          queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
          queue.watch(queue_name)
          queue.use(queue_name)
          queue.ignore('default')
          @queue_cache[queue_name] = queue
          return queue
        end
      end

    In the basic example each class had a function that got a job and did some work and deleted the job. It is easy to imagine workers that might have many different kinds of work to do on jobs. In every case they are going to grab a job, work on the job, and delete the job. We decided to break that up and make it easy to just pass a work block when workers get a job.

    def take_msg(queue)
        msg = queue.reserve
        #by calling ybody we get the content of the message and convert it from yml
        body = msg.ybody
        if block_given?
          yield(body)
        end
        msg.delete
      end
    
    #call take_msg like so
    take_msg(queue) do |body|
      #work on body
    end

    One other thing you should keep a look out for in the code below is checking if a queue has any jobs. Many times workers will check if jobs exist and take them, and if there aren't any jobs the process is free to do something else. I do this in this example, the server continually checks incoming results to immediately display. If no results have arrived yet, the server continues sending out job requests as fast as it can. This is useful since taking jobs from beanstalkd is a blocking call. They did add support for non-blocking calls in beanstalkd 1.1, but I haven't started using the newest version yet. I think everything else should be pretty self explanatory, feel free to ask me any questions. To run the code it is the same as before: download beanstalk_intermediate.rb, start beanstalkd, and run the example with ruby.

    $ beanstalkd &
    $ ruby beanstalk_intermediate.rb
    starting distributor
    starting client(s)
    distributor sending out jobs
    .......................................................
    .............................................
    received all the results our estimate for pi is: 3.142776

    # of workers time to complete
    1 real 0m7.282s
    user 0m4.114s
    sys 0m0.978s
    2 real 0m5.667s
    user 0m2.736s
    sys 0m0.670s
    3 real 0m4.999s
    user 0m2.014s
    sys 0m0.515s
    4 real 0m4.612s
    user 0m1.608s
    sys 0m0.442s
    5 real 0m4.517s
    user 0m1.474s
    sys 0m0.416s
    require 'beanstalk-client.rb'
    
    DEFAULT_PORT = 11300
    SERVER_IP = '127.0.0.1'
    #beanstalk will order the queues based on priority, with the same priority
    #it acts FIFO, in a later example we will use the priority
    #(higher numbers are higher priority)
    DEFAULT_PRIORITY = 65536
    #TTR is time for the job to reappear on the queue.
    #Assuming a worker died before completing work and never called job.delete
    #the same job would return back on the queue (in TTR seconds)
    TTR = 3
    
    class BeanBase
    
      #To work with multiple queues you must tell beanstalk which queues
      #you plan on writing to (use), and which queues you will reserve jobs from
      #(watch). In this case we also want to ignore the default queue
      #you need a different queue object for each tube you plan on using or
      #you can switch what the tub is watching and using a bunch, we just keep a few
      #queues open on the tubes we want.
      def get_queue(queue_name)
        @queue_cache ||= {}
        if @queue_cache.has_key?(queue_name)
          return @queue_cache[queue_name]
        else
          queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
          queue.watch(queue_name)
          queue.use(queue_name)
          queue.ignore('default')
          @queue_cache[queue_name] = queue
          return queue
        end
      end
    
      #this will take a message off the queue, and process it with the block
      def take_msg(queue)
        msg = queue.reserve
        #by calling ybody we get the content of the message and convert it from yml
        body = msg.ybody
        if block_given?
          yield(body)
        end
        msg.delete
      end
    
      def results_ready?(queue)
        queue.peek_ready!=nil
      end
    
    end
    
    class BeanDistributor < BeanBase
    
      def initialize(chunks,points_per_chunk)
        @chunks = chunks
        @points_per_chunk = points_per_chunk
        @messages_out = 0
        @circle_count = 0
      end
    
      def get_incoming_results(queue)
        if(results_ready?(queue))
          result = nil
          take_msg(queue) do |body|
            result = body.count
          end
          @messages_out -= 1
          print "." #display that we received another result
          @circle_count += result
        else
          #do nothing
        end
      end
    
      def start_distributor
        request_queue = get_queue('requests')
        results_queue = get_queue('results')
        #put all the work on the request queue
        puts "distributor sending out #{@messages} jobs"
        @chunks.times do |num|
          msg = BeanRequest.new(1,@points_per_chunk)
          #Take our ruby object and convert it to yml and put it on the queue
          request_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
          @messages_out += 1
          #if there are results get them if not continue sending out work
          get_incoming_results(results_queue)
        end
    
        while @messages_out > 0
          get_incoming_results(results_queue)
        end
        npoints = @chunks * @points_per_chunk
        pi = 4.0*@circle_count/(npoints)
        puts "\nreceived all the results our estimate for pi is: #{pi}"
      end
    
    end
    
    class BeanWorker < BeanBase
    
      def initialize()
      end
    
      def write_result(queue, result)
        msg = BeanResult.new(1,result)
        queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
      end
    
      def in_circle
        #generate 2 random numbers see if they are in the circle
        range = 1000000.0
        radius = range / 2
        xcord = rand(range) - radius
        ycord = rand(range) - radius
        if( (xcord**2) + (ycord**2) <= (radius**2) )
          return 1
        else
          return 0
        end
      end
    
      def start_worker
        request_queue = get_queue('requests')
        results_queue = get_queue('results')
        #get requests and do the work until the worker is killed
        while(true)
          result = 0
          take_msg(request_queue) do |body|
            chunks = body.count
            chunks.times { result += in_circle}
          end
          write_result(results_queue,result)
        end
    
      end
    
    end
    
    ############
    # These are just simple message classes that we pass using beanstalks
    # to yml and from yml functions.
    ############
    class BeanRequest
      attr_accessor :project_id, :count
      def initialize(project_id, count=0)
        @project_id = project_id
        @count = count
      end
    end
    
    class BeanResult
      attr_accessor :project_id, :count
      def initialize(project_id, count=0)
        @project_id = project_id
        @count = count
      end
    end
    
    #how many different jobs we should do
    chunks = 100
    #how many points to calculate per chunk
    points_per_chunk = 10000
    #how many workers should we have
    #(normally different machines, in our example fork them off)
    workers = 5
    
    # Most of the time you will have two entirely separate classes
    # but to make it easy to run this example we will just fork and start our server
    # and client separately. We will wait for them to complete and check
    # if we received all the messages we expected.
    puts "starting distributor"
    server_pid = fork {
      BeanDistributor.new(chunks,points_per_chunk).start_distributor
    }
    
    puts "starting client(s)"
    client_pids = []
    workers.times do |num|
      client_pid = fork {
        BeanWorker.new.start_worker
      }
      client_pids << client_pid
    end
    
    Process.wait(server_pid)
    #take down the clients
    client_pids.each do |pid|
      Process.kill("HUP",pid)
    end

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on November 19th, 2008 by Dan in Development, Hacking, Ruby.

  • Someone please build an awesome embeddable code widget

    One awesome thing about working at a startup is that you get to focus very deeply on the problem you're trying to solve. On the other hand, if you've taken the leap and founded a startup, it's probably because you tend to see solutions and opportunities everywhere. It can be really hard to focus on one thing when you often have ideas for services that you'd like to use, or better yet, build.

    Dan and I regulary talk about services that we wish existed but we simply can't work on due to our commitment to Devver. The other day we were discussing one problem we wish someone would solve: why can't we easily post nicely formatted code in our blog posts?

    All I want is this: I copy/paste some code into a web site, choose the programming language, copy some widget code and paste that code into my blog. The code is indented and formatted, has syntax coloration, wraps correctly (for any iPhone readers) and can be easy copied/pasted. Including line numbers (that don't mess with copy/paste) is a bonus.

    In other words, I just want Pastie in my blog posts.

    Yes, I know there are a few really nice projects that you can install on your server that will do all this. But we could all host our own video as well, but it's just easier to upload and embed a video on YouTube or Vimeo.

    This wouldn't just have to be for the good of humanity either. Such a service could make money off ads (each widget could have a link to the full-screen code on the main site, which could have ads for programming jobs, books, and conferences) or even sell off the data about which programming languages were most popular (in blogs and on the main site).

    Maybe there is a solution for this (if there is, please let me know in the comments. I'm more than willing to publicly display my ignorance in order to learn about it), but if there is, I don't see it widely being used and it's not easy to find on Google. If there isn't (yet), please go forth and build. I'll be anxiously waiting.

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on October 30th, 2008 by Ben in Development, Hacking.

  • Ruby Beanstalkd distributed worker basics

    At Devver we have a lot of jobs to do quickly, so we distribute our work out to a group of EC2 workers. We have tried and used a number of queuing solutions with Ruby, but in the end beanstalkd seemed to be the best solution for us at the time.

    I have only seen a few posts about the basics of using beanstalkd with Ruby. I decided to make two posts evolving a simple Ruby beanstalkd example into a more complicated example. This way people new to beanstalkd could see how easy it can be to get up and running with distributed processing using Ruby and beanstalkd. Then people that are doing more advanced work with beanstalkd could see some examples of how we are working with it here at Devver. It would also be great for more experienced beanstalkd warriors to share their thoughts as there aren't many examples out in the wild. The lack of examples makes it harder to learn and difficult to decide what the best practices are when working with beanstalkd queues.

    I have also shared two scripts we have found useful while working with beanstalkd. beanstalk_monitor.rb, which lets you see all the queue statistics about current usage, or to monitor the information of a single queue you are interested in. Finally, beanstalk_killer.rb, which is useful if you want to work on how your code will react to beanstalkd getting backed up or stalling (in beanstalkd speak, "Putting on the brakes"). It was a little harder to pull everything out and make a simple example from our code than I thought, and obviously the example is a bit useless. It should still give a solid example of how to do the basics of distributing jobs with beanstalkd.

    For those new to beanstalk, there are a few things you will need to know like how to get a queue object, how to put objects on the queue, how to take objects off the queue, and how to control which queue you are working with. For a higher level overview or more detailed information, I recommend checking out the beanstalkd FAQ. The full example code is below, but first taking a look at the basic snippets might help.

    #to work with beanstalk you need to get a client connection
    queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
    #by default you will be working on the 'default' tube or queue
    #if we wanted to work on a different queue we could change tubes, like so
    queue.watch('test_queue')
    queue.use('test_queue')
    queue.ignore('default')
    #to put a simple string on a queue
    queue.put('hello queue world')
    #to receive a simple string
    job = queue.reserve
    puts job.body #prints 'hello queue world'
    #if you don't delete the job when you're done, the queue assumes there is an error
    #and the job will show back up on the queue again
    job.delete
    

    How to run this example (on OS X, with macports installed)

    > sudo port install beanstalkd
    > sudo gem install beanstalk-client
    > beanstalkd
    > ruby beanstalk_tester.rb

    Download: beanstalk_tester.rb

    require 'beanstalk-client.rb'
    
    DEFAULT_PORT = 11300
    SERVER_IP = '127.0.0.1'
    #beanstalk will order the queues based on priority, with the same priority
    #it acts FIFO, in a later example we will use the priority
    #(higher numbers are higher priority)
    DEFAULT_PRIORITY = 65536
    #TTR is time for the job to reappear on the queue.
    #Assuming a worker died before completing work and never called job.delete
    #the same job would return back on the queue (in seconds)
    TTR = 3
    
    class BeanBase
    
      #To work with multiple queues you must tell beanstalk which queues
      #you plan on writing to (use), and which queues you will reserve jobs from
      #(watch). In this case we also want to ignore the default queue
      def get_queue(queue_name)
        queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
        queue.watch(queue_name)
        queue.use(queue_name)
        queue.ignore('default')
        queue
      end
    
    end
    
    class BeanDistributor < BeanBase
    
      def initialize(amount)
        @messages = amount
      end
    
      def start_distributor
        #put all the work on the request queue
        bean_queue = get_queue('requests')
        @messages.times do |num|
          msg = BeanRequest.new(1,num)
          #Take our ruby object and convert it to yml and put it on the queue
          bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
        end
    
        puts "distributor now getting results"
        #get all the results from the results queue
        bean_queue = get_queue('results')
        @messages.times do |num|
          result = take_msg(bean_queue)
          puts "result: #{result}"
        end
    
      end
    
      #this will take a message off the queue, process it and return the result
      def take_msg(queue)
        msg = queue.reserve
        #by calling ybody we get the content of the message and convert it from yml
        count = msg.ybody.count
        msg.delete
        return count
      end
    
    end
    
    class BeanWorker < BeanBase
    
      def initialize(amount)
        @messages = amount
        @received_msgs = 0
      end
    
      def start_worker
        results = []
        #get and process all the requests, on the requests queue
        bean_queue = get_queue('requests')
        @messages.times do |num|
          result = take_msg(bean_queue)
          results << result
          @received_msgs += 1
        end
    
        #return all of the results, by placing them on the separate results queue
        bean_queue = get_queue('results')
        results.each do |result|
          msg = BeanResult.new(1,result)
          bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
        end
    
        #this is just to pass information out of the forked process
        #we return the number of messages we received as our exit status
        exit @received_msgs
      end
    
      #this will take a message off the queue, process it and return the result
      def take_msg(queue)
        msg = queue.reserve
        #by calling ybody we get the content of the message and convert it from yml
        count = msg.ybody.count
        result = count*count
        msg.delete
        return result
      end
    
    end
    
    ############
    # These are just simple message classes that we pass using beanstalks
    # to yml and from yml functions.
    ############
    class BeanRequest
      attr_accessor :project_id, :count
      def initialize(project_id, count=0)
        @project_id = project_id
        @count = count
      end
    end
    
    class BeanResult
      attr_accessor :project_id, :count
      def initialize(project_id, count=0)
        @project_id = project_id
        @count = count
      end
    end
    
    #write X messages on the queue
    numb = 10
    
    recv_count = 0
    
    # Most of the time you will have two entirely seperate classes
    # but to make it easy to run this example we will just fork and start our server
    # and client seperately. We will wait for them to complete and check
    # if we received all the messages we expected.
    puts "starting distributor"
    server_pid = fork {
      BeanDistributor.new(numb).start_distributor
    }
    
    puts "starting client"
    client_pid = fork {
      BeanWorker.new(numb).start_worker
    }
    
    Process.wait(client_pid)
    recv_count = $?.exitstatus
    puts "client finished received #{recv_count} msgs"
    if(numb==recv_count)
      puts "received the expected number of messages"
    else
      puts "error didn't receive the correct number of messages"
    end
    
    Process.wait(server_pid)
    

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on October 28th, 2008 by Dan in Development, Devver, Hacking, Tips & Tricks.

  • Tracking down open files with lsof

    The other day I was running in a weird error on Devver. After running around twenty test runs on the system, the component that actually runs individual unit tests was crashing due to "Too many open files - (Errno::EMFILE)"

    Unfortunately, I didn't know much more than that. Which files were being kept open? I knew that this component loaded quite a few files, and that by default, OS X only allows 256 open file descriptors (ulimit -n will tell you the default on your system). If this was a valid case of needing to load more files, I could just up the limit using ulimit -n <bigger_number>.

    Fortunately, a quick Google or two pointed the way to lsof. Unfortunately, my Unix-fu is never nearly as good as I wish and I didn't know much about this handy utility. But I quickly discovered that it's very useful for tracking down problems like this. I quickly used ps to find the PID of the Devver process and then a quick lsof -p <PID> displayed all the files that the process had open. So easy!

    Sure enough, there were a ton of redundant file handles to the file that we use to store information about the Devver run. Armed with this information, it was easy to find the buggy code where we called File.open but failed to ever close the file.

    Unfortunately, I still don't know how to write a good unit test for this case. I guess I could do something ugly like call sytem("lsof -p pid | wc -l") before and after calling the code and make sure the number of descriptors stays constant, but that's really ugly. Is there a way to test this within Ruby? I'm open to ideas.

    Still, it's always good to learn more about a powerful Unix tool. I'm constanly amazed by the power and depth of the Unit tool set.

    Got a slow Test::Unit or RSpec suite?
    Devver can run it up to three times faster! Request a beta invite today.

    Posted on October 9th, 2008 by Ben in Development, Hacking, Testing, Tips & Tricks.