So in this example we are making an estimate of PI. Yes I know that there are far better approximations out there than my simple results, but this was what I came up with for an incredibly simple distributed computing problem. I based my example on the PI Calculation problem from an Introduction to Parallel Computing. The basic idea is that you can calculate pi by guessing random points in a square and then seeing how many points are inside a circle that fits inside the square (PI= 4 * points_in_circle/total_points).

I made a bunch of comments in the code that should help you follow but there are a few key sections worth pointing out.

In the Ruby beanstalkd Basics, both the Server and the Clients only used one queue at a time. Now since we are sending on one queue while also listening on another we need access to both queues at once. We simply have a helper function with a queue_cache to make getting and reusing multiple queues incredibly easy.

def get_queue(queue_name)
    @queue_cache ||= {}
    if @queue_cache.has_key?(queue_name)
      return @queue_cache[queue_name]
    else
      queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
      queue.watch(queue_name)
      queue.use(queue_name)
      queue.ignore('default')
      @queue_cache[queue_name] = queue
      return queue
    end
  end

In the basic example each class had a function that got a job and did some work and deleted the job. It is easy to imagine workers that might have many different kinds of work to do on jobs. In every case they are going to grab a job, work on the job, and delete the job. We decided to break that up and make it easy to just pass a work block when workers get a job.

def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    body = msg.ybody
    if block_given?
      yield(body)
    end
    msg.delete
  end

#call take_msg like so
take_msg(queue) do |body|
  #work on body
end

One other thing you should keep a look out for in the code below is checking if a queue has any jobs. Many times workers will check if jobs exist and take them, and if there aren’t any jobs the process is free to do something else. I do this in this example, the server continually checks incoming results to immediately display. If no results have arrived yet, the server continues sending out job requests as fast as it can. This is useful since taking jobs from beanstalkd is a blocking call. They did add support for non-blocking calls in beanstalkd 1.1, but I haven’t started using the newest version yet. I think everything else should be pretty self explanatory, feel free to ask me any questions. To run the code it is the same as before: download beanstalk_intermediate.rb, start beanstalkd, and run the example with ruby.

$ beanstalkd &
$ ruby beanstalk_intermediate.rb
starting distributor
starting client(s)
distributor sending out jobs
.......................................................
.............................................
received all the results our estimate for pi is: 3.142776

require 'beanstalk-client.rb'

DEFAULT_PORT = 11300
SERVER_IP = '127.0.0.1'
#beanstalk will order the queues based on priority, with the same priority
#it acts FIFO, in a later example we will use the priority
#(higher numbers are higher priority)
DEFAULT_PRIORITY = 65536
#TTR is time for the job to reappear on the queue.
#Assuming a worker died before completing work and never called job.delete
#the same job would return back on the queue (in TTR seconds)
TTR = 3

class BeanBase

  #To work with multiple queues you must tell beanstalk which queues
  #you plan on writing to (use), and which queues you will reserve jobs from
  #(watch). In this case we also want to ignore the default queue
  #you need a different queue object for each tube you plan on using or
  #you can switch what the tub is watching and using a bunch, we just keep a few
  #queues open on the tubes we want.
  def get_queue(queue_name)
    @queue_cache ||= {}
    if @queue_cache.has_key?(queue_name)
      return @queue_cache[queue_name]
    else
      queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
      queue.watch(queue_name)
      queue.use(queue_name)
      queue.ignore('default')
      @queue_cache[queue_name] = queue
      return queue
    end
  end

  #this will take a message off the queue, and process it with the block
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    body = msg.ybody
    if block_given?
      yield(body)
    end
    msg.delete
  end

  def results_ready?(queue)
    queue.peek_ready!=nil
  end

end

class BeanDistributor < BeanBase

  def initialize(chunks,points_per_chunk)
    @chunks = chunks
    @points_per_chunk = points_per_chunk
    @messages_out = 0
    @circle_count = 0
  end

  def get_incoming_results(queue)
    if(results_ready?(queue))
      result = nil
      take_msg(queue) do |body|
        result = body.count
      end
      @messages_out -= 1
      print "." #display that we received another result
      @circle_count += result
    else
      #do nothing
    end
  end

  def start_distributor
    request_queue = get_queue('requests')
    results_queue = get_queue('results')
    #put all the work on the request queue
    puts "distributor sending out #{@messages} jobs"
    @chunks.times do |num|
      msg = BeanRequest.new(1,@points_per_chunk)
      #Take our ruby object and convert it to yml and put it on the queue
      request_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
      @messages_out += 1
      #if there are results get them if not continue sending out work
      get_incoming_results(results_queue)
    end

    while @messages_out > 0
      get_incoming_results(results_queue)
    end
    npoints = @chunks * @points_per_chunk
    pi = 4.0*@circle_count/(npoints)
    puts "\nreceived all the results our estimate for pi is: #{pi}"
  end

end

class BeanWorker < BeanBase

  def initialize()
  end

  def write_result(queue, result)
    msg = BeanResult.new(1,result)
    queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
  end

  def in_circle
    #generate 2 random numbers see if they are in the circle
    range = 1000000.0
    radius = range / 2
    xcord = rand(range) - radius
    ycord = rand(range) - radius
    if( (xcord**2) + (ycord**2) <= (radius**2) )
      return 1
    else
      return 0
    end
  end

  def start_worker
    request_queue = get_queue('requests')
    results_queue = get_queue('results')
    #get requests and do the work until the worker is killed
    while(true)
      result = 0
      take_msg(request_queue) do |body|
        chunks = body.count
        chunks.times { result += in_circle}
      end
      write_result(results_queue,result)
    end

  end

end

############
# These are just simple message classes that we pass using beanstalks
# to yml and from yml functions.
############
class BeanRequest
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

class BeanResult
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

#how many different jobs we should do
chunks = 100
#how many points to calculate per chunk
points_per_chunk = 10000
#how many workers should we have
#(normally different machines, in our example fork them off)
workers = 5

# Most of the time you will have two entirely separate classes
# but to make it easy to run this example we will just fork and start our server
# and client separately. We will wait for them to complete and check
# if we received all the messages we expected.
puts "starting distributor"
server_pid = fork {
  BeanDistributor.new(chunks,points_per_chunk).start_distributor
}

puts "starting client(s)"
client_pids = []
workers.times do |num|
  client_pid = fork {
    BeanWorker.new.start_worker
  }
  client_pids << client_pid
end

Process.wait(server_pid)
#take down the clients
client_pids.each do |pid|
  Process.kill("HUP",pid)
end

Shazam is a pretty cool app on the iPhone to help you recognize and find music, it uses the microphone and listens to whatever music is near the iPhone. Shazam then compares the few second recorded sound to an online database of music data to try to determine the song. I was curious what would happen when it was given mashup music which contains parts of many songs. Would it just be entirely confused? Would it recognize the mashup? Would it recognize the currently most prominent sample in the mashup?

Conclusion
It was far more noticeable when the dominant sample was playing during both Jay-Zeezer and The Grey Album which seemed to mean that Shazam could pick out the individual songs easier. I was quite impressed when it even correctly identified some of the Grey Album as actually being a mashup. Girl Talk gave the program a harder time, which makes sense seeing as many samples are put together for very small amounts of times and many of the samples are slightly altered. Some songs where there were dominant samples for a period of time, it did a decent job. It is a cool way to figure out what songs contribute to a mashup if you are unsure about a piece of a song. I was pretty impressed with Shazam’s abilities to pick apart the pieces of the songs. I guess the results aren’t really that surprising, but it was still a fun way to spend some of a Saturday afternoon listening to music and testing some software.

Now we can just wrap code in <pre name='code' class='ruby'>...some code...</pre> and it will look like this:

class Foo
   def bar
     puts "Hello, world!"
   end
end

Pretty sweet! In fact, we ended up converting all of the code snippets on our blog to use SH.

Although SH is really, really cool, it does suffer from inherent drawbacks: it has to be installed and it has to be periodically updated. What I initially described in my previous post was something simpler - a web app where you could easily post code and then embed it in your blog. In other words, Pastie with a YouTube-like embed feature.

Dirceu Jr. pointed out Gist from GitHub, which is exactly what I was looking for. Even without signing up, you can post code (like so) and then embed it in your blog.

Dirceu Jr. even wrote a WordPress plugin if you really want to install something! The only weird thing is that while the embedded code is nicely syntax-highlighted, the original gist is not for some reason. Update: Yoan pointed out that you can get syntax-coloring by naming your gist properly (I renamed my gist foo.rb and it worked). It’s still not clear to me why it doesn’t work if I manually select ‘Ruby’ as the language for an unnamed gist, however.


Finally, while SyntaxHighlighter supports the most popular languages (and therefore is likely to be fine for most programming blogs), Gist supports a truly insane number of languages, making it a better choice if you want to post code snippets of say, Lua, Erlang, or Haskell.

Both of these projects provide great-looking ways to display code on your blog. If you know of other projects that you like, let me know in the comments.

This week, the group had Todd Vernon from Lijit come lead the discussion. Although Todd is currently CEO of Lijit, he was CTO at his former company, Raindance.

The group that was assembled was small but awesome - I had the opportunity to learn not only from Todd, but also from the CTOs of a few of last years TechStars companies.

The discussion touched on a ton of topics, but two (related) themes that were heavily discussed were the role of the CTO and how a company grows from a technology perspective. I’ve organized my notes below. Keep in mind that these are the collected thoughts from a number of different participants and I may not have captured their ideas with 100% accuracy.

The Role of a CTO

What is the difference between a CTO and a VP of Engineering?

CTO is about leadership for technical issues, interfacing with the business side, guiding the product, get people excited about product from a technical point of view.

VPoE is almost one step above Chief Architect, more on a management side, getting product delivered.

1st time CTOs need to figure out their exact role. It’s a very amorphous role, depending on the company.

CTO needs to be able to tell the whole business story, understand the good parts and bad.

Early on, CTO should insert themselves into the sales process as much as possible (especially right after you hire the sales person). You need to be able to hear what customers say they want, so you can translate that into what they really need.

The Technology Onion

There is a technology onion - make sure the core of the onion is owned by company (the outer layers, not as important). The CTO needs to figure out the relationship between the technologies and the company’s partnerships. The closer a technology is to the core of the onion, the more important it is to own it and to make sure it scales.

For instance, if you’re depending on Google for search, you’re powerless to change features if things don’t work as your customers want/expect. If search is core to your business (near the core of the onion), consider building it internally. It’s the CTOs role to make that case, because business people will never understand the need to spend money to get “the same thing.”

Having to re-architect a core component of a company can really hurt growth. Assume you’re going to be successful, so plan for that.

Along the same lines, one concern about using EC2 is that you get tied to the platform and your business is dependent on an outside force you can’t control. Hosting on EC2 can be quite different than hosting your own boxes.

Acceptable Failures

What is acceptable downtime? It depends on when - between midnight to 1-2 AM, it might be OK to be down for a few minutes. CTOs need to determine what acceptable downtime is and tell that the to the rest of management and have people agree.

CTOs need to make decisions (for instance, what is the acceptable down time, acceptable data loss, or acceptable time for page load) and then tell the entire organization. That way, when something bad happens, you can explain that everyone agreed on the specific numbers. It’s unlikely your business will need to be (or can be) 100% perfect on all metrics, but people need to understand what the goal is and why it’s realistic.

Growing/Scaling/Monitoring

If at least one person is using your service, you should have two web servers. It gives you ton of flexibility. Having two boxes forces you to work out most of the issues early (it’s a lot different getting to 2 boxes than 3 or 4). It’s not about load, it’s about reliability.

No matter how useful you are, if you are not reliable, someone will blog, “it’s cool, but it doesn’t work reliably.”

Downtime spreads very fast across Twitter. Consider tweeting about upcoming service interruptions ahead of time so customers are aware.

After more than 15 people, you need a dedicated operations person. Get some basic monitoring services early - after a server is under load, it’s really hard to diagnose. Try to detect stuff early, it’s easier to debug.

With startups, generally the problem tends to be slow requests rather than complete service downtime. Make sure your monitoring service will alert you with slow requests.

Get app specific stuff - a warning like “High CPU load,” is harder to understand (it might be a problem, or maybe the machine is just handling a lot of requests successfully), but “Page X takes 80 sec to load” is more obvious.

As you grow, try to measure more and more. Things often degrade slowly, and one day you just notice its too slow and it’s hard to go back and find the root of the problem.

Make two lists: a) the most catastrophic things that could happen and b) the most likely things that could happen. Where those lists overlap, you need to fix something. But there will be some risks that you decide are reasonable risks for the business (revisit these risks regularly as things change).

Regarding backup - always make sure you try to restore some data (before you really need it). You need to make sure it works and make sure its fast enough.

You should always be able to describe at a high level how the service will scale infinitely (it doesn’t have to be technically perfect, but it has to be believable). When someone wants to purchase, that’ll be a huge help - the business guy on the other side of the table will want to buy, but the technical guy doesn’t want to buy (he wants to build it in-house).

Dan and I regulary talk about services that we wish existed but we simply can’t work on due to our commitment to Devver. The other day we were discussing one problem we wish someone would solve: why can’t we easily post nicely formatted code in our blog posts?

All I want is this: I copy/paste some code into a web site, choose the programming language, copy some widget code and paste that code into my blog. The code is indented and formatted, has syntax coloration, wraps correctly (for any iPhone readers) and can be easy copied/pasted. Including line numbers (that don’t mess with copy/paste) is a bonus.

In other words, I just want Pastie in my blog posts.

Yes, I know there are a few really nice projects that you can install on your server that will do all this. But we could all host our own video as well, but it’s just easier to upload and embed a video on YouTube or Vimeo.

This wouldn’t just have to be for the good of humanity either. Such a service could make money off ads (each widget could have a link to the full-screen code on the main site, which could have ads for programming jobs, books, and conferences) or even sell off the data about which programming languages were most popular (in blogs and on the main site).

Maybe there is a solution for this (if there is, please let me know in the comments. I’m more than willing to publicly display my ignorance in order to learn about it), but if there is, I don’t see it widely being used and it’s not easy to find on Google. If there isn’t (yet), please go forth and build. I’ll be anxiously waiting.

I have only seen a few posts about the basics of using beanstalkd with Ruby. I decided to make two posts evolving a simple Ruby beanstalkd example into a more complicated example. This way people new to beanstalkd could see how easy it can be to get up and running with distributed processing using Ruby and beanstalkd. Then people that are doing more advanced work with beanstalkd could see some examples of how we are working with it here at Devver. It would also be great for more experienced beanstalkd warriors to share their thoughts as there aren’t many examples out in the wild. The lack of examples makes it harder to learn and difficult to decide what the best practices are when working with beanstalkd queues.

I have also shared two scripts we have found useful while working with beanstalkd. beanstalk_monitor.rb, which lets you see all the queue statistics about current usage, or to monitor the information of a single queue you are interested in. Finally, beanstalk_killer.rb, which is useful if you want to work on how your code will react to beanstalkd getting backed up or stalling (in beanstalkd speak, “Putting on the brakes”). It was a little harder to pull everything out and make a simple example from our code than I thought, and obviously the example is a bit useless. It should still give a solid example of how to do the basics of distributing jobs with beanstalkd.

For those new to beanstalk, there are a few things you will need to know like how to get a queue object, how to put objects on the queue, how to take objects off the queue, and how to control which queue you are working with. For a higher level overview or more detailed information, I recommend checking out the beanstalkd FAQ. The full example code is below, but first taking a look at the basic snippets might help.

#to work with beanstalk you need to get a client connection
queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
#by default you will be working on the 'default' tube or queue
#if we wanted to work on a different queue we could change tubes, like so
queue.watch('test_queue')
queue.use('test_queue')
queue.ignore('default')
#to put a simple string on a queue
queue.put('hello queue world')
#to receive a simple string
job = queue.reserve
puts job.body #prints 'hello queue world'
#if you don't delete the job when you're done, the queue assumes there is an error
#and the job will show back up on the queue again
job.delete

How to run this example (on OS X, with macports installed)

> sudo port install beanstalkd
> sudo gem install beanstalk-client
> beanstalkd
> ruby beanstalk_tester.rb

Download: beanstalk_tester.rb

require 'beanstalk-client.rb'

DEFAULT_PORT = 11300
SERVER_IP = '127.0.0.1'
#beanstalk will order the queues based on priority, with the same priority
#it acts FIFO, in a later example we will use the priority
#(higher numbers are higher priority)
DEFAULT_PRIORITY = 65536
#TTR is time for the job to reappear on the queue.
#Assuming a worker died before completing work and never called job.delete
#the same job would return back on the queue (in seconds)
TTR = 3

class BeanBase

  #To work with multiple queues you must tell beanstalk which queues
  #you plan on writing to (use), and which queues you will reserve jobs from
  #(watch). In this case we also want to ignore the default queue
  def get_queue(queue_name)
    queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
    queue.watch(queue_name)
    queue.use(queue_name)
    queue.ignore('default')
    queue
  end

end

class BeanDistributor < BeanBase

  def initialize(amount)
    @messages = amount
  end

  def start_distributor
    #put all the work on the request queue
    bean_queue = get_queue('requests')
    @messages.times do |num|
      msg = BeanRequest.new(1,num)
      #Take our ruby object and convert it to yml and put it on the queue
      bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
    end

    puts "distributor now getting results"
    #get all the results from the results queue
    bean_queue = get_queue('results')
    @messages.times do |num|
      result = take_msg(bean_queue)
      puts "result: #{result}"
    end

  end

  #this will take a message off the queue, process it and return the result
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    count = msg.ybody.count
    msg.delete
    return count
  end

end

class BeanWorker < BeanBase

  def initialize(amount)
    @messages = amount
    @received_msgs = 0
  end

  def start_worker
    results = []
    #get and process all the requests, on the requests queue
    bean_queue = get_queue('requests')
    @messages.times do |num|
      result = take_msg(bean_queue)
      results << result
      @received_msgs += 1
    end

    #return all of the results, by placing them on the separate results queue
    bean_queue = get_queue('results')
    results.each do |result|
      msg = BeanResult.new(1,result)
      bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
    end

    #this is just to pass information out of the forked process
    #we return the number of messages we received as our exit status
    exit @received_msgs
  end

  #this will take a message off the queue, process it and return the result
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    count = msg.ybody.count
    result = count*count
    msg.delete
    return result
  end

end

############
# These are just simple message classes that we pass using beanstalks
# to yml and from yml functions.
############
class BeanRequest
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

class BeanResult
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

#write X messages on the queue
numb = 10

recv_count = 0

# Most of the time you will have two entirely seperate classes
# but to make it easy to run this example we will just fork and start our server
# and client seperately. We will wait for them to complete and check
# if we received all the messages we expected.
puts "starting distributor"
server_pid = fork {
  BeanDistributor.new(numb).start_distributor
}

puts "starting client"
client_pid = fork {
  BeanWorker.new(numb).start_worker
}

Process.wait(client_pid)
recv_count = $?.exitstatus
puts "client finished received #{recv_count} msgs"
if(numb==recv_count)
  puts "received the expected number of messages"
else
  puts "error didn't receive the correct number of messages"
end

Process.wait(server_pid)

This is my list of apps and solutions I want for the iPhone. They don’t have to all necessarily be native apps. If you know of an web-app (that works on the iPhone) which provides the functionality I am looking for, let me know.

I want something like Brain Age for the GameBoy on the iPhone. Spending a few minutes doing little puzzles and math when I have downtime seems like a better use of my time then just playing random games. I have Brain Tuner, which is nice, but I want some more options/different puzzles.

I want a Google Contacts to Apple contacts one-time syncer, All my iPhone contacts are missing their emails. All my Gmail contacts are missing their phone numbers, someone help me sync that up.

I want full syncing between my Google calendar and the native apple calendar app. I always had this and it was really easy to do on pocket PC. I want a full 2 way sync. Google and Apple seem pretty buddy buddy, so get on this.

I want flash cards, with prebuilt decks. I would like to be able to work my way through some word decks building my vocabulary. I also would love to have some Spanish/English decks. I am working on improving my Spanish by listening to Coffee Break Spanish, and having a Spanish study deck would be great.

I want an EBook reader - oh wait, someone just pointed one out to me (thanks Matt) it is called Stanza. It’s seriously awesome, if they add a screen reader, it would be perfect. I could listen to some classics books while jogging/driving.

I want GPS tracking that works. I have a great iPhone app called Trailguru, which tracks movement/location with GPS and can tell me the speed and total distance I go when I run. The problem is my GPS seems to stop working after a day or two, and won’t come back until I restart the iPhone or re-sync. Then I want a driving direction helper, something that says out loud, “turn approaching in 200 feet,” just like every in-car navigation system.

I want an on-the-go web reader (Ben shared this idea with me). This would offer a way to open/transfer all the tabs or URLs currently in my browser over to the iPhone. Preferably, it would open them all in an offline mode allowing me to then read the articles through out my day, while being on the move. I really would like if this wasn’t even in Safari since that crashes the iPhone too often.

I want a full Flash player, or at the very least the ability to play Flash videos. There are so many sites with Flash videos and streaming video that are useless on the iPhone. I really wanted to watch the debates on my iPhone because I had to leave in the middle, but while every site on the internet was streaming the debates, not one of them had a way to view the debate on an iPhone. Apparently Adobe has confirmed working on Flash, but Apple is likely to block it. Screw you Apple, even Pocket PC phones years ago had Flash.

I want streaming internet radio. Yes, Pandora and a few others are nice, but why can’t I just browse and listen to any streaming net radio station? It would be even better if it could allow me to browse many of the well known stations (shoutcast), with out having to search through iPhone’s browser.

I want something similar to Elasticfox for managing and monitoring EC2 instances on the iPhone. Actually beyond that it would be cool to be able to manage a few scripts and SSH credentials on a site. It wouldn’t allow arbitrary SSH, but you could store ssh login keys and a few scripts it could run and return the results. This would allow you to ahead of time write some scripts to monitor, clean up, restart or do other tasks, which you could then execute and verify the results of remotely over your iPhone. A sysadmin’s dream, until then I have pTerm for slow clumsy SSH.

That is all I have for now, but if you have thought of iPhone apps you would like to have leave some comments. If you know of a solution to any of the problems I mention above let me know. Some of these apps I would be willing to pay for so developers get busy.

Unfortunately, I didn’t know much more than that. Which files were being kept open? I knew that this component loaded quite a few files, and that by default, OS X only allows 256 open file descriptors (ulimit -n will tell you the default on your system). If this was a valid case of needing to load more files, I could just up the limit using ulimit -n <bigger_number>.

Fortunately, a quick Google or two pointed the way to lsof. Unfortunately, my Unix-fu is never nearly as good as I wish and I didn’t know much about this handy utility. But I quickly discovered that it’s very useful for tracking down problems like this. I quickly used ps to find the PID of the Devver process and then a quick lsof -p <PID> displayed all the files that the process had open. So easy!

Sure enough, there were a ton of redundant file handles to the file that we use to store information about the Devver run. Armed with this information, it was easy to find the buggy code where we called File.open but failed to ever close the file.

Unfortunately, I still don’t know how to write a good unit test for this case. I guess I could do something ugly like call sytem(”lsof -p pid | wc -l”) before and after calling the code and make sure the number of descriptors stays constant, but that’s really ugly. Is there a way to test this within Ruby? I’m open to ideas.

Still, it’s always good to learn more about a powerful Unix tool. I’m constanly amazed by the power and depth of the Unit tool set.

I got some help from people on the EventMachine mailing list, here is the thread discussing sending large files with EventMachine.

Since I was already playing around with a few of our options, I decided to do some comparisons between using EM.send_data, EM.stream_file_data, an alternative buffer recommended by James Tucker, our crappy buffer we have been using before we discovered the default EM BufferedTokenizer, and layering compression on top of the various methods. We had hacked together our buffer tokenizer rather quickly, and it always performed well enough in our initial testing, but it shows why performance tests are worth a little bit of effort. The benchmarks on the various setups are below (This was all done on localhost, it is worth noting that compression helps much more between remote servers).

Sending log file with compression turned off (5 times)
OurBadBufferedTokenizer: 10.40 s
Standard EM BufferedTokenizer: 0.93 s
Tucker’s BufferedTokenizer: 0.92 s
steam_file_data w/ EM BufferedTokenizer: 0.98 s

Sending log file with compression turned on (5 times)
OurBadBufferedTokenizer: 1.02 s
Standard EM BufferedTokenizer: 0.99 s
Tucker’s BufferedTokenizer: 1.04 s
steam_file_data w/ EM BufferedTokenizer: N/A can’t use stream_file_data with on the fly compression

Sending compressed Mp3 file with compression turned off (5 times)
OurBadBufferedTokenizer: 18.55 s
Standard EM BufferedTokenizer: 1.09 s
Tucker’s BufferedTokenizer: 1.10 s
steam_file_data w/ EM BufferedTokenizer: 1.22 s

Adding compression to already compressed files like mp3s doesn’t change the time significantly. This is a longer run just to show how the times vary with a larger test. I also tested on full projects and the variance seemed to hold.
Sending compressed Mp3 file with compression turned off (25 times)
OurBadBufferedTokenizer: N/A takes too long
Standard EM BufferedTokenizer: 5.70 s
Tucker’s BufferedTokenizer: 4.38 s
steam_file_data w/ EM BufferedTokenizer: 4.82 s

Below are the little tests and examples I was working with. Obviously you won’t have the same files on your system or Tucker’s buffer, so I packed everything up as zip. To try everything out just download the EventMachine sending files tests zip. Then extract, and run ‘ruby em_send_file_test.rb’. Any thoughts or feedback are welcome, I am still learning the ins and outs of EventMachine so feel free to send me any tips.

dir = File.expand_path(File.dirname(__FILE__))
unless($LOAD_PATH.member?(dir))
  $LOAD_PATH.unshift(dir)
end

require 'test/unit'
require 'eventmachine'
require 'zlib'
require 'yaml'
require 'ruby-debug'
require 'buffered_tokenizer_pastie'
require 'benchmark'

Thread.abort_on_exception = true

SERVER_PORT = 7999
SERVER_IP = '127.0.0.1'
TOKEN = "|DEFAULTDELIMITED|"
#check with different types of files compression
#results varies a bunch for txt vs compressed like mp3
FILE_NAME = '~/development.log'
#FILE_NAME = '~/Blue.mp3'
COMPRESS = false
#COMPRESS = true

TIMES = 5

class EmClientExample < EventMachine::Connection

  def unbind
    puts "client connection has terminated"
  end

  def process(data)
    puts "client got data: #{data}"
    send_files() if data=="success"
    send(prepare("some_msg")) if data=="filesDone"
    send(prepare("quit")) if data=="ack"
    if(data=="goodbye")
      puts "Client successfully sent all data shutting down!!!!"
      EventMachine::stop_event_loop
    end
  end

  def send_files()
    puts "sending files"
    @files = Array.new(TIMES,[FILE_NAME, Time.now.to_s])
    send_files_loop
  end

  def send_files_loop
    if @files && @files.length > 0
      file = @files.shift
      EM.next_tick do
        send_file(file[0],file[1])
        send_files_loop
      end
    else
      puts "done syncing files"
      send(prepare("files_completed"))
    end
  end

  def send_file(path,mtime)
    puts "Syncing "+path
    contents = File.read(File.expand_path(path))
    contents = Zlib::Deflate.deflate(contents,Zlib::BEST_SPEED) if COMPRESS
    send(prepare("send_file #{path}, #{mtime}, content:#{contents}"))
  end

  def send(str)
    #puts "sending: #{str}"
    send_data str
  end

  def prepare(str)
    str+TOKEN
  end

  def self.push_start()
    EventMachine.connect(SERVER_IP,SERVER_PORT,self) do |c|
      c.send_files()
    end
  end

end

class EmClientExampleBadBuffer < EmClientExample

  attr_accessor :buffer

  def initialize(*args)
    super
    @buffer = DataBuffer.new
  end

  def receive_data(data)
    @buffer.append(data)
    while(command = @buffer.grab)
      process(command)
    end
  end

  def prepare(str)
    @buffer.prepare(str)
  end

end

class EmClientExampleBuffToken < EmClientExample

  def initialize(*args)
    super
    @recv_buffer = BufferedTokenizer.new(TOKEN)
  end

  def receive_data(data)
    @recv_buffer.extract(data).each do |m|
      process(m)
    end
  end

end

class EmClientExampleStreamBuffToken < EmClientExample

  def initialize(*args)
    super
    @recv_buffer = BufferedTokenizer.new(TOKEN)
  end

  def send_files_loop
    if @files && @files.length > 0
      file = @files.shift
      EM.next_tick do
        send_file(file[0],file[1])
      end
    else
      puts "done syncing files"
      send(prepare("files_completed"))
    end
  end

  def send_file(path,mtime)
    puts "Syncing "+path
    send("send_file #{path}, #{mtime}, content:")

    EM::Deferrable.future( stream_file_data(File.expand_path(path)) ) {
      send(prepare(""))
      send_files_loop
    }
  end

  def receive_data(data)
    @recv_buffer.extract(data).each do |m|
      process(m)
    end
  end

end

class EmClientExamplePastie < EmClientExample

  def initialize(*args)
    super
    @recv_buffer = BufferedTokenizerPastie.new(TOKEN)
  end

  def receive_data(data)
    @recv_buffer.extract(data).each do |m|
      process(m)
    end
  end

end

class EmServerExample < EventMachine::Connection

  def post_init
    if(@signature)
      client = Socket.unpack_sockaddr_in(get_peername)
      puts "Received a new connection from #{client.last}:#{client.first}"
    end
  end

  def unbind
    puts "server connection has terminated\n"
  end

  def process(data)
    #puts "server: #{data[0..15]}"
    send(prepare("success")) if data=="login"
    send(prepare("filesDone")) if data=="files_completed"
    send(prepare("ack")) if data=="some_msg"
    if data.match(/^send_file/)
      #puts data[0..40]
      puts "received file"
      start = data.index(", content:") + ", content:".length
      ender = data.length
      contents = data[start,ender]
      contents = Zlib::Inflate.inflate(contents) if COMPRESS
      file_contents = File.read(File.expand_path(FILE_NAME))
      if contents != file_contents
        puts "file was corrupted"
        puts "received length: #{contents.length} file lenght: #{file_contents.length}"
        #File.open(File.expand_path("~/copy.file"),"w") do |f|
        #  f << contents
        #end
      end
    end
    if data=="quit"
      send(prepare("goodbye"))
      close_connection_after_writing
    end
  end

  def prepare(str)
    str+TOKEN
  end

  def send(msg)
    #puts "server sent: #{msg}"
    send_data msg
  end

end

class EmServerExampleBadBuffer < EmServerExample

  def initialize(*args)
    super
    @buffer = DataBuffer.new
  end

  def receive_data(data)
    @buffer.append(data)
    while(command = @buffer.grab)
      process(command)
    end
  end

  def prepare(str)
    @buffer.prepare(str)
  end

end

class EmServerExampleBuffToken < EmServerExample

  def initialize(*args)
    super
    @recv_buffer = BufferedTokenizer.new(TOKEN)
  end

  def receive_data(data)
    @recv_buffer.extract(data).each do |m|
      process(m)
    end
  end

end

class EmServerExamplePastie < EmServerExample

  def initialize(*args)
    super
    @recv_buffer = BufferedTokenizerPastie.new(TOKEN)
  end

  def receive_data(data)
    @recv_buffer.extract(data).each do |m|
      process(m)
    end
  end

end

class DataBuffer
  FRONT_DELIMITER = "0x5b".hex.chr # '['
  BACK_DELIMITER = "0x5d".hex.chr #']'[0].to_s(16).hex.chr
  DELIMITER = "|#{FRONT_DELIMITER}#{FRONT_DELIMITER}#{FRONT_DELIMITER}GT_DELIM#{BACK_DELIMITER}#{BACK_DELIMITER}#{BACK_DELIMITER}#{BACK_DELIMITER}|"
  DELIM_ESCAPE = /#{Regexp.escape(DELIMITER)}/
    DELIM_ESCAPE_END = /#{Regexp.escape(DELIMITER)}\Z/

    def initialize
      @unprocessed = ""
      @commands = []
    end

    def grab
      new_messages = @unprocessed.split(DELIM_ESCAPE)
      while new_messages.length > 1
        @commands << new_messages.shift
      end
      msg_length = new_messages.length
      if msg_length > 0
        if msg_length == 1 && (@unprocessed=~DELIM_ESCAPE_END)
          @commands.push(new_messages.shift)
          @unprocessed = ""
        else
          #put the rest of the last statement back into the buffer
          while(cut=@unprocessed.index(DELIM_ESCAPE))
            @unprocessed = (@unprocessed[cut..@unprocessed.length]).sub(DELIMITER,"")
          end
        end
      end
      if @commands.length > 0
        return @commands.shift
      else
        return nil
      end
    end

    def prepare(str)
      str.to_s+DELIMITER
    end

    def append(data)
      @unprocessed = @unprocessed + data
    end

  end

  class EmSendFileTest < Test::Unit::TestCase

    def test_placeholder
      assert true
    end

    def start_server(server_type)
      server_pid = fork {
        EventMachine::run do
          EventMachine::start_server SERVER_IP, SERVER_PORT, server_type
          puts "Server now accepting requests..."
        end
      }
      server_pid
    end

    def start_client(client_type)
      client_pid = fork {
        EventMachine::run { client_type.push_start() }
      }
      client_pid
    end

    def run_against_server_client(client_example, server_example)
      assert_nothing_raised do
        puts Benchmark.realtime {
          server_pid = start_server(server_example)
          #make sure server is up for client to connect to
          sleep(0.2)
          client_pid = start_client(client_example)
          sleep(0.2)

          Process.wait(client_pid)
          puts "client finished"

          #I don't know a clean way to end event machine take it down
          Process.kill('KILL',server_pid)
          Process.waitall
        }
        puts "##############################################################"
      end
    end

    def test_em_send_files_with_em_buffered_tokenizer
      puts "send files test with em buffered tokenizer"
      client_example = EmClientExampleBuffToken
      server_example = EmServerExampleBuffToken
      run_against_server_client(client_example, server_example)
    end

    def test_em_stream_files_with_em_buffered_tokenizer
      puts "steam_file_data test with em buffered tokenizer"
      if COMPRESS == true
        puts "steam_file_data can't be used with on the fly compression"
      else
        client_example = EmClientExampleStreamBuffToken
        server_example = EmServerExampleBuffToken
        run_against_server_client(client_example, server_example)
      end
    end

    def test_em_send_files_with_bad_tokenizer
      puts "send files test with our bad bueffered tokenizer"
      client_example = EmClientExampleBadBuffer
      server_example = EmServerExampleBadBuffer
      run_against_server_client(client_example, server_example)
    end

    def test_em_send_files_with_pastie_tokenizer
      puts "send files test with the pastied tokenizer"
      client_example = EmClientExamplePastie
      server_example = EmServerExamplePastie
      run_against_server_client(client_example, server_example)
    end

  end

There have been a bunch of interesting tools released for Ruby lately. I decided to write about a few of my favorite Ruby tools and give some of the new tools a shot as well. Simply put, better tools can help you be a better developer. I am ignoring the entire topic of IDEs as tools, as I have written about Ruby IDEs before, and it is basically a religious war. If you use any Ruby tools I don’t mention be sure to let me know as I am always interested in trying something new out.

Lets get into it…

Roodi


dmayer$ sudo gem install roodi
dmayer$ roodi lib/client/syncer.rb
lib/client/syncer.rb:136 - Block cyclomatic complexity is 5. It should be 4 or less.
lib/client/syncer.rb:61 - Method name "excluded" has a cyclomatic complexity is 10. It should be 8 or less.
lib/client/syncer.rb:101 - Method name "should_be_excluded?" has a cyclomatic complexity is 9. It should be 8 or less.
lib/client/syncer.rb:132 - Method name "find_changed_files" has a cyclomatic complexity is 10. It should be 8 or less.
lib/client/syncer.rb:68 - Rescue block should not be empty.
lib/client/syncer.rb:61 - Method name "excluded" has 25 lines. It should have 20 or less.
lib/client/syncer.rb:132 - Method name "find_changed_files" has 27 lines. It should have 20 or less.
Found 7 errors.


After Refactoring:

~/projects/gridtest/trunk dmayer$ roodi lib/client/syncer.rb
lib/client/syncer.rb:148 - Block cyclomatic complexity is 5. It should be 4 or less.
lib/client/syncer.rb:82 - Rescue block should not be empty.
Found 2 errors.


I did have one problem with Roodi - the errors about rescue blocks just seemed to be incorrect. For code like the little example below it kept throwing the error even though I obviously am doing some work in the rescue code.

Roodi output: lib/client/syncer.rb:68 - Rescue block should not be empty.
begin
  socket = TCPSocket.new(server_ip,server_port)
  socket.close
  return true
rescue Errno::ECONNREFUSED
  return false
end

Dust

Flog

Running Flog against any subset of a project is easy, here I am running it against our client libraries
find ./lib/client/ -name \*.rb | xargs flog -n -m > flog.log

Here some example Flog output when run against our client code.

Total score = 1364.52395469781

Client#send_tests: (64.3)
    14.3: assignment
    13.9: puts
    10.7: branch
    10.5: send
     4.7: send_quit
     3.4: message
     3.4: now
     2.0: create_queue_test_msg
     1.9: create_run_msg
     1.9: test_files
     1.8: dump
     1.7: each
     1.7: report_start
     1.7: length
     1.7: get_tests
     1.7: -
     1.7: open
     1.7: load_file
     1.6: empty?
     1.6: nil?
     1.6: use_cache
     1.6: exists?
ModClient#send_file: (32.0)
    12.4: branch
     5.4: +
     4.3: assignment
     3.9: send
     3.1: puts
     2.9: ==
     2.9: exists?
     2.9: directory?
     1.9: strftime
     1.8: to_s
     1.5: read
     1.5: create_file_msg
     1.4: info
Syncer#sync: (30.8)
    13.2: assignment
     8.6: branch
     3.6: inspect
     3.2: info
     3.0: puts
     2.8: +
     2.6: empty?
     1.7: map
     1.5: now
     1.5: length
     1.4: send_files
     1.3: max
     1.3: >
     1.3: find_changed_files
     1.3: write_sync_time
Syncer#find_changed_files: (26.2)
    15.6: assignment
     8.7: branch
     3.5: <<
     1.8: to_s
     1.7: get_relative_path
     1.7: >
     1.7: mtime
     1.6: exists?
     1.6: ==
     1.5: prune
     1.4: should_be_excluded?
     1.3: get_removed_files
     1.3: find
... and so on ...

Saikuro

Saikuro Screenshot

Heckle

#install heckle
dmayer$ sudo gem install heckle

#example of the infinite loop Error Heckle run
heckle Syncer should_be_excluded? --tests test/unit/client/syncer_test.rb -v
Setting timeout at 5 seconds.
Initial tests pass. Let's rumble.

**********************************************************************
***  Syncer#should_be_excluded? loaded with 13 possible mutations
**********************************************************************
...
2 mutations remaining...
Replacing Syncer#should_be_excluded? with:

2 mutations remaining...
Replacing Syncer#should_be_excluded? with:
... loops forever ...

#Heckle run against our Client class and the process method
dmayer$ heckle Client process --tests test/unit/client/client_test.rb
Initial tests pass. Let's rumble.

**********************************************************************
***  Client#process loaded with 9 possible mutations
**********************************************************************

9 mutations remaining...
8 mutations remaining...
7 mutations remaining...
6 mutations remaining...
5 mutations remaining...
4 mutations remaining...
3 mutations remaining...
2 mutations remaining...
1 mutations remaining...

The following mutations didn't cause test failures:

--- original
+++ mutation

 def process(command)

   case command
   when @buffer.Ready then
     process_ready
-  when @buffer.SetID then
+  when nil then
     process_set_id(command)
   when @buffer.InitProject then
     process_init_project
   when @buffer.Result then
     process_result(command)
   when @buffer.Goodbye then
     kill_event_loop
   when @buffer.Done then
     process_done
   when @buffer.Error then
     process_error(command)
   else
     @log.error("client ignoring invalid command #{command}") if @log
   end
 end

--- original
+++ mutation
 def process(command)
   case command
   when @buffer.Ready then
     process_ready
   when @buffer.SetID then
     process_set_id(command)
   when @buffer.InitProject then
     process_init_project
   when @buffer.Result then
     process_result(command)
   when @buffer.Goodbye then
     kill_event_loop
   when @buffer.Done then
     process_done
   when @buffer.Error then
     process_error(command)
   else
-    @log.error("client ignoring invalid command #{command}") if @log
+    nil if @log
   end
 end

Heckle Results:

Passed    :   0
Failed    :   1
Thick Skin:   0

Improve the tests and try again.

#Tests added / changed to improve Heckle results
def test_process_process_loop__random_result
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    client.log.expects(:error).with("client ignoring invalid command this is random")
    client.process("this is random")
  end

  def test_process_process_loop__set_id
    Client.any_instance.expects(:start_tls).returns(true)
    client = Client.new({})
    client.stubs(:send_data)
    client.log = stub_everything
    cmd = DataBuffer.new.create_set_ids_msg("4")
    client.expects(:process_set_id).with(cmd)
    client.process(cmd)
  end
#A final Heckle run, showing successful results

dmayer$ heckle Client process --tests test/unit/client/client_test.rb

Initial tests pass. Let’s rumble.
**********************************************************************

***  Client#process loaded with 9 possible mutations

**********************************************************************
9 mutations remaining…

8 mutations remaining…

7 mutations remaining…

6 mutations remaining…

5 mutations remaining…

4 mutations remaining…

3 mutations remaining…

2 mutations remaining…

1 mutations remaining…

No mutants survived. Cool!
Heckle Results:
Passed    :   1

Failed    :   0

Thick Skin:   0
All heckling was thwarted! YAY!!!

rcov

ruby-prof

I have used ruby-prof a number of times to isolate the ways to speed up my code. I haven’t used it to identify why an entire Rails application is slow (there are better tools I discuss later for that), but if you have a small but highly important piece of code ruby-prof is often the best way to isolate the problem. I used ruby-prof to identified the two slowest lines of code of a spellchecker, which was rewritten to become twice as fast.

Most recently I used it to identify where the code was spending all of its time in a loop for a file syncer. It turns out that for thousands of files each time through the loop we were continually calling Pathname.new(path).relative_path_from(@dir_path) over and over. Putting a small cache around that call essentially eliminated all delays in our file synchronization. Below is a simple example of how a few lines of code can make all the difference in performance and how easily ruby-prof can help you isolate the problem areas and where to spend your time. I think seeing the code that ruby-prof helped isolate, and the changes made to the code might be useful if you are new to profiling and performance work.

changes in our spellchecker / recommender

#OLD Way
 alteration = []
    n.times {|i| LETTERS.each_byte {
        |l| alteration << word[0...i].strip+l.chr+word[i+1..-1].strip } }
 insertion = []
     (n+1).times {|i| LETTERS.each_byte {
        |l| insertion << word[0...i].strip+l.chr+word[i..-1].strip } }
 #NEW Way
    #pre-calculate the word breakups
    word_starts = []
    word_missing_ends = []
    word_ends = []
    (n+1).times do |i|
      word_starts << word[0...i]
      word_missing_ends << word[i+1..-1]
      word_ends << word[i..-1]
    end

 alteration = []
    n.times {|i|
      alteration = alteration.concat LETTERS.collect { |l|
        word_starts[i]+l+word_missing_ends[i] } }
 insertion = []
    (n+1).times {|i|
      insertion = insertion.concat LETTERS.collect { |l|

        word_starts[i]+l+word_ends[i] } }

Changes in our file syncer

#OLD
 path_name = Pathname.new(path).relative_path_from(@dir_path).to_s
 #NEW
 path_name = get_relative_path(path)

  def get_relative_path(path)
    return @path_cache[path] if @path_cache.member?(path)
    retval = Pathname.new(path).relative_path_from(@dir_path).to_s
    @path_cache[path] = retval
    return retval
  end

New Relic

New Relic screenshot

TuneUp

Update: FiveRuns is pretty amazing with their support. I haven’t got TuneUp fully working yet, but have made some progress. Some good things to know are that some plugins like safe_erb and output_compression can cause problems with TuneUp. They are aware of the issues, and actively looking into it.

Ruby Run

RubyRun screenshot

screenshot of a different RubyRun report

That is it, hope you learned about a new Ruby tool. So get to work, try a new tool, and get to know your code a little better than you did before.

While I was writing this article, people pointed out to me two more tools worth pointing out. I didn’t get a chance to try them out or review them, but thought I should point them out. Towlie, helps keep your code dry by finding redundant methods. and finally Source ANalysis (SAN), which is described as, “a Ruby gem for analyzing the contents of source code including comment to script ratios, todo items, declared functions, classes, and much more”.