The Devver Blog

I'm happy to announce that Devver is one of the sponsors of BizConf. The event is shaping up to be a great one, so I'm very excited about attending.

The presenter lineup looks great. Here are just a few of the talks that I'm looking forward to attending:

Corey Haines - Why Agile Will Probably Fail You

Jon "Lark" Larkowski - Getting Things Done

Ian McFarland - Rails Economics and the ARC Model

Check out the full list of speakers.

If you're interested in attending, the bad news is that early bird pricing is over. The good news is that if you use the code DEVVER when registering, you'll get $1000 off your ticket price.

If you do attend BizConf, please come find me to talk about getting started with Devver!

We are very happy to be a sponsor of LSRC. I am especially excited because that means I get to attend the event. I am looking forward to getting a chance to meet another Ruby community as I have never been to Austin Texas, and it seems like there are a lot of exciting things going on with the Ruby community. Find me and come by to talk about Ruby, testing, or Devver. Devver is also currently hiring, so if you are attending the conference and interested in highly distributed Ruby systems, definitely come talk to us. It is great to get to participate in events like these and spend time with the amazing Ruby community which is so supportive of new ideas, good code and testing, and startups.

Check out some of the great things that will be going on at Lone Star Ruby Conf this year.

I am particularly excited about:

• Mike Subelsky: Ruby for Startups: Battle Scars and Lessons Learned
• Larry Diehl: Dataflow: Declarative concurrency in Ruby
• Ian Warshak: Rails in the Cloud
• Jeremy Hinegardner: Playing nice with others. -- Tools for mixed language environments.
• Evan Light: TDD: More than just "testing"
• Jake Scruggs: What's the Right Level of Testing?
• Corey Donohoe, atmos: think simple
• Pradeep Elankumaran: Fast and Scalable Front/Back-end Services using Ruby and XMPP
• Danny Blitz: Herding Tigers - Software and the Art of War

Looking forward to meeting everyone in Austin, shoot me an email at dan@devver.net or message me on twitter @danmayer so we can meet up at the conference in person.

Have you ever started a long operation and walked away from the computer, and come back half an hour later only to find that the process is hung up waiting for some user input? It's a sub-optimal user experience, and in many cases it can be avoided by having the program choose a default if the user doesn't respond within a certain amount of time. One example of this UI technique in the wild is powering off your computer - most modern operating systems will pop up a dialogue to confirm or cancel the shutdown, with a countdown until the shutdown proceeds automatically.

This article is about how to achieve the same effect in command-line programs using Ruby.

Let's start with the end result. We want to be able to call our method like this:

puts ask_with_countdown_to_default("Do you like pie?", 30.0, false)

We pass in a question, a (possibly fractional) number of seconds to wait, and a default value. The method should prompt the user with the given question and a visual countdown. If the user types 'y' or 'n', it should immediately return true or false, respectively. Otherwise when the countdown expires it should return the default value.

Here's a high-level implementation:

def ask_with_countdown_to_default(question, seconds, default)
  with_unbuffered_input($stdin) do
    countdown_from(seconds) do |seconds_left|
      write_then_erase_prompt(question, seconds_left) do
        wait_for_input($stdin, seconds_left % 1) do
          case char = $stdin.getc
          when ?y, ?Y then return true
          when ?n, ?N then return false
          else                  # NOOP
          end
        end
      end
    end
  end
  return default
ensure
  $stdout.puts
end                             # ask_with_countdown_to_default

Let's take it step-by-step.

By default, *NIX terminals operate in "canonical mode", where they buffer a line of input internally and don't send it until the user hits RETURN. This is so that the user can do simple edits like backspacing and retyping a typo. This behavior is undesirable for our purposes, however, since we want the prompt to respond as soon as the user types a key. So we need to temporarily alter the terminal configuration.

  with_unbuffered_input($stdin) do

We use the POSIX Termios library, via the ruby-termios gem, to accomplish this feat.

def with_unbuffered_input(input = $stdin)
  old_attributes = Termios.tcgetattr(input)
  new_attributes = old_attributes.dup
  new_attributes.lflag &= ~Termios::ECHO
  new_attributes.lflag &= ~Termios::ICANON
  Termios::tcsetattr(input, Termios::TCSANOW, new_attributes)

  yield
ensure
  Termios::tcsetattr(input, Termios::TCSANOW, old_attributes)
end                             # with_unbuffered_input

POSIX Termios defines a set of library calls for interacting with terminals. In our case, we want to disable some of the terminal's "local" features - functionality the terminal handles internally before sending input on to the controlling program.

We start by getting a snapshot of the terminal's current configuration. Then we make a copy for our new configuration. We are interested in two flags: "ECHO" and "ICANON". The first, ECHO, controls whether the terminal displays characters that the user has types. The second controls canonical mode, which we explained above. After turning both flags off, we set the new configuration and yield. After the block is finished, or if an exception is raised, we ensure that the original terminal configuration is reinstated.

Now we need to arrange for a countdown timer.

    countdown_from(seconds) do |seconds_left|

Here's the implementation:

def countdown_from(seconds_left)
  start_time   = Time.now
  end_time     = start_time + seconds_left
  begin
    yield(seconds_left)
    seconds_left = end_time - Time.now
  end while seconds_left > 0.0
end                             # countdown_from

First we calculate the wallclock time at which we should stop waiting. Then we begin looping, yielding the number of seconds left, and then when the block returns recalculating the number. We keep this up until the time has expired.

Next up is writing, and re-writing, the prompt.

      write_then_erase_prompt(question, seconds_left) do

This method is implemented as follows:

def write_then_erase_prompt(question, seconds_left)
  prompt_format = "#{question} (y/n) (%2d)"
  prompt = prompt_format % seconds_left.to_i
  prompt_length = prompt.length
  $stdout.write(prompt)
  $stdout.flush

  yield

  $stdout.write("\b" * prompt_length)
  $stdout.flush
end                             # write_then_erase_prompt

We format and print a prompt, flushing the output to insure that it is displayed immediately. The prompt includes a count of the number of seconds remaining until the query times out. In order to make it a nice visually consistent length, we use a fixed-width field for the countdown ("%2d"). Note that we don't use puts to print the prompt - we don't want it to advance to the next line, because we want to be able to dynamically rewrite the prompt as the countdown proceeds.

After we are done yielding to the block, we erase the prompt in preparation for the next cycle. In order to erase it we create and output string of backspaces ("\b") the same length as the prompt.

Now we need a way to wait until the user types something, while still periodically updating the prompt.

        wait_for_input($stdin, seconds_left % 1) do

We pass wait_for_input an input stream and a (potentially fractional) number of seconds to wait. In this case we only want to wait until the next second-long "tick" so that we can update the countdown. So we pass in the remainder of dividing seconds_left by 1. E.g. if seconds_left was 5.3, we would set a timeout of 0.3 seconds. After 3/10 of a second of waiting for input, the wait would time out, the prompt would be erased and rewritten to show 4 seconds remaining, and then we'd start waiting for input again.

Here's the implementation of wait_for_input:

def wait_for_input(input, timeout)
  # Wait until input is available
  if select([input], [], [], timeout)
    yield
  end
end                             # wait_for_input

We're using Kernel#select to do the waiting. The parameters to #select are a set of arrays - one each for input, output, and errors. We only care about input, so we pass the input stream in the first array and leave the others blank. We also pass how long to wait until timing out.

If new input is detected, select returns an array of arrays, corresponding to the three arrays we passed in. If it times out while waiting, it returns nil. We use the return value to determine whether to execute the given block or note. If there is input waiting we yield to the block; otherwise we just return.

While it takes some getting used to, handling IO timeouts with select is safer and more reliable than using the Timeout module. And it's less messy than rescuing Timeout::Error every time a read times out.

Finally, we need to read and interpret the character the user types, if any.

          case char = $stdin.getc
          when ?y, ?Y then return true
          when ?n, ?N then return false
          else                  # NOOP
          end

If the user types 'y' or 'n' (or uppercase versions of the same), we return true or false, respectively. Otherwise, we simply ignore any characters the user types. Typing characters other than 'y' or 'n' will cause the loop to be restarted.

Note the use of character literals like ?y to compare against the integer character code returned by IO#getc. We could alternately use Integer#chr to convert the character codes into single-character strings, if we wanted.

Wrapping up, we make sure to return the default value should the timeout expire without any user input; and we output a newline to move the cursor past our prompt.

  return default

And there you have it; a yes/no prompt with a timeout and a visual countdown. Static text doesn't really capture the effect, so rather than include sample output I'll just suggest that you try the code out for yourself (sorry, Windows users, it's *NIX-only).

Full source for this article at: http://gist.github.com/148765

In the previous article we looked at some basic methods for starting subprocesses in Ruby. One thing all those methods had in common was that they didn't permit a lot of communication between parent process and child. In this article we'll examine a few built-in Ruby methods which give us the ability to have a two-way conversation with our subprocesses.

The complete source code for this article can be found at http://gist.github.com/146199.

Method #4: Opening a pipe

As you know, the Kernel#open method allows you to open files for reading and writing (and, with addition of the open-uri library, HTTP sockets as well). What you may not know is that Kernel.open can also open processes as if they were files.

  puts "4a. Kernel#open with |"
  cmd = %Q<|#{RUBY} -r#{THIS_FILE} -e 'hello("open(|)", true)'>
  open(cmd, 'w+') do |subprocess|
    subprocess.write("hello from parent")
    subprocess.close_write
    subprocess.read.split("\n").each do |l|
      puts "[parent] output: #{l}"
    end
    puts
  end
  puts "---"

By passing a pipe ("|") as the first character in the command, we signal to open that we want to start a process, not open a file. For a command, we're starting another Ruby process and calling our trusty hello method (see the first article or the source code for this article for the definition of the hello method RUBY and THIS_FILE constants).

open yields an IO object which enables us to communicate with the subprocess. Anything written to the object is piped to the process' STDIN, and the anything the process writes to its STDOUT can be read back as if reading from a file. In the example above we write a line to the child, read some text back from the child, and then end the block.

Note the call to close_write on line 5. This call is important. Because the OS buffers input and output, it is possible to write to a subprocess, attempt to read back, and wait forever because the data is still sitting in the buffer. In addition, filter-style programs typically wait until they see an EOF on their STDIN to exit. By calling close_write, we cause the buffer to be flushed and an EOF to be sent. Once the subprocess exits, its output buffer wil be flushed and any read calls on the parent side will return.

Also note that we pass "w+" as the file open mode. Just as with files, by default the IO object will be opened in read-only mode. If we want to both write to and read from it, we need to specify an appropriate mode.

Here's the output of the above code:

4a. Kernel#open with |
[child] Hello, standard error
[parent] output: [child] Hello from open(|)
[parent] output: [child] Standard input contains: "hello from parent"

---

Another way to open a command as an IO object is to call IO.popen:

  puts "4b. IO.popen"
  cmd = %Q<#{RUBY} -r#{THIS_FILE} -e 'hello("popen", true)'>
  IO.popen(cmd, 'w+') do |subprocess|
    subprocess.write("hello from parent")
    subprocess.close_write
    subprocess.read.split("\n").each do |l|
      puts "[parent] output: #{l}"
    end
    puts
  end
  puts "---"

This behaves exactly the same as the Kernel#open version. Which way you choose to use is a matter of preference. The IO.popen version arguably makes it a little more obvious what is going on.

Method #5: Forking to a pipe

This is a variation on the previous technique. If Kernel#open is passed a pipe followed by a dash ("|-") as its first argument, it starts a forked subprocess. This is like the previous example except that instead of executing a command, it forks the running Ruby process into two processes.

  puts "5a. Kernel#open with |-"
  open("|-", "w+") do |subprocess|
    if subprocess.nil?             # child
      hello("open(|-)", true)
      exit
    else                        # parent
      subprocess.write("hello from parent")
      subprocess.close_write
      subprocess.read.split("\n").each do |l|
        puts "[parent] output: #{l}"
      end
      puts
    end
  end
  puts "---"

Both processes then execute the given block. In the child process, the argument yielded to the block will be nil. In the parent, the block argument will be an IO object. As before, the IO object is tied to the forked process' standard input and standard output streams.

Here's the output:

5a. Kernel#open with |-
[child] Hello, standard error
[parent] output: [child] Hello from open(|-)
[parent] output: [child] Standard input contains: "hello from parent"

---

Once again, there is an IO.popen version which does the same thing:

  puts "5b. IO.popen with -"
  IO.popen("-", "w+") do |subprocess|
    if subprocess.nil?             # child
      hello("popen(-)", true)
      exit
    else                        # parent
      subprocess.write("hello from parent")
      subprocess.close_write
      subprocess.read.split("\n").each do |l|
        puts "[parent] output: #{l}"
      end
      puts
    end
  end
  puts "---"

Applications and Caveats

The techniques we've looked at in this article are best suited for "filter" style subprocesses, where we want to feed some input to a process and then use the output it produces. Because of the potential for deadlocks mentioned earlier, they are less suitable for running highly interactive subprocesses which require multiple reads and responses.

open/popen also do not give us access to the subprocess' standard error (STDERR) stream. Any output error generated by the subprocesses will print the same place that the parent process' STDERR does.

In the upcoming parts of the series we'll look at some libraries which overcome both of these limitations.

Conclusion

In this article we've explored two (or four, depending on how you count it) built-in ways of starting a subprocess and communicating with it as if it were a file. In part 3 we'll move away from built-ins and on to the facilities provided in Ruby's Standard Library for starting and controlling subprocesses.

In order to show how easy it is to configure Devver for a project, we've made a short screencast to walk you through the steps. We've used DataMapper as an example application. As you can see, it only takes a few minutes to set up Devver and then the specs complete in a fraction of the time. In fact, the whole process - setup and Devver run - takes less time than running 'rake spec'.

In order to see the commands clearly, you'll want to enter full-screen mode. Or, if you prefer, you can download the high-quality version.

Our friend Andrew Hyde recently helped us film a short video of us announcing some of the recent developments here at Devver. Check it out!

Devver Announces Funding from Andrew on Vimeo.

We are working hard to quickly expand our compatibility on Ruby projects. With that goal driving us, we are happy to announce support for Postgres and SQLite databases. With the addition of these database options, along with our existing support for MySQL, Devver now supports all of the most popular databases commonly used with Ruby. These three databases are the default databases tested against ActiveRecord and we expect will cover the majority of the Ruby community.

To begin working with Postgres or SQLite on Devver all you need to do is have a database.yml with the test environment set to the adapter of your choice. If we don't support your favorite database, you can still request a beta invite and let us know which database you want us to support. If we just added support for your database, perhaps we can speed up your project on Devver, so request a beta invite.