The Devver Blog

Last week, we had Avdi, the newest addition to our team, join us in Boulder, CO. It was great to get some face-to-face time, since Avdi will primarily be working from his home in Pennsylvania while Dan and I continue to work in Boulder.

We are excited about the benefits of having a distributed team, but we're also aware that there are a number of challenges. As a result, one of the things we worked on last week was figuring out the tools and practices we'll be using to work effectively from across the country. Luckily, both Avdi and Dan have experience working remotely which we can draw upon.

We evaluated a number of options, but settled on the following tools and practices.

Practices

• Daily Standup. Every day at the same time, we all get on video chat. We cover what we did yesterday, what we're working on today, and whether or not we're blocked on anything. The goal is to keep this meeting at 15 min or less.
• Minimize interruptions. Whenever we need to communicate with each other, we try to do so on the channel that is the least disruptive (and disrupts the fewest team members). Of course, sometimes we need to be disruptive if an issue is pressing, if someone is blocked, or if we need to have high-bandwidth communication (information, especially cues like body language, don't come across very effectively on channels like email)
• Keep it simple. We want to use the smallest number of tools and channels that will allow us to work effectively.

Channels and Tools

Less disruptive More disruptive	Channel	Tool	Properties
	Passive Updates	Present.ly	Asynchronous Not required reading
	Email	Any email client (in practice, Gmail)	Asynchronous Required reading (usually) Sometimes time-sensitive, sometimes not
	IM	Skype	Semi-synchronous (but usually synchronous) Usually time-sensitive
	Voice/video chat	Skype	Synchronous High bandwidth* (especially video chat) Best for meetings

* By "high bandwidth", I don't mean that the tool itself requires a lot of TCP/IP traffic (although this is true, it doesn't really matter). What I mean is that we can communicate a lot of information between team members in a short amount of time.

Other Tools

• Lighthouse for issue tracking
• GitHub for source control and our project wiki
• RealVNC for screen sharing (essential for remote pair programming)

This is our first attempt at finding a good set of tools and practices for remote collaboration. As time goes on, we'll undoubtedly iterate and improve upon these.

For another perspective (with a slightly different set of tools), here is a presentation from 2008 about virtual teams.

What tools and practices have worked (and which have not worked) for your team?

Dan and I are very excited to welcome Avdi Grimm to our small team here at Devver. Avdi has over ten years of professional experience working at Raytheon and MDLogix and has created and contributed to a number of open-source projects. He shares our vision of bringing developer tools to the cloud, our goal of using testing to consistently deliver high-quality software, and our commitment to openness between our company, our users, and the community at large. Welcome, Avdi!

We're very happy to announce that we've had a great developer accept our offer to join our team. We'll have more details on our newest team member soon.

This was our first attempt at hiring and we certainly learned a lot. While our process has improved since we started, it's still a work in progress.

The first lesson we learned is that hiring can take a long, long time - in fact, much longer than we expected. I looked back through my emails and we officially started looking for someone more than four months ago. One thing we have heard time and time again was the importance of hiring only the best people - and, just as importantly, people that fit well within your company culture.

As a result, we were extremely picky when it came to candidates. As the process goes on and on, it's easy to get frustrated and want to lower your bar. Avoid this temptation! If at all possible, go the other route: raise your compensation, improve your pitch, and raise your profile so you can attract candidates that meet or exceed your bar.

When we started, we severely underestimated how long it would take and we overestimated how many applications we'd get. Because of these bad assumptions, we made our first mistake - we chose to slowly "roll out" the announcement that we were hiring. First, we just told friends and mentors. A week or so later, we tweeted and put up a blog post. Later on, we posted on some news sites and newsgroups. And after that, we finally created a job posting at Startuply. Waiting between each step was a waste of time - we should have just broadcast as loudly as we could the first day.

We ended up getting a good number of applications, the vast majority of which were from great programmers. We assumed that going through them would waste a bunch of time. We were wrong. In reality, it was really easy to turn down many applications just from their resume (usually the applicant was good, but didn't have the skill set we were looking for). Phone calls were a bit more time-consuming, but were never a waste of time. We always enjoyed talking to candidates, always found a way to improve our process, and usually learned something new about the Ruby ecosystem.

One thing that we improved on was learning to say "no" quickly. Even with the manageable number of candidates we talked to, it was important say "no" as soon as we discovered something that wouldn't fit. Early on, we were a bit hesitant to do so and ended up continuing our process for too long. That wasn't fair to either ourselves or the candidates. It became easier to shut things down quickly once we had interviewed a few people and started to gain confidence that new applications would show up, even if the queue was currently empty.

We also learned that it's important to have a fairly short process. Of course, the biggest priority is to have a process that lets you learn enough about a candidate to be confident in your decision. That said, as you improve, you'll find that you can shorten your process while maintaining your confidence. Shortening the process is better for you (less time spent) and better for your candidates.

The initial version of our process went something like this:

1. Check resume, blog, Github account
2. Introductory phone call to learn about candidate and convince them Devver is cool
3. Second phone call to ask some high-level technical questions
4. Ask for and call references
5. Remotely pair on a project for a few hours with the candidate
6. Fly candidate out to Boulder to meet and discuss technical and business issues

The final version was a bit more compressed:

1. Check resume, blog, Github account
2. Phone call to learn about them, introduce ourselves, and cover high-level technical questions
3. Ask for and call references
4. (In parallel with #3) Ask candidate to write small Ruby application and review their code
5. Fly candidate out to do some pair programming and discuss technical and business issues

All in all, it was a good (if sometimes painful) learning experience. We deeply appreciate the time and effort spent by every single applicant (and their patience with us as we learned by trial and error).

If you've got your own lessons you've learned while hiring, please let us know. We know we've still got a lot to learn...

Today we released the beta version of Devver.net to our initial set of users. This release is a major milestone: while our alpha release showed that the concept could work, it didn't have the security or scalability for real-world use. This beta is a solid foundation for us to build on in the months ahead.

We're happy to have this release out, but it was certainly a difficult experience - we spent over two months on this release, which is more than a month longer than we expected. We've learned a lot from this - most importantly, that it's very difficult to predict the time to complete large releases. We felt it was necessary to make the big changes now, but going forward we'll be moving to much shorter release cycles (every 1-2 weeks).

In the coming months, we'll be looking for people to try out our beta. If you have a slow Ruby Test::Unit test suite that you'd like to speed up, contact us and we'll put you on our list.

At Devver.net, we send out weekly email updates to an awesome set of mentors. We do this for a number of reasons. First and foremost, we get valuable feedback and advice from our mentors on a variety of issues. But it's also an easy and effective way to keep us on track and even maximize our chances of success. As Paul Graham says in How Not To Die (he was talking directly to YC teams, but you'll get the idea):

"For us the main indication of impending doom is when we don't hear from you. When we haven't heard from, or about, a startup for a couple months, that's a bad sign.

...

Maybe if you can arrange that we keep hearing from you, you won't die.

That may not be so naive as it sounds. ... [The] mere constraint of staying in regular contact with us will push you to make things happen, because otherwise you'll be embarrassed to tell us that you haven't done anything new since the last time we talked."

Foodzie started emailing their mentors early in the summer. We actually borrowed (stole) their email format and best practices.

One thing we've tried to not do is send out a completely generic email to all our mentors. Depending on the content and the interaction we've had with a specific mentor, we'll adjust his email accordingly. We begin each email with their name and send it directly to them (in other words, we don't put a huge list of addresses in the To, CC, or BCC fields). We do this because we can tailor it and it helps elicit individual responses from each mentor (it's easier to ignore a question if it's sent to a group).

But, of course, sometimes the emails to a few mentors can be identical. In this case, my not-so-well-kept secret is that I just use a simple Ruby script to send out a duplicate email that appears to be hand-crafted (or at least copied and pasted).

I've been told that Outlook can perform this functionality easily, but I don't know of any way to do this within Gmail. If there is, let me know so I can feel a little silly (in any case, the Ruby code was fun to write).

To run this code, you'll need to install the highline gem. You'll also need to add your Gmail account, recipients, subject message, etc. Finally, you'll want to put your message inside a separate file within project directory. That way, you can easily modify, spellcheck, and format to your heart's content before sending.

You can get the entire gmailr source code (all two files!) at Github. Please use this script for good, not evil - no one likes a spammer. Enjoy!

We are looking for a top-notch Ruby hacker who wants to help us build the next generation of developer tools.

At Devver.net, you'll be solving new and interesting problems on a daily basis. You'll help us improve our architecture and build new features.

If you have a passion for Ruby, developer tools, and highly distributed systems, please read more about the job and contact us at jobs@devver.net

Although I'm not actually the CTO of Devver, I had the pleasure of attending the Boulder CTO lunch this past Monday since Dan was out of town.

This week, the group had Todd Vernon from Lijit come lead the discussion. Although Todd is currently CEO of Lijit, he was CTO at his former company, Raindance.

The group that was assembled was small but awesome - I had the opportunity to learn not only from Todd, but also from the CTOs of a few of last years TechStars companies.

The discussion touched on a ton of topics, but two (related) themes that were heavily discussed were the role of the CTO and how a company grows from a technology perspective. I've organized my notes below. Keep in mind that these are the collected thoughts from a number of different participants and I may not have captured their ideas with 100% accuracy.

The Role of a CTO

What is the difference between a CTO and a VP of Engineering?

CTO is about leadership for technical issues, interfacing with the business side, guiding the product, get people excited about product from a technical point of view.

VPoE is almost one step above Chief Architect, more on a management side, getting product delivered.

1st time CTOs need to figure out their exact role. It's a very amorphous role, depending on the company.

CTO needs to be able to tell the whole business story, understand the good parts and bad.

Early on, CTO should insert themselves into the sales process as much as possible (especially right after you hire the sales person). You need to be able to hear what customers say they want, so you can translate that into what they really need.

The Technology Onion

There is a technology onion - make sure the core of the onion is owned by company (the outer layers, not as important). The CTO needs to figure out the relationship between the technologies and the company's partnerships. The closer a technology is to the core of the onion, the more important it is to own it and to make sure it scales.

For instance, if you're depending on Google for search, you're powerless to change features if things don't work as your customers want/expect. If search is core to your business (near the core of the onion), consider building it internally. It's the CTOs role to make that case, because business people will never understand the need to spend money to get "the same thing."

Having to re-architect a core component of a company can really hurt growth. Assume you're going to be successful, so plan for that.

Along the same lines, one concern about using EC2 is that you get tied to the platform and your business is dependent on an outside force you can't control. Hosting on EC2 can be quite different than hosting your own boxes.

Acceptable Failures

What is acceptable downtime? It depends on when - between midnight to 1-2 AM, it might be OK to be down for a few minutes. CTOs need to determine what acceptable downtime is and tell that the to the rest of management and have people agree.

CTOs need to make decisions (for instance, what is the acceptable down time, acceptable data loss, or acceptable time for page load) and then tell the entire organization. That way, when something bad happens, you can explain that everyone agreed on the specific numbers. It's unlikely your business will need to be (or can be) 100% perfect on all metrics, but people need to understand what the goal is and why it's realistic.

Growing/Scaling/Monitoring

If at least one person is using your service, you should have two web servers. It gives you ton of flexibility. Having two boxes forces you to work out most of the issues early (it's a lot different getting to 2 boxes than 3 or 4). It's not about load, it's about reliability.

No matter how useful you are, if you are not reliable, someone will blog, "it's cool, but it doesn't work reliably."

Downtime spreads very fast across Twitter. Consider tweeting about upcoming service interruptions ahead of time so customers are aware.

After more than 15 people, you need a dedicated operations person. Get some basic monitoring services early - after a server is under load, it's really hard to diagnose. Try to detect stuff early, it's easier to debug.

With startups, generally the problem tends to be slow requests rather than complete service downtime. Make sure your monitoring service will alert you with slow requests.

Get app specific stuff - a warning like "High CPU load," is harder to understand (it might be a problem, or maybe the machine is just handling a lot of requests successfully), but "Page X takes 80 sec to load" is more obvious.

As you grow, try to measure more and more. Things often degrade slowly, and one day you just notice its too slow and it's hard to go back and find the root of the problem.

Make two lists: a) the most catastrophic things that could happen and b) the most likely things that could happen. Where those lists overlap, you need to fix something. But there will be some risks that you decide are reasonable risks for the business (revisit these risks regularly as things change).

Regarding backup - always make sure you try to restore some data (before you really need it). You need to make sure it works and make sure its fast enough.

You should always be able to describe at a high level how the service will scale infinitely (it doesn't have to be technically perfect, but it has to be believable). When someone wants to purchase, that'll be a huge help - the business guy on the other side of the table will want to buy, but the technical guy doesn't want to buy (he wants to build it in-house).

---

I hope those notes make some sense and give you a good feel for the discussion we had. I'm looking forward to attending more of these lunches (I hope they'll continue to let a few CEOs sneak in...)

At Devver we have a lot of jobs to do quickly, so we distribute our work out to a group of EC2 workers. We have tried and used a number of queuing solutions with Ruby, but in the end beanstalkd seemed to be the best solution for us at the time.

I have only seen a few posts about the basics of using beanstalkd with Ruby. I decided to make two posts evolving a simple Ruby beanstalkd example into a more complicated example. This way people new to beanstalkd could see how easy it can be to get up and running with distributed processing using Ruby and beanstalkd. Then people that are doing more advanced work with beanstalkd could see some examples of how we are working with it here at Devver. It would also be great for more experienced beanstalkd warriors to share their thoughts as there aren't many examples out in the wild. The lack of examples makes it harder to learn and difficult to decide what the best practices are when working with beanstalkd queues.

I have also shared two scripts we have found useful while working with beanstalkd. beanstalk_monitor.rb, which lets you see all the queue statistics about current usage, or to monitor the information of a single queue you are interested in. Finally, beanstalk_killer.rb, which is useful if you want to work on how your code will react to beanstalkd getting backed up or stalling (in beanstalkd speak, "Putting on the brakes"). It was a little harder to pull everything out and make a simple example from our code than I thought, and obviously the example is a bit useless. It should still give a solid example of how to do the basics of distributing jobs with beanstalkd.

For those new to beanstalk, there are a few things you will need to know like how to get a queue object, how to put objects on the queue, how to take objects off the queue, and how to control which queue you are working with. For a higher level overview or more detailed information, I recommend checking out the beanstalkd FAQ. The full example code is below, but first taking a look at the basic snippets might help.

#to work with beanstalk you need to get a client connection
queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
#by default you will be working on the 'default' tube or queue
#if we wanted to work on a different queue we could change tubes, like so
queue.watch('test_queue')
queue.use('test_queue')
queue.ignore('default')
#to put a simple string on a queue
queue.put('hello queue world')
#to receive a simple string
job = queue.reserve
puts job.body #prints 'hello queue world'
#if you don't delete the job when you're done, the queue assumes there is an error
#and the job will show back up on the queue again
job.delete

How to run this example (on OS X, with macports installed)

> sudo port install beanstalkd
> sudo gem install beanstalk-client
> beanstalkd
> ruby beanstalk_tester.rb

Download: beanstalk_tester.rb

require 'beanstalk-client.rb'

DEFAULT_PORT = 11300
SERVER_IP = '127.0.0.1'
#beanstalk will order the queues based on priority, with the same priority
#it acts FIFO, in a later example we will use the priority
#(higher numbers are higher priority)
DEFAULT_PRIORITY = 65536
#TTR is time for the job to reappear on the queue.
#Assuming a worker died before completing work and never called job.delete
#the same job would return back on the queue (in seconds)
TTR = 3

class BeanBase

  #To work with multiple queues you must tell beanstalk which queues
  #you plan on writing to (use), and which queues you will reserve jobs from
  #(watch). In this case we also want to ignore the default queue
  def get_queue(queue_name)
    queue = Beanstalk::Pool.new(["#{SERVER_IP}:#{DEFAULT_PORT}"])
    queue.watch(queue_name)
    queue.use(queue_name)
    queue.ignore('default')
    queue
  end

end

class BeanDistributor < BeanBase

  def initialize(amount)
    @messages = amount
  end

  def start_distributor
    #put all the work on the request queue
    bean_queue = get_queue('requests')
    @messages.times do |num|
      msg = BeanRequest.new(1,num)
      #Take our ruby object and convert it to yml and put it on the queue
      bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
    end

    puts "distributor now getting results"
    #get all the results from the results queue
    bean_queue = get_queue('results')
    @messages.times do |num|
      result = take_msg(bean_queue)
      puts "result: #{result}"
    end

  end

  #this will take a message off the queue, process it and return the result
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    count = msg.ybody.count
    msg.delete
    return count
  end

end

class BeanWorker < BeanBase

  def initialize(amount)
    @messages = amount
    @received_msgs = 0
  end

  def start_worker
    results = []
    #get and process all the requests, on the requests queue
    bean_queue = get_queue('requests')
    @messages.times do |num|
      result = take_msg(bean_queue)
      results << result
      @received_msgs += 1
    end

    #return all of the results, by placing them on the separate results queue
    bean_queue = get_queue('results')
    results.each do |result|
      msg = BeanResult.new(1,result)
      bean_queue.yput(msg,pri=DEFAULT_PRIORITY, delay=0, ttr=TTR)
    end

    #this is just to pass information out of the forked process
    #we return the number of messages we received as our exit status
    exit @received_msgs
  end

  #this will take a message off the queue, process it and return the result
  def take_msg(queue)
    msg = queue.reserve
    #by calling ybody we get the content of the message and convert it from yml
    count = msg.ybody.count
    result = count*count
    msg.delete
    return result
  end

end

############
# These are just simple message classes that we pass using beanstalks
# to yml and from yml functions.
############
class BeanRequest
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

class BeanResult
  attr_accessor :project_id, :count
  def initialize(project_id, count=0)
    @project_id = project_id
    @count = count
  end
end

#write X messages on the queue
numb = 10

recv_count = 0

# Most of the time you will have two entirely seperate classes
# but to make it easy to run this example we will just fork and start our server
# and client seperately. We will wait for them to complete and check
# if we received all the messages we expected.
puts "starting distributor"
server_pid = fork {
  BeanDistributor.new(numb).start_distributor
}

puts "starting client"
client_pid = fork {
  BeanWorker.new(numb).start_worker
}

Process.wait(client_pid)
recv_count = $?.exitstatus
puts "client finished received #{recv_count} msgs"
if(numb==recv_count)
  puts "received the expected number of messages"
else
  puts "error didn't receive the correct number of messages"
end

Process.wait(server_pid)

We're happy to announce that on Thursday we officially launched with our first alpha customer!

Dan and I were both pretty nervous, but we also knew that this was an important step forward for Devver. I won't say it went off without a hitch (we ran into a few bugs in the morning, which was pretty stressful). I don't think we've ever watched log files with such interest and anticipation.

Thankfully, our alpha customers were awesome - patient, forgiving, and funny all at once. We got tons of great feedback from them in just in the first few hours alone. And, by the afternoon, Devver was up and running on their code base - in fact, we brought a test suite that was taking around forty minutes to complete on one machine down to two minutes on Devver.

All in all, it was a successful launch. It was a huge thrill to see a customer using Devver. I'll leave you with a few real IM quotes from our customers:

"hurray! tests are zipping along."

"I'm actually really excited to work with this.  I think it will really improve the process around here."

And my personal favorite: "dots a'flowin' "

Behold my favorite picture of this awesome TechStars summer

Considering you can clearly see the Devver logo on Dan's shirt, I think this needs to be in all our future marketing material. Nothing says "developers love Devver" like a developer doing a backflip in excitement. Awesome stuff.