The Devver Blog

Devver started with a good idea (web-based tools for Ruby
hackers) and a working prototype, which couldn't scale. We started working on making Devver scalable and decided to go with EC2, from Amazon. Unfortunately we quickly learned that a library, Rinda, which we built our messaging system on, couldn't connect between multiple EC2 instances.

No worries, we thought, Amazon has its own queuing system, which can be used for messaging, SQS. We thought since Devver was on Amazon's EC2, it maked sense to switch to Amazon's Queues. We first refactored our code and isolated our old messaging away from the rest of the system. Then we we switched over to SQS which we noticed was slower. Since, Devver takes developer tools into the cloud, we have to run our tools against various projects to verify correct functionality and performance. Running Devver against on our test and toy projects, SQS was slower, but it was working out OK.

After building in a stats collection system allowing us to start charting our progress against real metrics, we started running larger projects on Devver. We quickly learned that Devver suddenly couldn't handle or deal with any real projects. Using Devver's metrics to guide us, we isolated the problem down to messaging.

We found out that Devver was spending approximately 60 seconds messaging our distributed machines, for every 6 seconds of work they would do. The round trip of messaging through Devver made the entire project useless. Knowing this wasn't a problem in our Rinda based prototype, we decided Devver just needed a better messaging system. Since we had just refactored the code, the messaging system part was very isolated from the rest of the system and it was easy to drop in a new solution.

While Amazon SQS is robust and reliable, fast it is not (.4 seconds put/take a message remotely, .04 within AWS (on EC2)). We did some research and found many different messaging systems for Ruby, and held a Ruby messaging shootout. This post isn't about what system we eventually went with, because I am sure many of the options would have been fine. I mostly want to share a story of using metrics to pinpoint the worst offender, and isolating parts of your application, so that in a single afternoon you can replace a highly critical part of a project. Changing out one component that is heavily relied on , especially if there is a flaw or a bottleneck, can be the difference between success and failure.

A quick example of the performance changes we saw from swapping out our messaging:

Devver processing Mocha with SQS, 63.8-80.8 seconds
Devver processing Mocha with new messaging, 3.9 seconds

That is it for this post, but look forward to some stats and our detailed review of Ruby messaging systems in the future.

Maybe it's just because quite a few of my plugins (notably Firebug) aren't supported (and therefore deactivated), but browsing my usual sites with FF3 is noticably faster. Download it now.

The other day I was talking to my brother about possible plans for Devver's future. I'd clearly been coding too much that day, so the conversation went something like this:

Me: "Of course, [plan A] and [plan B] aren't ... uh, you know ... one doesn't stop the other ... I mean, they aren't ... uh ... you know?"
Him: "You mean they aren't mutually exclusive?
Me: "Yeah, that's it ... all I could think was 'not (A xor B)'"
Him: "..."

The really sad thing is that while I was blanking on the phrase "not mutually exclusive", I really could see the Ruby code (!(A ^ B)) in my head. Sigh.

Andrew Hyde recently interviewed Dan and I for the TechStars Community site. You can listen to the podcast here (and see a picture of Dan and I staring dramatically at a piece of paper).

This is the final day of my TechStars Days series of posts.

Yahoo Day

Greg Cohn (Yahoo Open Strategy)

Greg talked a little about Yahoo's current focus, Making Yahoo relevant to how users manage their digital life. Yahoo is working on becoming dominant in the advertising market by leveraging its 500 million plus users. Lastly it is focusing on opening Yahoo up to allow others to build on and with Yahoo. Greg spent most of his time talking about Yahoo's Open Strategy, which is one framework for developing applications with Yahoo. The open strategy will include single user accounts, profiles, user records, and authentication APIs. This would all be supported by a single standard for consuming web services and supported by code from YDN (Yahoo Developer Network). We got a peek at the Yahoo Social API which is built on top of Open Social, supporting all of the Open Social standard, but also additional APIs. It was interesting to see how Yahoo really wants to transform its image to show that it is a social network and has always had the social abilities, it just never presented them.

Eric Marcoullier (MyBlogLog)

I saw Eric speak at TechStars for a Day, and it was funny to hear his story a second time. Eric just showed up and winged it, he has no prepared talk, slides, or real agenda. He goes off on tangents to talk about whatever he feels like discussing. He likes to apologize for his lack of style and ramblings, but it always turns out to make for one of the most interesting and blunt talks. I will just list out some of the key points from the discussion the TS teams had with him. MyBlogLog was a part of Yahoo Day because Yahoo eventually acquired the company.

• MyBlogLog did one thing originally, track outgoing links, but did it well
• Keep a narrow focus, the first version only had 3 different pages for the app
• Engage the community. Eric emailed the top 150 (or so) bloggers. Sending each blogger a personal email about what MyBlogLog did and why they should care (He received a 10-15% response rate)
• MyBlogLog being stats had no viral component, it needed something users could see, this idea slowly led to the widget.
• 6-7% would pay for a pro version, which was the same except it gave real time stats
• Don't always add more features sometimes find the market that wants the features you already have.
• Launch quickly and get feedback, show people something even if it is half baked and rough
• To engage mentors, try to be responsive to any of their first even small comments, because if you react to them and show them changes they are more likely to discuss and become involved with your idea
• After selling a first company successfully, you have a platform / reputation to help build another company

Wrap-Up

I would like to thank everyone that came and talked with us at TechStars, trust me, it was a great learning experience. I am sure I will be reading through my notes time to time in hopes of keeping myself on track. I know my attempt at writing about the experience doesn’t really do it justice, but I figured I should try to share some of the experience gained.

This is part three of my TechStars Days series of posts.

Microsoft Day

Dave Drach (Emerging Business Team)

Dave is heavily involved in the Boulder / Denver Startup scene, working with Lijit, NewsGator, Search-To-Phone, Filtrbox and others. Dave basically told us MS loves developers and does everything they can to nurture developers working on their platform. I didn't know that from the MS startup zone, you can basically get anything from MS free to start your company. Free Hosting (2 machines for 3 years), 5 free MSDN licenses, free developer support, and consideration for their Startup Accelerator Program. Dave ran us through many of the various offerings where MS could help you out if you are developing on or for their stack.

Don Dodge (Emerging Business Team)

Don has an amazing history, perhaps most famous as the VP of Napster. This position led to this message to our teams, paraphrased, "You can and have the power to change the world." He is right that the simple music sharing app changed the digital landscape forever, and other startups will lead the way with other revolutions. Don told us that MS liked to acquire young startups when innovation is happening. He would rather get 50 small innovative companies than a billion-and-a-half dollar YouTube like deal. He did admit to us that most of what MS acquires is on the MS stack. Since we are working in the Ruby world, I guess that means we aren't a target (maybe IronRuby? Interested?). Don is really plugged into the whole start up community and is very vocal about it on his blog, which I highly recommend checking out.

Anand Iyer "AI" (Microsoft Evangelist)

AI shared various demos and code with us, showing off some of the newest stuff in the MS stack. He spent a good amount of time showing off Silverlight, which can do some cool things. Silverlight is particularly interesting because we saw offerings all in related spaces from MS, Google, and Yahoo. Every company wants to be the plug-in that lets you develop richer applications in the browser. I think anyone that has seen some of the deep zoom / Photosyth stuff has to admit it is pretty impressive. We also got to see some demos of IronPython which is cool because I am interested in having powerful dynamic languages (IronRuby) play nicely in a VM.

Greg Reinacker (NewsGator)

Greg had a small consulting company and had an idea for a RSS reader that didn't suck. He thought it would be great to be able to read his feeds in Outlook. He mocked up some screenshots of reading feeds in Outlook and got 50 comments, "please build this." So he built it and sold 25 copies the day he released it. This project started outgrowing his consulting company, which he shut that down and raised some money to grow NewsGator. One of the problems promoting NewsGator was no one really knew what RSS feeds were at the time, so NewsGator had to become a really big advocate for RSS adoption, and explain RSS as part of their pitch. This is something we think about with Devver, because we feel like we will have to partially push agile programming and better programming practices to promote our tools. The stories shared by all the rising and finally acquired companies helps TS early teams try to really evaluate if we are focusing on the right things. NewsGator started as a single founder company and eventually was acquired by Microsoft.

This is part two of my TechStars Days series of posts.

Amazon Day

Mike Culver (Amazon AWS)

Mike did a great job presenting all of the various AWS offerings. The TechStars teams asked a ton of questions. It seemed after the presentation at least 4 of the teams were thinking about using some part of AWS. We found this whole session interesting as we had just made the decision to build Devver on top of some of the AWS offerings. Devver is currently working with EC2 and SQS, but we are strongly considering Amazon's SimpleDB as well. Most of the topics our group discussed about AWS are publicly available. It was still worth hearing a talk and seeing demos which explore the possibilities as it helped sell us on the AWS solution.

Josh Hug (Shelfari)

Amazon Day continued with Shelfari, which Amazon ended up investing in. Josh gave a candid talk about the growth of his startup. Josh had a different beginning than the average startup. He wanted to start a company, but didn't know what he should work on. He started looking at industry trends he could take advantage of, which eventually led him to Shelfari. He shared many tips about the early stages of a startup. Since my notes were just a list of bullet points I will share a few of the best.

• Focus on getting first version done
• Early focus in a startup has the largest impact
• Use A/B testing to really see what your users prefer
• Outsource a payroll service
• Do your own bookkeeping and have it checked once a month
• Actively engage in your own PR
• Keep initial costs really low
• A final great quote, "The grass is always greener on the other side of the technology fence."

The first couple weeks, of TechStars has been flying by in a blur. The first weeks were filled with many meetings, introductions, contacts, and informal group meet-ups. I am pretty sure during the first two weeks each team gave their elevator pitch 45+ times, giving us all chances to refine our pitches. During these first weeks besides the regular meetings, there were some full meeting days, which I call "TechStars Days".

The thought behind our initial meeting-packed schedule is that TS front loads many of the longer meetings at the beginning of the program. This allows us to try to absorb as much of the advice as possible before we get moving. If we came in the program and immediately started going as fast as possible we might find that we were running the wrong direction. Devver has slightly shifted the way we think of our business, and I think we are better off for it. At the heart of these full days there were four large, themed days: Google Day, Amazon Day, Microsoft Day, and Yahoo Day. We just finished the last of our so-called themed days, and I have far too many thoughts and notes to even try to turn them all into blog posts. So I will just try to cover some key points and notes from my experience.

Over the next four days I will post all of my TechStars Days thoughts, starting with Google Day below.

Google Day

Kevin Marks (OpenSocial)

Kevin talked about the “Social Cloud”, as he put it. The idea is that we don't have to build new social networks and have our social graph data trapped inside any particular website (i.e. Facebook). We can use the existing web infrastructure of linking to establish our social circles. Using XFN (which apparently Kevin prefers) or FOAF you can link to websites and express your relationship to those sites. I could for example link from my blog to my girlfriend's blog and use XFN to express that she is my girlfriend. As this metadata grows on the web, Google does what it does best, and crawls all this information and allows people to search and query against the data via an API (Google's Social Graph API). Now the next time you sign up for a new service like Twitter, instead of finding all your friends again, it can query my social graph and recommend contacts that I have already expressed on the web. This is also far safer and better than many services asking for my Gmail password to find my contacts. The open social graph is a very cool and big idea, but it will really require a large adoption of XFN by many and preferably highly social sites.

Kevin also talked about Google providing a solution to the Gmail password problem. Google has a Contacts API which will get all of your contacts information from Gmail without you having to give a third party system their password. The contact API initially is entirely separate from the social API, but I am sure you can see the likely overlap.

Lastly Kevin got into Open Social, the large social initiative Google has been pushing to allow developers to write a single social app that should play nicely in any Open Social container. So a site like LinkedIn would run an Open Social container and apps that were developed for Open Social could run on this container and access my LinkedIn social graph so the app could use those relationships. The same social app could run on other Open Social containers as well like MySpace, iGoogle, or BeBo. Open Social sounds really promising to me if I was working in the social or viral space.

Kevin had interesting thoughts on viral growth and organic growth, the second of which he believes is the correct way to try to spread through a network. He also discussed limitations that social growth can run into, for example language or cultural barriers. He didn't get into this topic as deep as he could have, but recommended that if we were interested, we should really read Danah Boyd, who has been working in the social space for a long time.

Dion Almaer (App Engine, Google Gears)

Google Gears isn't just providing offline experiences, it is improving the speed and performance of websites. It also offers a unique way to offload work from the server to the client side. An interesting example of this is MySpace providing message searching to users via Google Gears. Utilizing SQLite DB, local caching, and local thread pools allows developers to perform many operations client side better and simpler than server side. Google Gears also allows full icon links, growl for the web, multiple file upload without flash, and a variety of other cool things.

Dion also talked briefly about Google's location API which sounded pretty cool, being able to identify the users location based on Wi-Fi hotspots, cell towers, GPS, and a variety of other sources. Besides giving a location it will give accuracy ratings. I don't know how much of this is live now, but it could be very useful if you are working in the location-based services.

Dion of course talked about Google App Engine last, which I really enjoyed hearing about, but pretty much everything we heard is publicly available. It seems to have some really nice benefits over Amazon like being free for up to 5 million page views, having a stronger DB offering, and seemingly better support. It also has some huge limitations which seemed to make most people, myself included, feel it isn't ready for many real applications. If you want to build a sweet Python web app or Django-powered site, get on App Engine, but if you need any back end control App Engine just isn't the way to go yet. It was great to hear this as we are currently very interested in cloud computing. Dion wasn’t allowed to discuss upcoming features or the widely discussed rumors of other languages and MapReduce support, admitting only that some big things are in the works.

Rick Klau, Dick Costolo (FeedBurner)

As part of Google Day, we got to hear from a company that was acquired by Google. My cofounder did a great post about the talk, FeedBurner get all the feeds, so I will leave that to him.

On Tuesday, we had Mike Culver from Amazon give a great talk about Amazon's web services. We had pretty much decided on using AWS for Devver, but Mike's talk convinced us even more.

So for the past few days we've been porting our code to work with EC2 . We've also been building scripts that will configure the machine instances on boot.

When you're using EC2, you can store custom machine images, but you really don't really want to be saving a new image for every tweak you make to the machine. It's a lot more flexible to just save a basic image and give the instance a script to run when it boots, like this:

ec2-run-instances [some-image-name] -f config.rb

In this example, I'm passing a Ruby script, but you can pass any type of file at all.

I found a couple of great tutorials, but unfortunately they didn't do quite what I wanted, so I hacked up my own solution. It's just a few lines in the /etc/rc.local file (which runs after the machine boots) that grabs the config.rb file and runs it.

Unfortunately, for some reason, my Ruby script was not getting run. After a painfully long debugging session, I figured out that the environment had not been configured for RubyGems when my script was run, so my 'require' statements were breaking the code. The key lesson is that no matter what language you're using, /etc/rc.local is running in a different environment than a logged-in user, so environment variables like PATH (and in this case, RUBY_OPT) will not be set up the same way.

Anyway, here's the code I added to my /etc/rc.local file:

# AMI configuration code, based on code from
#http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1085&categoryID=100

# grab the file from Amazon's web service. Yes, this IP address is always the same
/usr/bin/wget http://169.254.169.254/latest/user-data -O /tmp/payload.rb

# if wget error code is 0, there was no error
if [ $? -eq 0 ]; then
  if [ -e /tmp/payload.rb ]; then
    # DON'T forget the -rubygems option! You need it if you call 'require' in your config code
    /usr/bin/ruby -rubygems /tmp/payload.rb
  else
    echo "rc.local : No payload.rb script to run"
  fi
else
  echo "rc.local : error retrieving user data"
fi

I hope that helps anyone out there starting to play with EC2...

Last night, Dick Costolo and Rick Klau from Feedburner stopped by the TechStars office to give a fascinating (and often hilarious) talk about their experiences -starting with just four guys with a tiny market and no revenue through getting acquired by Google.

There was a lot of great advice in the talk, but I want to touch on two points that really got me thinking.

First, Dick and Rick mentioned that from the beginning, Feedburner understood what Feedburner was about: they were going to provide value by sitting between publishers and their audience. Furthermore, they were going to interact with publishers and be virtually invisible to the audience.

The interesting result of this is that they turned down several lucrative revenue streams because they didn't fit with their vision. For instance, some companies offered them money to install Feedburner on corporate intranets. Although this would have been a source of revenue at a time when they didn't have any, they had the confidence to stick with their vision and say "no, that's not what we do."

Secondly, they had a core goal/metric they executed on: "get all the feeds". Notice that this isn't the same as their business model, slogan, or investor pitch. Rather, it was used internally as a simple goal that got everyone working towards the same thing. Dick was apparently famous for saying stuff like (and I'm paraphrasing here), "Do you have all the feeds? No? OK, then get back out there and get more feeds". Jim Collins goes into this more (he calls it "The Hedgehog Concept") in Good to Great.

These points got me thinking: what is our vision for Devver? And what is our our core goal/metric? I'll save the answers for another post after I've had some time to really think about it.