Moving at Scale : Etsy

I know the etsy engineers like to graph, graph, graph, but I hadn't heard too much about the rest of their release cycle. They have taken specific steps to attempt to "Stay a Startup as we grow larger." Watch as they expound on the benefits of "one button deployment", empowering releases, branching-in-code, A-B testing and configurable features.

Warning, the video is a couple of hours long.

Watch live streaming video from etsy at livestream.com

Source

1. Make deployments easier
2. No-fault post-mortems
3. everyone works on head, no branches in source control, lots of if(0){ ... } blocks
4. configuration file that determines which features are live, to which percentage of the population.
5. public logging and metrics
6. ...
7. Change are quick, easy and encouraged.
8. maintains the "can do" "default is yes" attitude of the early startup.
9. mean time to notice problems: <5 minutes, mean time to fix problems: 4.5 minutes.
10. ...
11. profit

They have released two projects as openSource, available on github:

logster

https://github.com/etsy/logster

Forked from ganglia-logtailer, they removed the daemon mode to simplify the code.

Logster is a utility for reading log files and generating metrics in Graphite or Ganglia. It is ideal for visualizing trends of events that are occurring in your application/system/error logs. For example, you might use logster to graph the number of occurrences of HTTP response code that appears in your web server logs.

statsd

Github: https://github.com/etsy/statsd

Blogpost: measure anything measure everything/

A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to graphite.

They make heavy use of Graphite for displaying graphs. http://graphite.wikidot.com/