Using Amazon EC2 like a cheap, confusing VPS

Let's say you wanted a VPS-like server somewhere that runs all the time, gives you root access, and lets you install whatever you want. Let's say you also wanted to attach tons of space, a CDN, a database server, and anything else. Well you'd want to use some of the Amazon AWS products but you'd probably not want to have your VPS service hosted somewhere else, so you can centralize.

So you'd think, EC2 lets you run servers, and EBS lets you use them like they have regular hard drives attached, let's just buy a yearlong EC2 reservation and get a fat server for an amortized $75/month! It's a great plan, until you start to try it and realize everything in AWS is designed in little tiny pieces, not systems.

So then you try something like RightScale, or Judo, or Scalr, and you think, I don't need this autoscaling clouding magic scripty crap, I just want a server! Is that so hard? Well no, but it's $250/month from Slicehost, and you don't want to pay that.

Let's do something different. Let's get as close to a VPS as we can in EC2 using the simplest tools possible - just the console and the ec2 API tools.

First, some concepts. AWS has its own language that is not very familiar. I strongly recommend taking an hour or two and reading the EC2 User Guide so you can get a handle on what's going on here. I tried to avoid it but really it's best to just read it straight through. This isn't a quickie project.

The core units we'll be focusing on are EBS volumes and EBS-backed AMIs. These are going to form the core of your server's identity.

EBS volumes can start as either a Snapshot, which is basically a tape backup of a drive at a point in time, or as an empty drive. Once created, they can then exist either as a detached ("available") volume, which is just like an unplugged hard drive sitting on the table, or an attached ("in-use") volume, which is like a drive plugged into a server. (Note that the server they are attached to doesn't have to be running.) You can leave EBS volumes sitting around detached as long as you like, and take snapshots of them whenever.

EBS-backed AMIs are awesome, because we can take any Snapshot and make it the root device of an AMI. Then whenever you start the AMI, it would be like like taking that Snapshot (remember, tape backup), buying a server, copying the tape data onto the server's main hard drive, and starting it up. You still have the tape, and the server now has the tape data (as copied to its hard drive, a new EBS volume) as its starting point.

The thing about AMIs is they don't really exist as servers, they exist as the idea of a server. Sort of a specification, not a saved state. They have a particular set of disks (or snapshots) to attach, a kernel to run, and an architecture, but that's about it. In most cases they're designed to destroy all the data they create during their lifetime, because AWS likes you to run things on-demand, not forever. Well in a VPS you want forever, so we're going to do some tricks to get there.

What we're going to create in AWS is an AMI that, when you create an instance from it, creates and boots from an EBS volume containing the data in a Snapshot. When the instance is stopped (shut down) or terminated (deleted), the EBS volume it made will sit there detached in your list.

This is important because while you could use an EBS-backed instance as a VPS simply by never terminating it once it's running, if for some reason it was terminated, you'd lose all the data on it unless you took a snapshot, but even then you'd be restoring from your last snapshot, not the moment the instance stopped. This is why termination protection exists, but a checkbox on a webpage is not enough to protect production data.

Let's do some work. Look for a the AMI you want in the AMI list. At the time of writing ami-3202f25b (Ubuntu 10.04 20110201.1) is a good one. Launch it, and go to Instances. When it's running, right-click it and select Create Image (EBS AMI). You'll see an AMI go to Pending in your list of AMIs Owned By Me, and a Snapshot go Pending as well. Get some paper and write down the Snapshot ID, then go back to AMIs and write down the Kernel ID.

Now drop into your terminal where we'll be using some of the ec2 tools (you installed them, right?). You can't do what we want from the web console, which is to set up a server that keeps all its disks around when you terminate it. Use ec2reg like so:

ec2reg -n 'Ubuntu 10.04 20110201.1 Base' -d 'Basic ubuntu server configuration' --root-device-name /dev/sda1 -b /dev/sda1=your-snap-id:8:false -a x86_64 --kernel your-kernel-id

What you just did was make an AMI (again, the idea of a server) that runs Ubuntu 10.04. What's special about it is that when you make instances out of it, those instances don't destroy the data they made over their lifetime. The false at the end of the -b argument is the trick.

So how many servers do you want? Just launch as many of those AMIs as you like and even if you terminate them, the stuff they do will still exist. You probably never actually want to terminate them, only stop them, but you would be safe even if you did. Now you can safely install software right on your instances without running launch scripts or pasting in a bunch of userdata.

Once you start running your instances, you'll still want to take snapshots of their volumes every so often. As long as you don't terminate your instances, only stop them, they'll work just like servers with hard drives. If you do terminate one, you'll need to do a little runaround. You'll have a detached volume that was that instance's root device. You can't boot a server from a detached EBS volume, only a snapshot. So take a snapshot of the detached root device, then run ec2reg again with that snapshot as the snap-id. You now have a new 'rescue' AMI that really only represents that one particular instance, so when you launch a replacement instance from it, you should be right back where you left off. You can keep that AMI around if you want, the important thing is the snapshot. You can always create another AMI based on a standing snapshot that has the data you want, just make sure you pick the right kernel and mountpoints. Documentation helps here.

So this is great for software on a server's root drive, but let's say you want to run a data storage engine on your VPS-like EC2 setup. You wouldn't be getting much of the benefits out of EC2 by having all that data stored right on the boot drive, far better to use a separate EBS volume for your data, or even a couple of them RAIDed together. This allows you to take data snapshots separately from system snapshots.

There's 2 ways you could do this. First, we'll design a setup where all the instances you want will have a data volume mounted when they are launched.

We'll need to make a different AMI to represent this, because you can't edit the launch block device configuration of an AMI once you've made it. We'll use ec2reg again:

ec2reg -n 'Ubuntu server with 10GB at sdf1' -d 'Ubuntu 10.04 with new 10GB volume at /dev/sdf1' --root-device-name /dev/sda1 -b /dev/sda1=your-snap-id:8:false -b /dev/sdf1=:10:false -a x86_64 --kernel your-kernel-id

This will create a new 10GB EBS volume at /dev/sdf1 when this AMI is launched. It won't actually be mounted in the OS, or even formatted, because the root device snapshot we're working with has no idea that it exists. If you're launching these for the first time this should be fine, because you can format it, declare it in fstab, etc. and your changes will be safe due to the persistent root device. If you terminate an instance and have to reattach its snapshot to a new AMI later, you'll need to snapshot both the root and data volumes and include them in the rescue AMI:

ec2reg -n 'Redis slave server rescue AMI' -d 'Rescuing the redis slave from snapshots' --root-device-name /dev/sda1 -b /dev/sda1=instance-snap-id:8:false -b /dev/sdf1=instance-data-snap-id:10:false -a x86_64 --kernel your-kernel-id

This will make an AMI that will let you relaunch that instance with both its root device and its data intact.

The other way to do this is to simply create some EBS volumes and attach them to the instances you made from the first AMI. They won't be destroyed if you terminate the instances because they were attached after the instances launched. Just don't forget to reattach them if you have to rescue the instances, or include them in the rescue AMI as shown above.

Taking into account the costs of data transfer, snapshot and EBS storage, and instance type runtime costs, this may or may not actually be cheaper than a standard VPS. You get a lot more headroom though, and easy integration with other AWS products.

Rails routing gotcha: Don't name a route not_found

When moving our app to Bundler, I got the following error:

(__DELEGATE__):2:in `not_found': wrong number of arguments (2 for 0)
(ArgumentError)
from (__DELEGATE__):2:in `send'
from (__DELEGATE__):2:in `not_found'
from /app_dir/vendor/rails/activesupport/lib/active_support/
option_merger.rb:20:in `__send__'
from /app_dir/vendor/rails/activesupport/lib/active_support/
option_merger.rb:20:in `method_missing'
from /app_dir/config/routes/plaza_routes.rb:5
from /app_dir/vendor/rails/activesupport/lib/active_support/core_ext/
object/misc.rb:78:in `with_options'
from /app_dir/vendor/rails/actionpack/lib/action_controller/routing/
route_set.rb:51:in `namespace'

The secret was a route that looked like this:

global.not_found '/not_found', :controller => 'home', :action =>
'not_found'

The solution? Rename the route to something else. No idea why this wasn't exposed before, but at least there's a solution!

Cisco VPN vs. Parallels 4.0

If you're using Cisco VPNClient on OS X 10.5, and you have Parallels 4.0 installed, you may be treated to the following error when attempting to start a VPN connection over ppp - a typical use case for an on-call developer using a 3G modem like the USBConnect Mercury:

305 10:28:16.787 05/22/2008 Sev=Warning/2 CVPND/0x83400011
Error -28 sending packet.

...

Output size mismatch. Actual: 0, Expected: 237. (DRVIFACE:1319)

The fix for this is to uninstall Parallels and buy VMWare fusion instead because Parallels is slow and it sucks. If there's a workaround I don't have the patience to find it.

More on practice

In a great post by Jeff Atwood.

Ada Byron was Homeschooled

Continuing this discussion from Geeknews and greatjustice.

The post at geeknews doesn't make much of a statement except in the title, but since this is a generally inflammatory topic, they've earned a pile of readership from that alone. And to prove I'm not the guy with the degree trumpeting the wonders of elitism, I went to a second-place state college. Even that was a better education than I could've managed on my own, but then it doesn't take MIT to do that.

I've been a vocal supporter of bootstrapping your way into other careers for a couple years now, but recently have changed my mind. I think the question of degree vs. hard knocks goes deeper than the "value of an education" or "real-world experience". Those are great things, but what will really turn a journeyman to a master is deliberate practice (PDF).

According to Ericsson, et. al., to really be deliberate practice, not just any task will do:

The most cited condition [for optimal improvement] concerns the subjects' motivation to attend to the task and exert effort to improve their performance. In addition, the design of the task should take into account the preexisting knowledge of the learners so that the task can be correctly understood after a brief period of instruction. The subjects should receive immediate informative feedback and knowledge of results of their performance. The subjects should repeatedly perform the same or similar tasks.

This probably sounds obvious, but take a step back. What they're asserting is that learning is doing. It is a task. You can sit on your ass reading CS articles on Wikipedia for 4 years and never learn how to write a program. You could even read through MIT's "Structure and Interpretation of Computer Programs" course material for free and not learn a damn thing about Scheme.

Whether you're learning from Wikipedia, a professor, a colleague, or a book, you're not going to start learning till you care about the subject and start stretching. A great programmer is one that is not afraid to do just that, and at the end of the day it doesn't matter whether he went to college or not. He will be great because he wanted to be and was not afraid to do what it takes to get there: intentional, deliberate, ongoing practice.

This practice of learning has been around much longer than universities. The system of apprenticeship survived for thousands of years. Any good tutor can create the above conditions for their student, but at the end of the day it is the student that must put the work in, not just the reading. Or the student can put that work in even without a tutor, though it will take much more effort.

So when hiring, look for the curious, unafraid, diligent one. Don't let a degree fool you into thinking that, just because they had the opportunity to learn, they did anything but read.

Further reading: A Reg Braithwaite post in a similar vein.

On Elegance

Lately I've been working on a new media serving backend for Treemo. We've outgrown the features on our existing setup, and since the vast majority of our hits are content downloads, it's time to optimize those.

One thing that we've discovered needs changing is the URL structure for our content. Most large content-focused sites like Flickr, Fotolog, or Imageshack use an unguessable portion of the URL to enforce visibility restrictions. This allows easy connection to a CDN, as you can simply pass out those URLs to a hosting service secure in the knowledge that they will only be presented to the appropriate people. Livejournal is an exception, they do authorization and byteserving on their balancers. We copied this setup initially, but as is the case with many Livejournal tools, we found we had to really go whole hog copying the rest of their setup (in this case, perlbal) to make it useful. So we're changing our approach.

Obviously changing your URL scheme for embedded content is a rather major migration, and not to be done lightly. Among the many issues that have arisen during this process is what the URLs should look like. I spent a lot of time evaluating the Flickr URL scheme, as they're the biggest and we were looking to model their architecture instead of Livejournal's this time. One thing I came to appreciate about their scheme is the elegance. You can check it out here:

http://www.flickr.com/services/api/misc.urls.html

The longer I worked on various combinations of schemes that would give us the structure we wanted, the more I realized that their scheme included everything you needed and nothing you didn't, and did so in under 80 characters. Making something that nice or nicer became a goal.

I got to the point of evaluating how to alter our object model to accommodate the new information that the filesystem would need to back up the new scheme, and darn if it wasn't hard. Adding the data I wanted was going to be expensive. There were easier schemes I could use involving long hashes or other things that would be just as secure, just as functional. But they'd be huge. And something in me didn't like that, and a few halfhearted dust-ups with my officemates found me unable to articulate why.

Part of it felt like well, Flickr has already proved this scheme works. They've learned lessons about it that we don't haven't, so let's not reinvent the wheel here. But that doesn't get me very far. Other, easier schemes would work, and who cares what the URLs look like?

And I couldn't escape the fact that I do, and I had to leave it at that.

But the more I thought about it, stronger I felt that not only should a new scheme be functional, but it should be nice. It should bear the mark of a system made with care for people, not tools. I think the ability to make this small part of the system exceed requirements in an elegant and surprising way became important, because if all I do is make tools that work, I get bored. Anyone can write code that does what the customer wants. The magic happens when we make code the customer doesn't realize he wants. You might not realize that nice URLs make the system more humane to use, but when they're obnoxious, it's noticed. "Huh," your users will say, "that's funny-looking." Not a thought that should be going through your users' minds, even for the half-second their subconscious will think about it.

I think we as developers have a lot of responsibility to think like a user even - maybe even especially - when it makes our jobs harder. Thinking for them helps us develop products that exceed their expectations, which allows teams to gel, which is the only way great products are made.

PHP in a nutshell

# is_writable — Tells whether the filename is writable
# is_writeable — Alias of is_writable