peat dot org

Amazon will soon be releasing their SimpleDB service under a limited beta program.

I’m very excited about this. Persistent, high performance databases are a big missing piece in Amazon’s cloud computing initiative — EC2 doesn’t offer storage that persists across reboots, and S3 isn’t structured to provide the IO required by a database.

Conceptually, SimpleDB is very compelling. It’s designed for real time querying, has no hard limits on storage, and is metered based on storage and CPU time. It looks a lot like Amazon’s Dynamo technology … and it wouldn’t surprise me if they released the Dynamo paper to gauge interest in exposing such a service.

But, there are three big caveats.

It’s not SQL. This isn’t actually as big a deal as it seems, but I know there are going to be a lot of people who are bent out of shape on this one. Why isn’t it a big deal? Because …

It’s not relational. SimpleDB provides a big flat table, with arbitrary attributes per row. So, queries are all about filtering through data, and while they can have very complex rules, it doesn’t behave like the “normal” relational databases we’re accustomed to using.

Updates are “eventually consistent.” This means that if you immediately query for data you just pushed into SimpleDB, it may not show up. You have a guarantee that it will show up within a few seconds, but not immediately. Amazon calls this “eventual consistency.”

It may be a little scary for some folks who are most comfortable with the traditional model of building apps around a single relational database. On the other hand, it appears to be a great system for people who have built big websites, and are already comfortable dealing with lazy synchronization and custom data sources.

I’m looking forward to playing with it!

(Tip o’ the hat to @grigs)

Update: Here’s a great post that goes into a little more detail about the give and take of SimpleDB. Fun fact: it’s written in Erlang.

Update:  More stats.  Looks like their opening offering lets you create up to 100 “domains” containing up to 10 GB of data each.   That’s a good start.

Here’s an interesting excerpt from Werner Vogels’ Dynamo paper about some of the guts behind Amazon’s e-commerce platform:

“For example a page request to one of the e-commerce sites typically requires the rendering engine to construct its response by sending requests to over 150 services. These services often have multiple dependencies, which frequently are other services, and as such it is not uncommon for the call graph of an application to have more than one level.”

There’s no doubt Amazon uses extensive caching to keep performance up, but 150+ service calls to render a page is remarkable, regardless of how you cut it. Even more impressive is how all of these services are built around the assumption that something, somewhere is failing: disks are crashing, networks are flapping, and processes are dying.

Check out the paper for more details.

Update:  It looks like Ars Technica got interested and put together a little write up on Dynamo.

I had lunch yesterday with some of the fine folks at JanRain, and one of our discussions was about Amazon’s S3 … can a business actually save money, using it for file storage and distribution? It turns out there’s a few pretty good cases for it, the most impressive being SmugMug saving half a million bucks vs. their DIY approach.

But things are changing in June. Amazon unveiled a new pricing model for S3, which is a little more complex than the previous $0.15 per gigabyte stored per month, with $0.20 per gigabyte transferred (simple, ‘eh?).

The storage cost is the same, but transfers have been lowered and put into a tiered structure, and there’s an additional charge for each request:

• $0.10 per GB - all data uploaded
• $0.18 per GB - first 10 TB / month data downloaded
• $0.16 per GB - next 40 TB / month data downloaded
• $0.13 per GB - data downloaded / month over 50 TB
• $0.01 per 1,000 PUT or LIST requests
• $0.01 per 10,000 GET and all other requests

I’m not a big fan of complexity, but Amazon seems to think it’ll save most of us some money:

“If this new pricing had been applied to customers’ March 2007 usage, 75% of customers would have seen their bill decrease, while an additional 11% would have seen an increase of less than 10%. Only 14% of customers would have experienced an increase of greater than 10%.“

Fair enough, I suppose.

More Firefox goodies!  A short while ago I noted the S3Fox extension for wrangling files on Amazon’s S3 file storage service.  Today I found a great extension for managing EC2 machines — the aptly named EC2 UI.  It’s about as simple as it gets.  Very handy!

I’m a huge fan of my little white MacBook.  I think it’s just about the best deal on the planet for a personal computer, laptop or otherwise.  The only thing that isn’t so hot is the screen size.  Great for use as a laptop, not so great for sitting at a desk all day.

So I went out looking for a cheap monitor that wasn’t totally hideous and rendered text nicely, and I found this:  a Hanns G 19″ widescreen.  No, I’ve never heard of them before … but it gets the job done nicely.  I bought it at a local shop, but Amazon has it for $180.  We now have a couple of ‘em, one at home on the Mac Mini, and one at work.  No troubles what so ever.

If you need a little extra screen space, and you’re not a stickler for color accuracy or incredibly high resolution, it’s a pretty sweet deal.

Dangit.  There aren’t any good tools out there for merging PDF documents.  I mean, there are tools, but they either crash randomly or produce HUGE documents.  Or cost money.  Lame.

Thank goodness for Automator.  I whipped out a three step workflow for selecting PDFs, merging them, and previewing the final result.  It seems to work pretty alright, although you gotta name your files alphabetically to set the order they’re combined.  Easy!

So, if you think it might be useful, you’re welcome to download PDF Combiner version 0.1.

On a slightly tangental note, I’ve hosted the file on Amazon’s S3 service.  I was looking for an easy way to upload files, and came across S3Fox … a handy and very useful extension to FireFox for doing just that.

Jesse Newland took Amazon EC2 and Capistrano out for a hot date, and ended up with Capazon — a brilliant set of tasks for managing EC2 instances.

Go forth and play with it, ye adventurous developers.

Good lord. Something happened in the flash memory industry over the last couple of months, and it kinda freaks me out (in a good way). Prices didn’t just halve on flash memory — they quartered. You can get 2GB for under $15, for a USB drive smaller than a stick of Trident gum.

That’s not rational pricing. What happened? Over enthusiasm on the part of flash memory manufacturers? If the trend continues, a gigabyte is going to be “throw away” memory by the end of the year, and 4GB (the size of a DVD) will be under ten bucks.

Neat.