Martin Kochanski’s web site / Software

 

(Originally published as a series of articles on the Cardbox blog)

S3 in Business: 1 - Introduction

Amazon’s S3 Simple Storage Service is a metered data storage service that works across the Internet.

Starting an Internet business

S3 means that you don’t have to spend time deciding how many terabytes of space you might need (wasting money if you buy too much and risking your existence if you buy too little). You don’t have to have high-security data centres with uninterruptible power supplies, with staff ready to rush in at the bleep of a pager if anything goes wrong. You don’t have to struggle with the dilemma of backup: back up too little and you risk massive data loss, back up too much and you risk one of your backups going missing, full of confidential information.

With S3, you just sign up and pay by the gigabyte. The more you use, the more you pay; the less you use, the less you pay. Amazon pay for a distributed network of data centres; Amazon pay for the staff; Amazon have software and procedures ensure that if one data centre is hit by a hurricane then another one will already have copies of all the data.

S3 seems the ideal choice for Internet startups that need data storage, and it’s already catalysing the invention of new Internet computing architectures. Whether companies will continue using it beyond the startup and adolescent phases remains to be seen, but even if S3 finds its place in the market as something you aim to grow out of eventually, it will have done a good job. It is far easier for a potential investor to assess an application with 10,000 users than it is to assess one that exists only on a whiteboard.

Individual computer users

S3 sounds perfect for backup and archiving. For every obsessive maker of daily backups, there are ten “wing and a prayer” people who know that their computer won’t go wrong today because it didn’t go wrong yesterday. And (as I fill yet another cupboard with my daily DVD backups, written and never used) I can see why they think like that.

With S3, a computer user need never make backups again. A simple invisible program (there are several on their way to the market) will work quietly in the background, taking anything that hasn’t been backed up recently and making copies of it on Amazon’s servers.

Too good to be true?

Altogether S3 looks like a Good Thing. It makes data storage really, really dull and boring. Duller than electricity, even: most electricity users have matches and a candle somewhere, just in case, but S3 doesn’t even need candles.

But is Amazon S3 boring enough to use? Can you build a business on it? If you do build a business on it, will anyone fund you or will their analysis throw up risks you never thought of?

These are the questions that this series of blog postings will explore.

Next time, to make the whole discussion more concrete, I’ll describe a business that could be built on top of S3.

S3 in Business: 2 - Tunesafe

Last time I promised you an outline of an Internet business that could be built on Amazon S3, so that we could see whether S3 was a solid enough foundation for a real business.

Rather than invent yet another Web 2.0 application, I’m going to concentrate on backup. We’ve had a little practice with this (the Cardbox Server already has automatic Amazon S3 backup built in) but the business I’ll be talking about will be much simpler than Cardbox and it’ll concentrate on nothing but backup.

The business need

I am impossibly virtuous when it comes to backup. Floppy disks (three sizes), tape (several sizes), CD, DVD, memory stick – I have used them all. Friends with data disasters have learnt not to cry on my shoulder, because they hate to see me trying not to look impossibly smug.

But I have my Achilles’ heel. It’s called an iPod.

My iTunes music library contains 23.4 gigabytes of music. That means six DVDs of slowly changing data, so obviously I don’t back it up daily. In fact I back up my iTunes once a month. Well, actually I backed it up once a month once, in February and I haven’t done a backup since. Here are my arguments:

  • The music exists in duplicate anyway, one copy on my iPod and one on my computer.
  • Most of my music comes from CDs, so I can always import it again if I have to.
  • Now that I’ve imported all my CDs, nothing much changes from one month to the next.
  • It hasn’t gone wrong yet, so it won’t go wrong in the future.

To take the arguments in turn:

  • Have you ever tried to copy music back from your iPod to your PC? Having tried, have you succeeded without making an unholy mess of the track titles? And anyway, are you 100% confident that iTunes, faced with an intact iPod and a wrecked PC (or vice versa) won’t replace the good data with the bad?
  • How many days and nights of soul-destroying labour will it take to re-import everything and correct the idiotic track names that the online databases assign to the music? And: when I said “most of my music comes from CDs”, that leaves a certain amount of stuff from other sources; especially the iTunes Music Store as well. Do I really want to buy every track all over again?
  • With classical music, the track data are a horrible mess. At least one night a week I do a little tidying up: for instance, I try to merge Jean Sibelius, Sibelius, “Sibelius, Jean” and “Sibelius, Jean (1865-1957)” so that they all look like the same person. Over the months, that adds up to a lot of work that can never really be duplicated. I wouldn’t have the time or the energy to sit down and do it all over again.

These are powerful arguments, but they’re still not enough to make me sit down regularly and create another six DVDs to add to all the others I’ve already got. So I continue to keep my fingers crossed…

The business

This is where our new S3-based Internet business comes in. It’s called Tunesafe.

  1. I sign up to Tunesafe. I download and install a small program from Tunesafe.
  2. Every night, invisible, automatically, the Tunesafe program uploads part of my iTunes music library to the Tunesafe service.
  3. Eventually all my library is there. From now on, the nightly backup only uploads any changes I have made recently, so that my Tunesafe backup continues to be an exact reflection of what I have in iTunes.

All in all, I have absolute data security without ever consciously having to back anything up at all.

The nice thing about constructing a business round backup is that it is a good, stable, long-term application. Web 2.0 businesses come and Web 2.0 businesses go, but backup goes on for ever. And people’s backup requirements can only increase with time.

Next time, I’ll describe how this will work if we adopt the straightforward approach of making the tunesafe.com server the central point of contact and hiding all the S3 functionality behind it.

S3 in Business: 3 - tunesafe.com

Last time I introduced Tunesafe, a program that backs up your iTunes music library continuously, invisibly, automatically. That’s how the user sees it, but how do we actually provide the Tunesafe service?

First of all, let’s see how it would work if we did it all by setting up a server on the Internet at tunesafe.com.

Technical

On the Internet there is a site called tunesafe.com. On that site there is a space that is designated as yours and is accessible only to you. That space is intended to hold an exact image of your entire iTunes music library.

On your PC there is a program called Tunesafe. The Tunesafe program lurks quietly on your computer. Whenever there isn’t much going on, it goes into action automatically. You could configure it to work at night, or only in the day (if it’s a home computer and you’re at work), or whenever the computer has been idle for a while.

The Tunesafe program makes a picture of your iTunes world: what music tracks there are, what they contain, and what their names are. It asks tunesafe.com for a similar picture, compares the two, and if any changes are needed to bring tunesafe.com up to date, (such as uploading new music tracks), it makes them.

Uploads can take a long time (we’ll cover this in a later posting) so Tunesafe is happy to be interrupted whenever you need the computer for something else: the next time it wakes up, it doesn’t begin again from the beginning but continues from where it left off.

Once everything has been backed up, Tunesafe continues to wake up at intervals, to see if you’ve made any changes to your iTunes music library: if you have, it makes corresponding changes to your space on tunesafe.com, so that the backup is always up to date.

It looks to you as if everything is being handled by tunesafe.com, but behind the scenes, all the music is stored on Amazon S3, so the tunesafe.com server that we set up won’t need much in the way of data storage.

In the same way, although it looks to you as if your music is being uploaded to tunesafe.com, it doesn’t really pass through the tunesafe.com server at all but directly from your PC to the S3 servers. Thus tunesafe.com doesn’t need to have much data transmission capacity either.

Commercial

The tunesafe.com site has to pay for its own server and it also has to pay Amazon the fees for S3 data transmission and storage.

How does tunesafe.com get the money? S3 charges for both data transmission and data storage. Because of the way iPods are normally used, it may be possible to charge a flat subscription rate or a graduated one according to capacity (so much for a 20GB iPod, so much for a 40GB iPod). After the initial 20GB (or 40GB) of upload, our data transmission costs are likely to be practically zero.

Otherwise we could charge for usage. Since the server knows what has been transmitted for any given user, and what is being stored for that user, it could charge the user’s credit card each month (just as Amazon are charging us) or it could take a deposit, eat away at it, and when it was about to run out it could ask for an extra deposit.

Technical risks

With backup, it pays to think in the long term and it pays to be pessimistic.

Anyone who has been in computing for any length of time has come across bitrot - the decay of digital data.

Physical bitrot is when you want to retrieve important data from an 8? floppy disk five years after you threw away your last 8?-disk PC (for a high-profile public example of this problem, see this article about the BBC Domesday Project). Amazon S3 eliminates physical bitrot because it eliminates our contact with physical media.

Logical bitrot is when the bits and bytes are readable but incomprehensible. NASA nearly lost the archives of their Viking project to this: as this article describes, they had CDROMs full of data from the Mars landing but no-one could remember what the data meant. We take great care that Cardbox should not suffer from this: we have always ensured continuity of data formats and can still retrieve users’ data from 8-bit Cardbox databases of the early 1980s. But we have been hit by logical bitrot ourselves. For almost a year we used a commercial backup program to create compressed backups on CDROMs, and then the manufacturer released a new version for Windows XP. The new version couldn’t read backups created in the old format. We abandoned that program but the year 2003 remains a thin one in our archives.

Protecting against logical bitrot is Tunesafe’s responsibility and we’ll take it seriously when we set up the system. The aim will be to allow retrieval of backup in some form even if Intel-based PCs become obsolete and the Tunesafe program dies with them.

Commercial risks

Bitrot isn’t the only risk. You are storing your valuable iTunes data in storage space that isn’t yours and that you don’t pay for directly.

  • What if the people behind tunesafe.com get bored and decide to do something else?
  • What if they go bust?

The trouble is, the commercial relationship is between tunesafe.com and Amazon: you, the user, are not involved. So if tunesafe.com stop paying Amazon, Amazon will quite reasonably erase tunesafe.com’s data - including your music.

We can try to build up your confidence in the stability and trustworthiness of tunesafe.com; but in the long term most businesses fail. It is 25 years since we launched the first version of Cardbox. None of the giants of the software world that were around then are still alive and independent today.

You can trust us” is a good message for any business to convey; but “you don’t even need to trust us” would be even better.

Next time, I’ll describe how Tunesafe can be made to work without a central server, so that if tunesafe.com disappeared tomorrow you could continue to make backups and retrieve your data from your backups for as long as you wanted.

S3 in Business: 4 - My Tunesafe

Last time I described tunesafe.com, a beautiful, smooth, elegant service with just one drawback: if the company behind it went bankrupt or changed its strategy, your service would end. With backups, this matters: you don’t want your backup data’s survival to depend on someone else’s financial stability.

So let’s take a different approach. Open an Amazon S3 account of your own. Amazon make this easy. They let anyone do it. They don’t charge you for opening the account and, month by month, they charge your credit card with just the cost of what you have used: data storage, and upload and download.

There will still be a program called Tunesafe on your PC, but this time it will be rather cleverer. It will maintain a duplicate copy of your iTunes music library directly on Amazon’s S3 space without needing a tunesafe.com server to help it. It will communicate directly with S3 for all its data storage and retrieval. If our tunesafe.com business disappears tomorrow, your Tunesafe program will carry on working just the same. Your contract is with Amazon: as long as you pay your bills, Amazon S3 will store your data.

There is just one snag left. We have arranged for you to pay for your data storage yourself – that’s no problem – but who pays us for the Tunesafe program that you’re using, and how? If the thing doesn’t get paid for somehow, then it won’t get written and you won’t be able to use it.

Next time, I’ll go into the ways in which the writer of Tunesafe might be able to make money out of it. This will include a suggested modification to the current S3 service that could lead to an explosion of new businesses built on S3.

S3 in Business: 5 - Money in golden buckets

Last time I described a variant of the Tunesafe business. Instead of requiring a special server at tunesafe.com, there’d be a Tunesafe program on your PC that would let you maintain your iTunes music backups directly on Amazon S3 without having to depend on anyone else. It all sounded perfect but there was the question of how that program would be paid for.

Selling the software

We sell Cardbox. People either buy it straight off or they get a trial licence and upgrade it when their month’s free trial period is over. Cardbox has S3 support built in. We even have a downloadable sample database that implements a photographic archive. I use it myself for all my digital pictures: it stores a 360×480-pixel miniature version inside a Cardbox record that can have as much or as little indexing information as you like, and it stores the original digital JPEG image on S3, for a cost of approximately a quarter of a cent per photograph per year.

But Cardbox is a proper, solid, general-purpose software package, with a 300-page colour handbook and everything, so it makes sense for us to charge money for it, and it makes sense to give away applications like the Photo Archive. Tunesafe is quite different from Cardbox. Without S3, it’s useless. It’s simple, single-purpose, with no printed documentation, and practically invisible in normal use.

If Tunesafe were sold for money (the way Cardbox is), the price would either be too expensive for the customer or too little for anyone to make a living out of it. And the payment comes at the wrong time: at the beginning, when the customer hasn’t yet learnt how much he really needs the product.

The clincher is that an Internet business doesn’t really want paying customers. If you have a million paying customers, that means a million sales invoices to generate and audit and a million separate transactions that may go wrong and require human intervention.

That’s why it’s so attractive when you have a web site that pays for itself through advertising: it means that you have one or two big sources of revenue instead of a lot of little ones… and half the time they even generate your invoices for you, and pay you without being asked.

Not selling the software

Since every Tunesafe customer already has to sign up with Amazon to pay for the storage he uses, why not take advantage of that fact? Here’s where a simple addition to Amazon’s offering would make all the difference.

Let’s think about telephones for a moment. When you make a normal phone call, you pay the phone company the cost of the call. When you call a premium-rate number (900 in the USA, 070, 0870, or 090 in the UK) you pay the phone company quite a lot more. The phone company deducts the cost of the call and passes the rest of the money on to the owner of the number being called. So a lottery information line receiving a thousand calls a day effectively has just one paying transaction per month instead of 30,000: and that one transaction is with the phone company, which rarely argues and always pays its debts on time.

So the proposal is that Amazon S3 should have the equivalent of premium-rate phone lines. Since Amazon calls its fundamental unit of data segregation a “bucket”, the obvious term is “golden buckets”.

A golden bucket works just like any other S3 bucket except that data transfer and data storage are charged at a percentage of the normal rate – 150%, perhaps, or 200%. The customer pays Amazon at the higher rate, Amazon deducts 100% of the normal rate and pays over the rest to the bucket’s sponsor. Knowing Amazon, they’ll probably take a cut of the payment that’s made to the sponsor, just as they do with Amazon Marketplace.

Everyone wins.

  • Amazon use their existing payment infrastructure (the user had always been going to pay for a normal bucket anyway) but get a lot more money out of it.
  • The customer gets a service he wants without having to pay for the software to access it.
  • The sponsor / software supplier gets revenue from just one source, Amazon. It has an incentive to create good software that people will want to carry on using for ever. If someone downloads the software, uses it once and then throws it away, it simply won’t generate the revenue.

I’m suggesting that Amazon should make a change in their S3 service. How much work is there in this for them? Not much. On the technical side everything works just as it always does. The only real work will be on the administrative side: they’ll have to ensure that the customer explicitly knows when he’s signing up for a golden bucket, since they’re more expensive than ordinary ones. They’ll have to ensure that the Tunesafe program can verify that the bucket it’s being asked to use for backups really is a golden one and really is sponsored by tunesafe.com: we don’t want a clever user spoofing the Tunesafe program into offering a free service.

The benefits of this change are huge. Putting this sort of structure in place will enable the creation of a whole class of new Internet businesses with minimal startup costs: it liberates them from the burden of setting up and maintaining invoicing and credit card facilities, just as S3 itself has liberated them from the burden of setting up and maintaining massive storage facilities.

Next time we’ll have a slow interlude, to give Amazon time to digest this proposition.

S3 in Business: 6 - A slow interlude

Resting from the excitement of thinking up an entire new business model (last time), it’s time to do some sums about speed. How practical is it to use S3, or any other Internet service, for backups? Let’s look at how fast data can travel between your PC and Amazon S3.

ADSL stands for “asymmetric digital subscriber line”. A 2Mb ADSL line has an upload speed of 288kb (36kB) per second, with the rest of the speed being used for downloads. That upload speed equates to about 8 hours per gigabyte.

For my 23.4GB iTunes music library, this means a total backup time of 187 hours. Whether you do nocturnal backups (8 hours per night, every night) or daytime backups while you’re away at work (11 hours per day, Monday to Friday) that works out at just over three weeks to back up my entire music library.

(For an 8Mb line the figures are 3.3MB per minute, or 5 hours per gigabyte, adding up to 120 hours, or about two weeks).

This sounds horrible. It’ll be enough to put many people off, but we’re rescued by a couple of facts. First of all, the Tunesafe program is designed to do a little work at a time and it doesn’t mind being interrupted: however long the journey, with one little step after another you’ll get there in the end. The other helpful fact is that iTunes music libraries are quite special things. The individual files within them are not all that big and they don’t change much over time, so that once your library has been backed up it stays backed up. After the initial burst of activity in the first few weeks, the uploads will slow to a trickle and your ADSL connection will be able to handle them easily.

Technical point: If you’ve rummaged inside your music folders then you’ll have seen that whenever you take a track and change something simple like the name of its artist or composer, the entire MP3 file changes. This is why an ordinary file backup program isn’t good enough for iTunes music: change “Mozart, Wolfgang” to “Mozart” and you could let yourself in for a whole extra night’s worth of backups! Tunesafe is cleverer and it doesn’t have to re-upload an entire music track just because you’ve changed the track title or the artist’s name. This is one of the reasons why Tunesafe is worth paying for.

If you ever need to restore from your backup, you won’t have nearly as long to wait. Because ADSL is asymmetric, downloading is a lot faster than uploading – nine times as fast on a 2-megabit line or 17 times as fast on an 8-megabit line.

If you’re not looking at ADSL but at a project where you’re transmitting data to and from an Internet server, everything is a lot faster but you should be aware that Amazon S3 isn’t worldwide yet. To give an idea of the speeds you might expect, uploading from our Rackspace server to S3 took 40 minutes per gigabyte over the transatlantic link from the UK. This is a lot faster than uploading from a PC, but it certainly isn’t instantaneous and it’s worth taking account of it if your business plans are international.

Next time, we’ll start thinking like potential investors in the Tunesafe business. We’ll look on the dark side: what are the risks of depending on Amazon S3?

S3 in Business: 7 - An introduction to risk

Let’s imagine that Amazon have thought a lot about it and eventually taken up the Golden Buckets idea and let’s think forward a few months, to the moment when we’ve done the planning, done some programming, and are starting to approach possible financial backers.

As such people do, the backers appreciate our excitement but see it as their job to contribute the prudence that we evidently lack. Alternatively, they are a bunch of cynical pessimists looking to talk down the value of our project so they can get more of it for their money. It doesn’t really matter which description you choose: the fact remains that they come up with a list of possible things that might go wrong. For each risk, they want to know how likely we think it is to happen, how much it will cost us if it does happen, and how we can protect or insure against it. Even if we don’t have backers and are funding it all from our own pocket money, we still need to consider these risks and ask ourselves the same questions.

I’ve divided the risks into seven classes:

  1. Credit risk: what if people get stuff from us and then don’t pay? This doesn’t apply to the Tunesafe project because, with golden buckets, none of our users pays us anything anyway. The only people who do pay us are Amazon, and their creditworthiness can be neatly considered under…
    Commercial risk: this business depends entirely on repackaging one service from one supplier. How vulnerable are we to that supplier’s actions?
    Technical risk: this business assumes that Amazon S3 works. What if it doesn’t, or is unreliable?
    Legal/contractual risk: does the contract we have with Amazon allow us to run Tunesafe? Could it be used to stop us later?
    Political risk: could users of Tunesafe be sent to prison?
    Client-side risk: what if people can’t run the Tunesafe software? Worse, what if they do run it but something happens later on that makes them unable to connect to their data?
    Slashdotting and doshslatting: what if our site is overwhelmed with traffic and we are bankrupted as a result? This doesn’t apply to the Tunesafe project because users pay for their own data transfer directly, so it doesn’t cost us anything, but it is still a risk that concerns a lot of potential Amazon S3 users.

Next time: commercial risk. How dependent are we on Amazon’s commercial decisions? What can Amazon do to reassure us? What can we do to protect ourselves?

S3 in Business: 8 - Commercial risk

It is risky to offer a business that depends entirely on a service provided by someone else. Amazon S3 might stop tomorrow, or it might become too expensive for us to use. It’s only been going for a few months, so we don’t even have a track record to go by. Here are some of the things that might happen to S3 in the next few years:

  • Amazon go bust.
    Amazon are taken over by someone with no interest in continuing Amazon Web Services as an external product offering.
    The stock market decides that Amazon ought to stick to selling books, and, faced with a steep decline in its share price, the company obeys.
    At the next scheduled review of the project, Amazon decide that Amazon Web Services hasn’t been the success they hoped. They get tired of it and pull the plug.
    Amazon decide that it’s too much bother dealing with all those tiny customers with tiny payments, and impose a $100-per-month minimum spend on S3.
    Amazon realise they got their sums wrong and double their prices.

We are like a fly hitching a ride on a racehorse: it is taking us where we want to go, fast and with no effort, but we could be swatted off at any moment. How do we reassure our backers that we are safe? How can Amazon reassure us that we are safe?

Contractual reassurance

Amazon need to reassure us that S3 will not be switched off tomorrow. How do they do this while keeping their commercial freedom of manoeuvre? It is for them to think of something and for the market to decide whether it is good enough. Perhaps they could write a notice period of a few months into their contract, long enough for users of S3 to retrieve their data and store them somewhere else. They might also ring-fence S3 financially in such a way that it would continue for at least the notice period (possibly with reduced performance) whatever else might be happening at Amazon as a whole.

S3 without Amazon?

When the Cardbox Server does its automatic backup to S3, it sends S3 commands to a server and waits to receive S3 responses back from the server. Tunesafe would work in the same way. The server used is Amazon’s S3 server at s3.amazonaws.com, but it doesn’t have to be that particular server. Cardbox (and Tunesafe) would work just as well with any other server, as long as it also understood S3 commands and sent S3 responses in return.

If a different data storage supplier also supported the S3 protocol then we could choose whether to use Amazon’s service or its rival’s, and if one company failed then we could switch to the other one. There is nothing new about second-sourcing, which has long been a standard method of risk limitation in the defence and aerospace industries. There is an art to it: you want the second source of your designs to be big and efficient enough to be credible but not so big and efficient that they will take too much business away from you.

Amazon claim that the big value in S3 is not the S3 protocol itself - anyone could have designed that - but the enormous investment that they have made to ensure that information stored in S3 is always available, available fast, and never lost. That being the case, it would not be a threat to their business if someone else also offered an S3-protocol service. It would encourage customers to commit themselves to S3, knowing that they would not be completely dependent on Amazon’s continuing benevolence.

One could even imagine Amazon publishing a simple open-source program that turns any single computer into an S3 server. A single computer (or even a single data centre full of computers) would not be a rival to the full S3 service, but it would mean that no matter what happened - even if every contractual assurance offered by Amazon failed - a business dependent on S3 would have something to fall back on.

If this discussion seems a bit abstract, let’s compare Amazon’s data storage service with the electricity supply. Defining the S3 protocol is the equivalent of deciding that electricity should be provided at 220 volts and 50 cycles per second. Once that decision has been made, the market for electrical appliances blossoms because everyone can build appliances to use just one kind of supply. If some people get themselves diesel generators that generate 220V/50Hz as a backup in case the mains supply fails, that is no threat to the electricity company - it’s actually a benefit, because it means that those customers won’t look for alternative kinds of power. Even giving away diesel generators would make sense, if they were cheap enough, because no-one would rely exclusively on generator power when the mains supply was cheaper than diesel, more reliable, and of unlimited capacity. (If you are in America, please read “220" as “110" and “50" as “60".)

Allowing widespread use of a protocol one has developed is not quite risk-free and Amazon will have to be careful how they do it. But if they can state publicly that they have no problem with certain kinds of services that use the S3 protocol then that is a perfect additional reassurance for us to give to doubting investors. “We use Amazon S3,” we can say. “We hope that we’ll always use Amazon S3. But if ever we can’t, for whatever reason, then we do have an alternative that will work without any of our systems having to be changed”.

Next time, technical risk. What if Amazon S3 doesn’t work exactly as advertised? What if it starts forgetting things from time to time?

S3 in Business: 9 - Technical risk

Last time, I covered the commercial risks associated with Amazon’s continued existence and continued offering of Amazon S3. Now it’s time to turn to technical risks. How reliable is the Amazon S3 service? Can we trust our backups to it? Can we build a business on it?

Computers go wrong. That is what they do. The job of a service provider is to build a reliable service on top of unreliable machinery. Amazon don’t say much about how they achieve reliability except that they have multiple data centres in different locations and S3 doesn’t even acknowledge receipt of a data object until it has been successfully stored in at least two geographically separate places. The idea is that a failure in one place is improbable but a failure in two places is inconceivable.

One can measure reliability in two ways: the percentage of time that the service is available, and the probability of a stored data item being lost or corrupted. In the case of Tunesafe, an occasional loss of service doesn’t matter much (the Tunesafe program can just go to sleep for a bit and try again later) but the loss or corruption of data is very serious indeed. It is in the nature of backups that no-one looks at them for years and years, and this means that any errors that have been introduced into the backup are discovered far too late for anything to be done about it.

Amazon make a claim about service availability (at least 99.99%, which means one hour’s downtime per year) but they make no claims about the integrity of the stored data. And they make no contractual promises at all about anything.

Amazon make a great thing of the fact that they use Amazon S3 themselves for their own business. This is a great recommendation but it isn’t actually a guarantee. Quite apart from anything else, we can’t compare Amazon’s tolerance for data loss with our own. If we use S3 as our only backup we may be far more sensitive to losses than Amazon are. Amazon with one or two books missing is still Amazon: a backup with one or two files missing is practically useless.

Why data loss is an easy risk

Data loss has one encouraging property: it is easy to tell whether it has happened. With S3 it’s easy to prove whose fault it is, as well: if Amazon maintain a tamper-proof log of every PUT and DELETE request to S3 then anyone can tell, objectively and beyond dispute, whether any object stored on S3 has (a) disappeared without a DELETE request (b) changed since the PUT request that created it.

So we have something that

  1. is a rare event
  2. causes harm
  3. can be identified with certainty.

An event like that is a prime candidate for insurance.

Suppose that Amazon paid compensation for each lost or corrupted S3 object – perhaps a thousand dollars per object, perhaps one cent per byte – how much would that cost them? We don’t know, because we don’t know the quantity of data that S3 loses; but Amazon do know, or they can get a mathematically inclined intern to find out.

Suppose that Amazon divide that total cost of compensation by the number of S3 requests received. Does the total come out at one cent per gigabyte? More? Less? Again, we don’t know but Amazon do.

A revolution in insurance

Traditional IT insurance is cumbersome and expensive. It requires the insurers to assess the degree of risk beforehand, with experts going through every bit of software, hardware and operating procedure. Handling claims is expensive as well, since the insurer has to assess exactly how much harm has been done to the business by an IT failure.

Things don’t have to be this way. To see this, let’s take the second point first. The “purest” form of insurance is pluvius insurance - you are having an open-air event on a certain date and you pay a premium so that you can be compensated if it’s rained off. Pluvius insurance is pure (a) because the insured person has no influence over whether the loss happens and (b) consequently the insurance can be for any amount that the insured person feels like. If I insure my local school’s fund-raising fair for $10m, no-one cares as long as I pay the premium: I can’t make it rain. So assessing the risk is as simple as reading weather records, and paying out the claim takes no time at all.

The kinds of insurance we’re familar with are “impure” by comparison, because the insured person has a strong influence over the probability of loss. My car insurers will want to know in advance what sort of driver I am and whether I live in a rough neighbourhood; and if my car is wrecked or stolen they won’t pay me any more than they say it’s worth. If it were otherwise, I could turn my car into money a lot more easily than by selling it second-hand. (I am told this used to happen in parts of Australia where they had fixed-value car insurance: you’d park your car in a certain spot with A$200 on the front seat and the keys inside the exhaust pipe, go for a drink and come back to find the car gone.)

Our risk of data loss and corruption is more like rain than like car theft. It is easy to assess in advance (if you know the error rates) and because we cannot influence the outcome, it can be insured for any amount that we feel like insuring it for, without any need for tedious calculation of the exact value of the loss suffered. All this would make S3-based data loss insurance very, very cheap to administer. It would lead a revolution in insurance just as S3 itself is leading a revolution in data storage.

Who should offer this insurance? The obvious answer is Amazon. They have the exact figures for the probabilities of data loss and no-one else has. It is like offering life insurance when you are the only one with accurate mortality tables. How they would do this would follow the pattern of many other industries around the world: they set up a wholly-owned insurance subsidiary, which offers data loss insurance for data stored on Amazon S3. That subsidiary underwrites part of the risk and reinsures the rest of it in the wholesale insurance market.

Amazon can thus give us a choice: the current S3 storage service at the current low cost, complete with all the empty guarantees about “we use our own service so it must be safe”; or an insured service for a few cents extra.

A cautious user of S3 could pay the extra premium; a braver user could use the size of the premium as an indication of how risky S3 storage actually is.

A revolution in IT responsibility

There is a Peanuts cartoon in which Lucy goes round getting everyone to sign a piece of paper that absolves her from all blame. No matter what happens any place or any time in the world, this absolves me from all blame!

When Lucy grew up she went into IT.

Every software and hardware supplier has contracts absolving itself from all blame if its products go wrong. As a result, IT remains an infantile industry. Because blame is disclaimed, there is no need for anyone to take responsibility. Moreover, where there is no blame there is no apportionment of blame. For example, if someone has difficulty printing, there is no way of finding out whether the cause is (a) the application program, (b) the operating system, (c) the printer driver, (d) the print spooler, (e) the printer firmware, (f) one of the fonts. And that is assuming that you’re not printing across a network!

As suppliers of Cardbox we spend a lot of time pinning down the causes of problems suffered by our users. We always hope that the cause is a bug in Cardbox, because then we can cure it; but all too often the cause is somewhere in the thicket of letters listed above. And there is no quick way of pinning it down, because none of the components have been designed to do a proper audit trail to help with troubleshooting. So the user suffers; and every day computing becomes less like engineering and more like medicine.

Suppose that Tunesafe wanted to offer insured backups to its users. It could use insured S3 storage, which accounts for most of the risk. The remaining risk would be that data loss or corruption was being caused by bugs in Tunesafe. Tunesafe being small and simple, that risk wouldn’t be too hard to assess and the company could decide to carry it itself. Of course there is also the risk of a greedy user trying to make Tunesafe go wrong so that he can get his hands on the compensation, so the Tunesafe program will have to have a secure log so that it can be proved whether (for example) a backup has been deleted because the user asked for it to be deleted rather than because something went wrong with the program.

So now blame can be reliably apportioned between “Amazon S3?, “Tunesafe”, and “elsewhere”; and the logs that Tunesafe keeps in self-defence will also be a valuable tool for debugging the rest of the complete system. Thus, one slice at a time, IT can be de-Lucified, with each component in a complex system able to identify whether it is to be blamed for an error or whether the error is someone else’s fault - and, if so, whose. It will take a long time, but S3 is a very good place to start.

S3 in Business: 10 - Contractual risk

Last time, I covered the technical risks associated with using Amazon S3, investigating how sure we could be that S3 was reliable enough for our purposes.

Now (rather late) let’s turn to the simplest question of all: will Amazon let you use S3 to back up your music?

You will not use Amazon S3 in … any manner that might be discriminatory based on race, sex, religion, nationality, disability, sexual orientation or age.

So before you use Tunesafe, you must erase Debussy from your iPod (“The Golliwog’s Cakewalk”), all of Wagner (a known racist), and anything by NWA - to name but a few.

Yes, you say, but surely I could encrypt everything so that Amazon don’t notice?

You agree to provide such additional information and/or other materials related to your Application as reasonably requested by us or our affiliates to verify your compliance with this Agreement.

Well, no, actually. Amazon require the right to listen to all your music to make sure that it is the sort of thing they want to have on S3. This is reasonable. How can they tell whether your encrypted confidential data are illegal or not, if you don’t let them read your data?

What if we ignore this and go ahead anyway?

If your Application is determined (for any reason or no reason at all, in our sole discretion) to be unsuitable for Amazon Web Services, we may suspend your access to Amazon Web Services or terminate this Agreement at any time, without notice.

How thoroughly will Tunesafe have to censor its users’ content? Would it be OK to have music by Tom Lehrer on your iPod, or is poisoning pigeons in the park an illegal activity? (after all, the Audubon Society objects to it). To answer this kind of question definitively you will have to have a detailed knowledge of the criminal law of the state of Washington.

Your lawyers will not be idle. Apart from keeping tabs on Washington law and Washington politics they will have to check the Amazon Web Services Agreement hourly, since

We may modify any of the terms and conditions contained in this Agreement, at any time and in our sole discretion, by posting a change notice or a new agreement on the Amazon Website. Your continued use of Amazon S3 following our posting of a change notice or new agreement on our site will constitute your binding acceptance of the change.

…so that even if possessing Poisoning Pigeons in the Park isn’t grounds for erasure today, a quick addition of “cruelty to animals” to Amazon’s list of Banned Things could easily make it so tomorrow.

Committing crimes with electricity

Amazon S3 is meant to make data storage into a commodity, just like electricity. So I called my local electricity company to ask them if I was allowed to use electricity for criminal purposes. If they knew I was committing a crime, could they cut me off? Could they monitor my activities in order to check whether they needed to cut me off?

No, they said. I can use electricity to make defamatory statements, or to publish racist and genocidal propaganda, or to be cruel to badgers or bats. I could even use it to grow cannabis plants (except that British cannabis growers steal their electricity by bypassing the meter). As long as I pay my bills, the electricity company will continue to supply me with power to continue my illegal activities.

The contractual basis of doing business with Amazon

Amazon will claim that their licence agreement was drafted by a moron in a hurry and that they don’t really mean any of it. “We don’t mean any of these nasty things, really. You know we’re nice guys so just go ahead and don’t worry about the contract”.

This is a widespread modern trend in both contracts and legislation. It puts the customer (or the citizen) permanently in the wrong but spared the consequences of his wrongdoing by magnanimous corporations and governments. It essentially abolishes the rule of law and replaces it with administrative fiat.

The Amazon licence agreement is not a suitable basis for any serious commercial relationship and anyone who bases a business on this contract is failing in his duty of care to himself and his investors.

Why the contract is the way it is

Internet law is a shambles – international, national and state law all cemented together into a legal breccia, with no-one knowing quite where the lines of fracture will go when the whole thing is stressed. There is no universal consensus as to where criminal liability lies for the storage or transmission of data. Different jurisdictions take contradictory approaches. Some of them enact absurd laws and trust that they won’t be used: in the UK, for example, if I ring up a friend and encourage him to have a cigarette to get over an emotional crisis, my telephone company has committed an offence simply by transmitting my voice.

It is natural that Amazon’s lawyers (aware that their company is a large target with deep pockets) try to shrug the entire liability onto their users by banning any activity that might get Amazon into trouble. It is also natural that the users, reading the contract carefully, will avoid any activity that might get Amazon into trouble by avoiding any activity at all involving Amazon.

For the health of the entire Internet services market, someone has to establish the extent to which data storage and transmission are to be treated, like electricity, as an opaque commodity without a legal weight of their own. I can’t do it and Tunesafe is too small. It is the responsibility of large corporations such as Amazon to buy a few legislators (preferably, ones with international connections) and get the job done properly, to the benefit of us all.

Next time, how Amazon’s future actions could mean S3 users being liable to arrest and imprisonment if they ever travel to the UK.

S3 in Business: 11 - Political risk

Last time, I covered the legal risks of entering into a contract with Amazon for S3 or its other web services. The complete loss of your service and data at any time, for any reason or for no reason at all, may seem a bad enough risk; but there may be worse to come.

Amazon boast a network of data centres. At present this “worldwide” network only covers countries that send teams to the World Series; but at some time in the future Amazon plan to put data centres in Europe and elsewhere to increase access speed, which is not really adequate outside North America (see A Slow Interlude).

Let’s imagine that Amazon place a data centre in the UK. This means fast access for UK users, who will no longer have to rely on their ISPs’ transatlantic links.

As part of Amazon’s worldwide network of data centres, the UK centre will receive replicated copies of data from all over the place - not only data originating or being used in Europe - as Amazon’s proprietary algorithms ensure maximum data security through maximum geographical diversity. If you scan a document in Peoria for use in Chicago, and store it in S3, it is as likely to end up in England as anywhere else.

You will probably be prudent and encrypt your data.

In England and Wales, the Regulation of Investigatory Powers (RIP) Act 2000 says that as part of an investigation “in the interests of national security, for the purpose of preventing or detecting serious crime, or for the purpose of safeguarding the economic well-being of the United Kingdom” the police may demand the keys to any encrypted data. It is an offence (with sentence of up to two years in jail) to fail to provide your encryption keys, and it can also be an offence to reveal the fact that the keys have even been demanded. Ross Anderson, the Professor of Security Engineering at Cambridge University has been forced to cover himself like this:

Here is my PGP key. If I revoke this key, I will always be willing to explain why I have done so provided that the giving of such an explanation is lawful.

So, suddenly, simply because Amazon have replicated some of your data into the UK without your knowledge, you open yourself to police demands to reveal your encryption keys with the threat of a prison sentence if you don’t.

A great deal of work is being done on this: see, for example, this blog entry on Cambridge’s Scrambling for Safety conference, which was devoted entirely to the RIP Act (there is BBC coverage here). On the government side, the draft code of practice is all emollience, containing safeguard after reassurance after safeguard in the best manner of the British Civil Service. But these reassurances ultimately come from the government of a country where a Labour Party member (an 82-year-old Holocaust survivor) was detained under anti-terrorism legislation for daring to heckle a minister at his own party’s annual conference (BBC report here), and so we may be forgiven for not being entirely reassured. How long before “the economic well-being of the United Kingdom” is interpreted as “the profits of a company in which the ruling party owns shares”?

I have mentioned one country but there will be others. The restrictions and penalties on S3 data will be a combination of all the possible ones in all the countries where Amazon choose to have a data centre. Do you really want to investigate the law and politics of every country where Amazon might decide to put a computer? This kind of international political risk can only get worse as Amazon’s data centres spread round the world.

Perhaps now we know why Jeff Bezos is so interested in space flight. S3 servers in low earth orbit could in the end be cheaper to buy than politicians. Googling for “low earth orbit routing protocol” already returns 12,800 pages

Next time I’ll mention a collection of risks that I’ve filed under “M for miscellaneous”: denial of service by anti-virus software, and cost inflation through slashdotting and doshslatting. It’ll break all my previous promises by being quite a technical posting, but be patient because the time after next will bring the conclusion to this whole series. Thank you for reading so far.

S3 in Business: 12 - A miscellaneous interlude

Every good filing system has a section called “M for miscellaneous”. So does this blog series.

Anti-Internet software

It is possible for an attacker to provoke Norton Internet Security into blocking all access to Amazon S3 from your computer. So if a Tunesafe user visits a site that contains a specially designed page - or views a forum entry containing a specially designed image - his Tunesafe backups will cease to be accessible.

This does not simply affect Norton Internet Security and Amazon S3: it is potentially a problem for all Web services when used with any anti-Internet software. There is a detailed blog posting here and an executive summary here.

Slashdotting

Slashdot is a site that offers “news for nerds”. To appear on its front page is to be read by hundreds of thousands of people. If the news story contains a link to your web site, very many of those readers will visit your site in a very short time, probably leading to the complete collapse of your server under the load. Your site has been “slashdotted”.

A malicious analogue of slashdotting is also possible. One of the more vicious Internet worms of the last few years aimed to recruit a network of millions of computers all of which would attack one site (such as a US government one) at the same moment. That particular attack failed because the worm was not well designed; but it is quite common to find high-profile sites such as Worldpay having to operate at reduced capacity for several days as a result of a denial-of-service attack.

One of the attractions of putting your publicly accessible data on Amazon S3 is that you are immune to slashdotting.

One of the dangers of putting your publicly accessible data on Amazon S3 is that you are immune to slashdotting.

Why is it a danger? Because a normal slashdotted site becomes slower and slower and eventually crashes or grinds to a halt. This limits the size of the attack. But if you have a 2MB photograph stored on Amazon S3, access to it will be as fast with a million visitors as it is with one; and at $0.0004 per visit, a million visitors will cost you $400. You are safe from a denial-of-service attack but open to a draining-of-bank-account attack.

I haven’t gone too far into this risk because Tunesafe is immune to it: nothing is publicly accessible at all, and each user’s backups are stored in his own private data space on S3. But if you are planning a public service, assess the risks carefully. Various S3 users are lobbying Amazon for automatic limits and cutouts so that access to their data is suspended in case of attack. Amazon are quite reasonably resisting their requests for now. For a start, it would imply real-time gathering of billing information across Amazon’s network, whereas at present they can get away with consolidating the data at relatively long intervals. This may end up as one of those risks that just have to be budgeted for.

Doshslatting

Doshslatting is slashdotting backwards. Amazon charge for uploads as well as downloads: so an attacker could wreck your budget by uploading vast amounts of data. (Again, because it doesn’t have publicly accessible data spaces, Tunesafe is immune to this.)

Doshslatting can only be malicious, which makes it rather easier to deal with. Given the cheapness of S3 and the slowness of data transfer, an attacker would have to have access to a “botnet” of thousands of subverted PCs in order to cost you very much money. The motives for doshslatting could only be vandalism and extortion. Given that a botnet can be used for many lucrative purposes, vandalism isn’t worth the bother; and extortion is not only criminal but necessarily involves contact between the extortioner and his victim. There are easier ways to make money out of botnets.

Amazon S3 already contains partial protection against doshslatting. Rather than having a publicly writeable area of your storage space, you can make all of the space private and accessible only by signed requests from a client. You then make your web application work as follows:

  1. The web browser asks your own server for a signed write request.
    Your server considers the identity of the user, and if it thinks the user is legitimate it creates a signed request and sends it to the browser.
    The web browser sends the signed request to Amazon S3 along with the data it wants to upload.

This is good, rational protection, and it works. If Amazon allowed the signed request to include a size limit - “please write a data object, the object to be no more than 4MB in size” - things would be better still (technically, the modification would be very easy indeed). What is even better is that signed requests in S3 can have time limits, so your server can issue a request that will only work for 15 seconds after it was created. This will prevent most kinds of botnet attack.

Summary

The anti-Internet problem
This will become more and more of a problem as the use of web services grows. The problem will be stopped eventually, because the makers of anti-Internet software will have to make their products work rationally or people will stop buying them.
Slashdotting
There are technical impediments to Amazon providing precise rate-limiting mechanisms, but an imprecise one may eventually become available (eg. limiting starts when the traffic has reached somewhere between 1 and 5 times the chosen limit). Even so, this is a risk that has to be borne by the business that uses Amazon S3. Methods of monitoring and mitigation have to be considered and a budget set aside to pay Amazon’s fees if a surge in demand does occur. Not every risk can be prevented or pushed onto someone else’s shoulders!
Doshslatting
Methods exist to reduce the risk to the point where only a really dedicated attacker could have a measurable impact; and such attackers will have many more lucrative things to do with their time and resources.

Next time, this series is brought to a conclusion.

S3 in Business: 13 - Conclusion

I am writing the first draft of this posting less than 24 hours after the announcement of the public beta release of Amazon EC2 (Elastic Computing Cloud) and it is making me relive the emotions that I and many other developers felt when Amazon S3 was announced. A shared culture is clearly at work. In each case, Amazon create and offer a service that has very few features: but those that it has, it implements consistently, cleanly and robustly. The documentation has the forceful simplicity of a 1970s IBM manual, saying exactly what needs to be said, straightforwardly and unambiguously: it has been written by someone with a mind with the intention that it should be read by someone with a mind. The developer forums are monitored by intelligent people who get to the bottom of the questions that are asked - sometimes deeper than the questioner himself - and give answers uncontaminated by considerations of marketing or public relations. Nothing like that has been seen since the Microsoft Windows 3.1 beta test programme.

Outsiders do not realise how deeply technological development, like scientific research, is shaped and driven by emotion. If you have ever looked at an iPod and wished that you needed one, you will have had an inkling of it. The reason that half of us are in computing at all is that we see computers as things that we can make beautiful things out of: and S3 and EC2 arouse the same emotion.

Some people have criticized S3 and EC2 for being bony, but that is the point of them. You cannot go wrong if you have good, strong bones to build on. If the foundation is right, things just go on getting better. A woman with good bones is six times as beautiful at 60 as she is at 20.

To take one example: when an EC2 computer is shut down, it evaporates. Completely. It is as if it had never been. In particular, any data that it stored are lost. This sounds ridiculous, but it keeps the EC2 architecture simple and beautiful. Anyone building a system on EC2 already has access to data storage of great robustness and infinite capacity: it’s called S3. So each system developer can put together the bones - instantly expandable computing capacity and bulletproof data storage - to make the body he needs. There is no need to contaminate EC2 by building into it assumptions about whether persistent storage is needed, and how much of it is needed, and how it should be administered. Each developer can make the decisions that are right for him and not be constrained by someone else’s idea of what he ought to want.

Conclusion

This series has been about applying the excitement generated by S3 (and now, by EC2) to doing real work in the real world. It has been about reconciling the romantic thrill of “darling, we’re going to have a baby” with the reality of coping with something damp that yells.

For server-based services such as tunesafe.com, S3 is ideal. The user interacts with the web server but all big data transfers are shunted directly into and out of S3. S3 costs money, of course; but it costs so little money that in many cases (such as probably tunesafe.com) it is possible to charge a flat fee because the wholesale data storage and data transfer costs can be bundled up into a single retail offering that still has an attractive price.

For startups that want a million users but don’t want to issue a million invoices a month, the availability of premium-rate storage (”golden buckets”) would be a major enabling technology. The profit from the golden buckets would pay for software development and the maintenance of the necessary servers; Amazon would take their cut; the user would have just one monthly bill to pay; and again, because S3 is so cheap, even a premium-rated version of it would be competitive and affordable. Whole new business models will be possible as a result. The demand is certainly there from developers: we all hope that Amazon will get round to providing this option.

No business plan is complete without some assessment of risk. For a start, Amazon need to find a way of convincingly reassuring us that we are not betting our companies on Amazon’s continuing interest and benevolence. Possible reassurances include the provision of limited guarantees of continuity or the availability of rival services that behave identically to S3. It might even be in Amazon’s interest to help set up such services.

S3 is claimed to be reliable but there are no figures to support this and users of S3 are thus taking on an unquantified risk. Quantifying the risk by publishing data is one way forward. A more interesting approach is to offer insured data storage for a small extra premium. Apart from being potentially extremely profitable, this could catalyse the transformation of the computing industry into one that takes responsibility for its products.

The contract under which S3 is currently offered is entirely unsuited to its purpose. It is not only unacceptable to anyone who plans to depend on S3, it also contradicts everything we know about Amazon’s intentions and corporate philosophy. It needs to be torn up and rewritten from scratch.

Amazon intend to spread their data centres worldwide so that computing capacity (EC2) and storage capacity (S3) are available at equally high speeds all round the world. They must pay serious attention to the legal consequences of storing data in various countries because there is no way for a user to control where, physically, data are stored in the S3 network. There is no point in virtualising away the technical differences between storage technologies if you cannot similarly virtualise away the legal differences between various jurisdictions. It is impossible to expect every Amazon S3 user to ensure that his data comply with the laws of the United States, Britain, Spain, China, and so on. In a world where every country tries to extend its jurisdiction over the whole of the Internet, we need Amazon to work on our behalf to obtain some sort of extraterritorial status for our data.

Let me finish by referring back to the start of this posting. With S3 and EC2, Amazon are revitalising computing by bringing it back to its roots. They have started a revolution. If some of my criticisms have sounded harsh, it is because I am applying some of Amazon’s own perfectionism to the services they have created and because I think that they deserve a large share of the profits from the revolution that they have started.