We use a fair amount of storage for all the metrics we capture, and we use different storage for different reasons. Some of our data resides within MySQL, some in Redis, and the largest amount of our data (the metrics themselves) actually reside in a purpose-built datastore we internally call NestDB (you put eggs in a nest, you can see how we got the name). NestDB is interesting in itself, but we won't talk about that here. One of the most important requirements for most data stores is consistent, high performance storage. Lets put down our wish list of requirements for storage:
1) Persistent – does not go away when the system reboots or crashes
2) Stable – available when you need it
3) Performant – does the requested amount of actions per second (be it operations or bytes)
4) Predictable – does those actions in a predictable amount of time over time
5) Cost effective – can't break the bank to fulfill the requirements
Amazon has a few storage options (we're talking block storage here, not object stores like S3): Instance store, EBS (Elastic Block Storage), and now Provisioned IOPS EBS. Lets talk about each one.
Instance store is fairly predictable since the storage is 'local' to the instance (meaning it is not shared with other instances). The performance is ok, but really not great, and you can't get any more from it by striping it since you get a fixed amount. The biggest problem is if the instance goes away, you lose all the storage unless you make AMI's but that is not really practical when talking about a database or other quickly changing application.
Instance store effectiveness:
Persistent: [ ] Stable: [#### ] Performant: [## ] Predictable: [### ] Cost effective: [#####]
EBS volumes are persistent, and you can take snapshots of them and create new volumes from those snapshots (which is handy when you want to 'clone' a volume or use that for backups), and now you can also copy them to new regions, which is very powerful when you want to migrate a service or application to a new region for resilience, cost, etc. Since EBS volumes are on a shared storage platform, the performance is ok, but inconsistent. The reason is because most users use storage in a very bursty way, doing many reads and writes in a batch (that's how applications typically use storage). If those bursts are high enough, or there are many customers using the same set of storage disks, the performance becomes unpredictable since those bursts can easily overlap and max out the peak IO capacity or throughput capacity of the storage it is using. If you are trying to get a consistent amount of IO or throughput, you will quickly learn that EBS volumes are unpredictable. They typically perform quite well, and we recommend striping or RAIDing them together for increased performance (stripe, RAID 5, RAID 10, etc are all options but you should research and determine what works best for your needs). Of course, EBS is not free, so you end up paying for the persistence of that storage.
Persistent: [#####] Stable: [### ] Performant: [## ] Predictable: [# ] Cost effective: [#### ]
Provisioned IOPS EBS volumes are somewhat new to AWS, and are Amazon's answer to EBS performance capacity and predictability. Provisioned IOPS EBS (I'll just call them PIOPS EBS from now on) have the same qualities of EBS volumes in that they can be snapshotted, copied, and are persistent over time. You can attach them to any system as well, just like EBS. The main difference here is simple, but very important: you can provision a specified amount of IO (operations per second, not throughput) for a given volume, and you will get it. What this means is that you can now get great performance, that is EXTREMELY predictable, with the other benefits of EBS. Of course, there is a price to pay for this – however it is relatively small.
PIOPS EBS effectiveness:
Persistent: [#####] Stable: [### ] Performant: [#### ] Predictable: [#####] Cost effective: [### ]
As an example, a 100GB EBS volume would cost about $10/mo for the storage, and lets pretend we use 200 IO per second for the whole month (that is about what you will get from it on average). That's about 520 Million IO's for a month, so that would be an additional charge (EBS charges on volume and IO usage) of about $52 for a month. The total is roughly $62 for traditional EBS. Now, what about PIOPS EBS you say? Let's find out.
Provisioned IOPS are a bit more expensive for the capacity (25% actually), so 100GB of PIOPS EBS will run you $12.5 for the storage, and then we need to add the IOPS. Since you don't pay-per-use on PIOPS but instead pay-per-requested-use, it works like this: you pay $0.10 per provisioned IOPS-month. So, for 200 IO/s (using the same example), you would pay 200 * $0.10, or $20 additional, so $32.50. Wait – isn't PIOPS more expensive? Well, in some cases it is not, like the example here. You can test your own scenarios with the Amazon Simple Monthly Calculator (http://calculator.s3.amazonaws.com/calc5.html). Keep in mind though that when you provision 200 IOPS for a volume, you pay for those whether you use them or not, and you cannot burst over that. You will get 200, almost EXACTLY 200, IOs per second. We tested this many times, with different settings, and watched using our own metrics using CopperEgg monitoring, and we also used some other tools like iostat, and we can say that the storage is extremely predictable and the performance is precisely what you request.
Some downsides to PIOPS EBS (that I am certain Amazon will fix soon) is there is a max amount of IOPS you can provision per volume, which is currently 2000, and you cannot change this once the volume is provisioned. This means that on some linux systems which only can have 11 volumes attached (I understand this may be changing in the near future – yay!) you can get a max IO capacity of 22,000 IOPS for a single system. If you need more than that and can ignore persistence, I highly recommend the hi1.4xlarge systems.
Provisioned IOPS EBS volumes make wonderful database storage volumes due to the extreme predictability and high performance (about 10x per volume compared to standard EBS). Keep in mind though that you are still limited by throughput (the 'bytes per second' you can get) to those volumes, however if you need to eek out some more storage bandwidth (keep in mind that storage bandwidth and network bandwidth travel on the same 1gbit network on standard instance types – unless you use a 10gbit system), then you can choose an EBS optimized instance, which uses a dedicated network interface for storage so your network and storage traffic to not fight over a single pipe.
We chose to use PIOPS EBS in conjunction with hi1.4xlarge systems (and others) to provide the right requirements for our data stores, since we really want dependable and scalable storage performance, and we've been quite happy with those aspects. Highly recommend checking it out.
For other best practices when deploying on AWS, check out our Whitepaper.