KevBurnsJr

Many decisions are made at the outset of a brand new product/company. Recently I participated in an all-hands meeting to kick off development of a company’s first product (CEO guy, Product guy, Engineering guy [me]). We talked about the product and the architeture and came around to the infrastructure. It was decided that we would use AWS for its handy scalability.

During the conversation, the product guy (who has some tech chops) said something that caught me off guard. His quip was something like: “Cloud is the way to go. We’re definitely going to do everything in the cloud.”

Below is my lengthy response.

Hi Chris,

You mentioned something in that first meeting which I’ve been wanting to elaborate on and partially refute.

I’d like to caution that The Cloud is not a silver bullet.
While AWS may be an excellent solution for the immediate future, there may come a point where moving to a datacenter proves to be more reliable and cost-effective.

There are a number of properties of the infrastructure involved in cloud computing which guarantee that some cloud resources will never match the performance and reliability of bare metal.

See for instance the recent Reddit outage in which AWS EBS volumes became unavailable for several hours, resulting in serious data problems.
http://blog.reddit.com/2011/03/why-reddit-was-down-for-6-of-last-24.html

In AWS, the hard drives (EBS) are connected to the machines (EC2) over a shared 1Gb/s ethernet connection.
In a real server, the hard drives are connected to the machine over a dedicated 6Gb/s SATA cable.

There are some ways to help mitigate these bottlenecks
http://victortrac.com/EC2_Ephemeral_Disks_vs_EBS_Volumes

1) you can set up multiple EBS volumes in a RAID configuration to increase throughput
2) you can load balance traffic across multiple availability zones to help mitigate network congestion and availability issues like those experienced by Reddit

But also keep in mind that there will come a point at which renting a cabinet (42U) in a datacenter will become far more cost effective than running a server farm in the cloud.
A full cabinet with 15Amps of power and a 100Mb/s connection runs $500/mo with a signed 3 year lease.
A quarter cabinet (7U) with 2Amps of power and a 100Mb/s connection runs $200/mo with a signed 3 year lease.

So if we took one of these $200 PE 1850s I have sitting here and plugged it in to a quarter cabinet at Hurricane Electric, we would already have more processing power and it would save us $40/mo over our current us-west m1.large instance.

But there are 2 advantages to the cloud:
1) If we need more capacity, we can provision it immediately without having to provision hardeware
2) If we need capacity in a different location, we can easily spin up a cluster in Japan or Europe or the East Coast without having to physically be there.

In the long term I hope we’ll come to a hybrid approach where we can
- leverage the best of real-world datacenters (for backups and heavy number crunching)
- and leverage the cloud for all its advantages (for geo dispersion, fault tolerance and early stage ramp-up)

Remember the words of Devops Borat
http://twitter.com/DEVOPS_BORAT/status/40816140632596480

“Formation of hybrid cloud: 1) roll in datacenter 2) SHIT! roll to cloud 3) SHIT! roll half back to datacenter.”

- Kev

Thanks, Chris for suggesting I post this to the blog.