I came across this old essay of mine written in 2009, and thought it interesting to read, with the benefit of 4 years of hindsight. Perhaps it will spark some discussion, or at least provoke some thoughts. Here goes.

Cloud Computing is a sysadmin industrial revolution. The PC introduced a common OS API that developers wrote to. No longer did they need to write applications specific to the hardware configuration of a machine. Writing applications became easy, and applications became portable. You then saw a proliferation of applications, and a new market be created. Larry Ellison did the same with Oracle. Relational databases had no standard, so Ellison decided to port Oracle to every mini architecture out there. This way he “presented” an API before the market had “decided” and then the market decided based on that. These created the PC software and the relational database markets from which Microsoft and Oracle respectively made a killing.

Then came the Internet, which presented a common API for communications: HTTP over TCP/IP. Once again, a new market was created, this time for communication, which led to wikipedia, twitter, facebook, and google so far. In the above examples, you have the following: first there is a labor-intensive situation, tedious and repetitive as it were (porting applications to specific hardware, sending out snail mail to all your leads / customers), then a new paradigm comes along and industrializes the process, creating opportunities.

Cloud Computing is an industrial revolution for systems administration.

Here’s the key historical example: until the 17th century, about 80% of the workforce labored in farms, and saw over their lifetime no dramatic change in productivity. Then the industrial revolution came along and mechanization started in the very large aristocrat-owned farms. As machine prices fell, productivity gains trickled down to the masses. Today, only 2% of the population works in farms. Similarly, the bulk of systems administration is boring, tedious, error-prone, repetitive, and above all, labor intensive – like farming of yore. Gartner estimated at 80% the amount of resources spent on keeping existing systems running. The remaining 20% being spent on new systems to help and expand business. This is 80% of a multiple trillion dollar industry, growing at 10% yearly!

Large IT organizations like Google and Yahoo! have fairly automated systems, but the technology hasn’t trickled down yet. Cloud Computing is the mechanization paradigm that will allow smaller users to benefit. Amazon CloudFront is one such example: before, you required very large volume to justify a content distribution network. Now the smallest IT shop can afford it. The barriers to entry have been shattered. I’m sure you see the pattern here: IaaS is industrializing procurement of IT resources, and once they get big enough, they will be able to mechanize – that is, get robots to replace failed hard drives, install and network blade servers, etc. This is not doable for small datacenters, just like it isn’t possible for small farmers to invest in huge tractors and processing equipment. Only large operations can afford it. EC2 and IaaS control and industrialize everything from the hypervisor down. Procurement of resources, resource consumption monitoring, provisioning of virtual machines and compute capacity, etc.

The Cloud management layer industrializes everything above the hypervisor. There you find automation in the form of configuring databases (replication, scaling, backups, administration), configuring applications (libraries, deployments, profiling), and of course application scaling (as opposed to resource scaling: you can add more boxes, but your application won’t necessarily take advantage of them).

That’s what Scalr is all about. Industrializing systems administration, using Cloud Computing as the key enabler.

In our experience, the elasticity accounts for less than 1% of total IaaS expenditure. Spikes as a proportion of resources consumed is minute. However it is an excellent use case and marketing argument. But the reality is that elasticity is just one of the many possibilities of systems administration automation, one that requires automating the provisioning of a new server, installing the right software to turn it into a component of the overall architecture, and configure it to work with the rest. So in response to Ping Li’s post – yes, big data will be huge in the enterprise, and in my opinion the value in the stack will eventually move there. Akin to the value in agriculture now being in food processing, not production. However the industry needs to first undergo mechanization.

Thoughts?

Open Source thrives on strong communities. So do all open collaborative projects. Wikipedia is a prime example of this. Where would Wikipedia be without its legions of editors?

We want Scalr to be as successful as Wikipedia, and become the ultimate tool to manage resources for applications. So we need to create a vibrant community.

First step is obvious

Encouraging the first step.

Engaging by asking opinion.

Guidelines for discussions – don’t send link to a previous conversation. Provide a summary, then link.

Thanks the champions. Frédéric Sidler (http://www.mixin.com/users/fredericsidler/)!

For more information on creating communities, s

More on producing open source software

More on building an open source community

https://docs.google.com/fileview?id=F.c09865e1-314b-4b69-9758-a92a3652ddff