Tools that benchmark performance promise to reveal the strengths and weaknesses of competing cloud providers.
New software developed to measure the performance of different cloud computing platforms could make it easier for prospective users to figure out which of these increasingly popular services is right for them.
Right now, developers have little means of comparing cloud providers, which lease access to computing power based in vast and distant data centers. Until actually migrating their software to a cloud service, they can't know exactly how fast that service will perform calculations, retrieve data, or respond to sudden spikes in demand. But Duke University computer scientist Xiaowei Yang and her colleague Ang Li are trying to make the cloud market more like the car market, where, as Yang says, "you can compare specifications like engine size or top speed."
Working with Srikanth Kandula and Ming Zhang of Microsoft Research in Redmond, WA, Yang and Li have developed a suite of benchmarking tools that make it possible to compare the performance of different cloud platforms without moving applications between them. These tools use algorithms to measure the speed of computation, and shuttle data around to test the speed at which new copies of an application are created, the speed at which data can be stored and retrieved, the speed at which it can be shuttled between applications inside the same cloud, and the responsiveness of a cloud to network requests from distant places. The researchers used the software to test the services offered by six providers: Amazon, Microsoft, Google, GoGrid, RackSpace and CloudSites. Results of those tests were combined with the providers' pricing models to allow for quick comparisons.
The results are among the first attempts to compare the performance of several clouds platforms, says Yang. "We found that it's very hard to find a provider that is best in all metrics," she says. "Some are twice as fast for just 10 percent extra cost, which is a very good deal, but at the same time their storage service is actually very slow and has a lot of latency variation." Another provider showed good computation speeds but was less quick at spawning new instances of an application--something that might be necessary for a service that experiences peaks in demand, as a video site does when some of its content goes viral. "It seems like in today's market it is hard to pick a provider that is good at everything," says Yang.
Some say that recent high-profile cloud outages are just a matter of bad design.
A startup makes your computer's storage capacity seem bottomless by connecting it to the cloud.
A marketplace sells unused computing capacity—but a lack of transparency may put off some customers.
Future standards for cloud performance?
The research reported in this article should be of great interest to I.T. managers who have to compare cloud systems, estimate ROI, and justify decisions/proposals.
I expect there must be some proprietary methods for evaluating cloud performance that have been developed by the cloud vendors themselves. I wonder if this type of work will eventually lead to an industry standard for performance measurement tools for clouds?
Manufacturing in the United States is in trouble. That's bad news not just for the country's economy but for the future of innovation.
This document is part of the “How-To Guide for Most Common Measurements” centralized resource portal. This tutorial provides a detailed guide for measurement and device considerations to take temperature measurements using thermocouples. Get an introduction to thermocouples, which are inexpensive sensing devices widely used with PC-based data acquisition systems. Also review some specific thermocouple examples and learn how thermocouples work and ways to integrate them into a data acquisition measurement system.
View full PDF >
Listen to story > 
Our list of the 50 most innovative companies, including the following:
On Twitter
Become a Fan on Facebook
Subscribe to the Feed
mattgroom
290 Comments
Oh yes
I used to do metric testing of an entire world wide network, and found from time to time the carrier liked to change things...ie slow things up for us.
Only being a large consumer could we really control our network provider or perhaps it was me being on the ball and being able to tell them to the minute when they made changes i didnt like. (Yes i was that good or anal retentive..take your pick).
Anyway as you put it you need to check daily...which answers your question to which one is best... Metric testing is good (great) but unless your provider is responsive to your complaints/demands youre in a failing relationship already.
The average company will do what is good for that day, tomorrow is different...but by then they are locked in and metrics are useless.
I hope my insight has helped a someone somewhere...
Reply