What do Obamacare and HBO GO have in common?
So many times I’ve seen companies fail to test the scalability of their applications or websites because it is “too hard” to simulate that much activity. The most famous, current, example of a failure to test scalability has been healthcare.gov, also known as “obamacare” and the “Affordable Care Act”. For months we’ve all heard about the problems trying to connect and not being able to. What does the government and HBO have in common? When Game of Thrones premiered season 4 on April 6th there were so many people that tried to use the streaming HBO GO service that it crashed, leaving many paid subscribers unhappy, and I imagine also leading to a spike in pirated downloads of the episode. As an aside, Game of Thrones holds the dubious honor of being the most pirated show in history.
So how could have HBO avoided the problems they had on opening night? By making use of the testing software available from SOASTA. They are the leader in cloud based stress testing. Maybe stress testing limits the scope of what they do, but that’s how I like to think of it. In the old days of the web, we worried about being “slashdotted”. This was when a story would run on slashdot.org about your company and everyone would click through to your site. The first time that happened to me, my server was overwhelmed, this was probably in 2001. We really had no way to know what the impact would be, we had to guess, fortunately we guessed right and the next time it happened a year or so later, we were able to handle about 90,000 visits in a day without dying, however it would have been nice to know ahead of time if we would be ok, me and my team were sitting around watching the server and hoping we might be able to tweak something in real time if it became a problem, we just didn’t know which resource was going to blow the server up.
Now you’ve got millions or tens of millions of people that could hit your site, they might be streaming, or filling in forms, or shopping, we all know about all the big sites, but there is really no way to guess if what you’ve done will scale once you start getting bombarded by hits. SOASTA takes care of that for you, and they go beyond testing your website, they also can script and stress test your mobile applications. They’ve got a really cool user interface paradigm that is oriented around the concept of a DAW (Digital Audio Workstation, or a virtual mixer). I absolutely love how this works, you create test scenarios, say like filling in a form, and you apply it to a fader. Another fader might be filling a shopping cart, another might be browsing, etc. As you bring up and down the sliders, it increases the load, so you could go from 10 users to 500 users over the course of 30 seconds and then hold it for 2 minutes and bring it down, or lock the faders and increase them all at the same time. These two product are known as Cloudtest and Touchtest.
With that long pre-amble, I was able to chat with President and co-founder Tom Lounibos to get some answers to some specific questions:
Shawn Gordon: Thanks for carving out some time for me Tom, I hadn’t even been aware of your company till a few days ago, and now I’m ashamed I didn’t, because I should have. After spending some time going through your information and demos, I’m just blown away, this tech is so necessary for so many web sites and applications. Can you tell me about about the heart of it and we’ll branch out from there?
Tom Lounibos: Thank you Shawn, we’re very proud of what we’ve built here. There are really two fundamental aspects to what we do, one is of course how we make use of cloud computing to really exercise your site, the other is the rich analytics that allow you to do something useful with the information you discover.
SG: Can you tell me more about the analytics? I assume you’re not using Google or something like Piwik.
TL: Correct, we’ve built an incredible set of analytics tools that is proprietary as well as patented. With our predictive analytics, we can tell you where the bottlnecks are at and various methodologies to resolve them, for example: We have the largest test system ever built, we can simulate up to 1 Billion users hitting your site. With that traffic, we can tell you something like if you cut that video by 2 seconds, it will allow enough bandwidth for another 50,000 users for example. Those analytics give you that granular view that allow you to make decisions about how to optimize your users experience.
SG: I love the DAW paradigm on the Dynamic Ramp tool (http://bcove.me/f4mhe6jr), I already covered how it works, but can you give me an interesting scenario that people might not intuitively realize?
TL: Sure thing. During a big test, you could have something like 100 people watching it run. There might be people from sales that knows they have a huge market in California, so they can have them ramp up the hits from California dynamically instead of stopping and rewriting the tests. So say you are testing 100,000 globally dispersed connections but you know 80% of your traffic is California, then we’d move the connections so now 80,000 are coming from California. All quickly, easily and in real time. In the old days it took 6 weeks to write tests to simulate 4,000 users, now it takes about an hour to set it up and be able to do what I just described.
SG: The scale of this is just mind boggling, the number of servers and users and bandwidth you’re dealing with. Can you give me some specific numbers for our readers?
TL: As mentioned, we can generate up to a Billion connections to your site, this requires a lot of servers, so we have partnerships with virtually every Cloud Computing company, what we do wouldn’t be possible without them. So the way the numbers shake out is that we need 1 server per 100 users. If you are trying to test 100,000 users, then we need 1,000 servers. We have access to over 1 Million servers around the world. The ad-hoc client cost (non-subscription model) will run from $.10 to $.25 per hour, per server. On any given day, we will be provisioning 10,000 to 15,000 servers to accommodate our customer needs. One of our other patented technologies is a cross cloud deployment grid API. This gives you a unified API in to all those cloud services so you just deal with one interface layer to be able to work with all of them. This greatly increased our productivity and is unique in the world.
SG: You mentioned pricing, can you dive in to that a little bit more.
TL: We have an ad-hoc, pay by the hour model and a subscription model. I mentioned some of the pricing earlier on the ad-hoc model, it provides a very cost effective way for a company to test their site before launch or perhaps before a big event. We have some freemium options as well as our mPulse application, which is a fun way to see what the global disbursement of people on your site.
SG: Let’s touch on “TouchTest” to wrap things up. So this is for testing mobile applications. Functionally, what are you doing here? Are you recording taps and swipes and playing them back? How are you stressing it specifically?
TL: That’s pretty much exactly what we’re doing. We have a proprietary library that we embed in the object repository, then you just use the app as normal, it will record all your actions, including delays in using it, like you got a phone call or something, everything to reproduce actual usage in the application. The beauty of our model is that it doesn’t require you to jailbreak the device as it is recording from within the application. You can then play back the script to ‘n’ number of devices to test your scalability. This doesn’t really apply on a stand alone app, but say something like “Words With Friends” where you have multi user applications that are talking to or through a server, it allows you to test specifically without just guessing if you’ve done a good job.
SG: Thanks for your time Tom, I might be as excited as you are with what you are doing, I hope that HBO got themselves a license after their Game of Thrones fiasco
TL: My pleasure Shawn, it was great getting a chance to discuss our products and solutions with you.
Some of SOASTAs clients include Intuit, Microsoft, Netflix, Hallmark, American Girl and many other big companies. They’ve just announced local test labs for Apple iOS and Google Android smart phones and tablets with their TouchTest Private Device Cloud. With this product/service, mobile development and testing teams can now remotely manage devices, Apps and Tester activity. There are some companies out there that absolutely should not be without SOASTA.