Thanks to Amazon Web Services, driven by Appvance IQ (AIQ), you can save 99% on your mobile load tests. Before explaining how, first some context.
Amazon has disrupted industry after industry and is now disrupting mobile device testing. AWS Device Farm, their mobile device cloud, uses immense scale to lower the cost of testing mobile apps by 90% compared to previous device clouds. Not surprisingly, AWS’s trademark combination of low price and broad coverage has quickly made it the go-to service for compatibility testing. That’s great, but load and performance testing is the truly expensive testing process in terms of device consumption, so is the domain in which AWS Device Farm offers the most eye-popping savings.
This is because the only practical way to test mobile apps for load and performance had been to use a specialized and often expensive vendor (Perfecto Mobile being the most well known). An hour-long mobile load test for 10,000 virtual users (VUs) could cost from $8,000 to $27,000 in device farm charges each time it was run. In comparison to a service going down, that is inexpensive. But in comparison to what we can now do with AWS, it is incredibly expensive, especially if you are doing this a few times a month.
AWS Device Farm offers thousands of devices representing 346 device/OS combinations. However, it’s not really set up or meant to perform a load test itself. But in conjunction with Appvance IQ’s multi-system load testing capabilities, mobile load testing costs just fell by 99%.
Here’s how. With Appvance IQ, a tester can create HAR, JMeter, SoapUI, Java or other API level scripts to generate most of the load against the servers. These scripts can even be automatically generated by Appvance IQ with its HAR capture utility and auto-correlation. This is run concurrently with actual devices on the AWS Device Farm for device-level timing as well as server response times. Let’s state that more clearly: Appvance IQ can combine HTTP load with real device timing and produce results equal to (but far less expensive) than using all devices or other legacy device clouds.
With Appvance IQ orchestrating AWS, test nodes are ramped on-demand at AWS and can be on the enterprise’s own account or on servers supplied by Appvance. At the same time, one or more actual device at AWS Device Farm can be added to the test scenario, providing real live device performance metrics under increasing load.
The AWS Device Farm costs only 17 cents/device-minute. A load test to 10,000 users might run one hour, as above. Depending on the complexity of the API scripts, AIQ requires approximately 10 large EC2 VMs to generate the needed VUs at the same time. If you are using your own AWS account, the cost of the on-demand servers is approximately $1.50/hr.
Let’s assume the use of ten different devices for a variety of data, plus 10,000 VUs together to run the Appvance IQ load-test with AWS. This 1-hour test would cost $117 (17 cents x 10 x 60 = $102 plus 10 servers at $1.50/hr x 10 servers = $15). That $117 compares to $8000-$27,000, or about a 99% savings.
Learn more about low cost mobile load testing at AIQ Load and Performance Testing.
