For the better part of 20 years, the e-commerce QA test industry has known that every one-second delay in response, they can lose up to half the page audience. Not because the user bought somewhere else, but because they became distracted. Today’s distractions are probably much higher than they were when those original studies were done 20 years ago. Even on your computer, you can easily get distracted waiting for the screen to fully load, when you get an important email, and you forget about what you were going to buy.
This is also true on mobile. In fact, it’s an even bigger problem on mobile because you’re always on a mobile network. Often you’re not on WiFi. Mobile networks always add delays, so it’s important not to add to those delays (with slow performance). There are tricks that can be done in mobile in terms of caching, where people are learning to deliver first responses in under a second. Which means that even though your mobile device is waiting for more information through the mobile network, you’re not seeing that as a user. You’re getting responses, things are happening, and it’s all good.
Test it the way humans do
When you look at overall performance, you want to look at it just the way humans do.
For testing purposes, there is a difference between performance and load. Load testing is where you just ran a Super Bowl ad and know 10 million people are going to come to your mobile app or website. You’d better make sure it doesn’t fail under the load of 10 million users.
Performance is the normal number of users that are on the application, and the average response time they experience. The drivers of performance are what’s happening at the server, what’s happening in the mobile network, and then on the client-side code. This code resides on the mobile phone itself. Today it has a lot to do with performance. You’re now potentially able to cache user content in a content delivery network (CDN), or on the phone itself. You can preload it. You know that every time a user comes on your site, they look at a particular pair of glasses. You can just cache them so next time they visit, it’s right there. Humans want instantaneous response. You do not want someone waiting for the mobile network and for the server to respond.
Important functions to test
To test well, you must follow a chosen set of tasks or a set of paths that the user follows. Maybe they log in and search for a product if it’s e-commerce, then add it to their cart. Maybe they then take it out of the cart. Eventually, they go to the cart to check out. They certainly don’t want to wait that long to check out. Amazon is famously good at checkout. They just accept the fact that your card is probably good and check you out. Later they tell you if that card was declined. So in parallel to closing out the cart, selling the product, they inform you, “Oh, we had a problem with this card.” What they don’t want you to do is abandon the cart because the card failed. They want to get you through that sale process and then resolve the payment issue. Amazon is smart for doing this, others are not.
All of those steps that the user takes are important to that user. It’s very hard to step back, and say, “Well, the most important is” because it varies… If a user is searching for some odd pair of glasses, they need to come up in under a second. And if that search function doesn’t come back in under a second, they start to get bored and start to think of other things. Within three or four seconds, they start to think about whether they want to spend the money on those glasses. It’s that fast. You have to respond. For some users, it’s going to be the search function. Other users don’t search at all. Maybe it’s getting through the purchase for that first product that comes up. For other users, it’s going to be something else.
If you’re supporting internal users on an internal web application or mobile app that runs some internal functions, then you have to consider productivity. Companies notoriously ding their employees and make them work on apps that are horrifically slow. They rationalize that it’s just their employees. But add up how many minutes a day per employee. For instance, five extra minutes a day times 100 thousand employees. Then add up the lost productivity across a year. It’s $100 million. Better to make the app faster. That would be much cheaper. We all understand e-commerce, but we have to understand that these same rules apply to everything including Salesforce, ServiceNow, to whatever you’re using internally as your ERP system.
How do you realize the need for speed?
Unlike web apps, mobile apps don’t easily provide code visibility where you can potentially see how something came back in a particular way. The best way to test is to use AI to recognize the actual images on the screen and recognize when the image that the user is interested in is loaded. For example, you want to know when that sneaker materializes and fully draws itself on the screen, which, by the way, can happen in 50 to 100 milliseconds or so. When it’s done rendering on the screen, the user considers that they can do something with that sneaker. They can buy it, look at it, or cycle through the pictures. This is interesting because for the first time, you’re able to look at a mobile screen the way a user does, literally through their eyes. Appvance has been doing this for several years in its mobile performance area. Now it uses AI to look at the image, timing it to the way a user’s eyes see it, not when the data came back from the server. The user only cares about when this sneaker shows up on the screen, and they can buy them.
What’s more, Appvance’s mobile suite provides these metrics competitively for different companies where they test out their own app and they have Appvance test their competitors. They end up with a competitive analysis so that they can see how they’re performing step by step by step. Let’s search for something, let’s select, let’s put it in the cart, let’s try to buy it, let’s get it out of the cart, let’s go do something else. How do we compare? In step by step, doing it with eyes the way you do it, with AI, you can see five-, six-, or seven-seconds difference between one app and another. In that seven seconds, you’ve probably lost the client.
Appvance does it across 500 runs so you get a bell curve of performance because every time that image comes on the screen, it might be slightly different timing. There are outliers, some long, some short. But right in the middle, it was 2.3 seconds to bring that up. Your competitor was 0.8 seconds. Well, they certainly pre-cached it on the phone.
If you pre-cached it, you might have that image in no time. The user’s request didn’t have to make a round trip to the server. It was just ready to render, sitting right there in memory. You can’t do it in every step, but there are lots of tricks to use to go fast. Plenty of technology exists today to have your servers and your databases respond incredibly quickly if you care, if you want to spend the money. Incredibly, Appvance has clients who think 4.5 seconds to serve an image is pretty good, but they’re mistaken. “Pretty good” is one second. They don’t believe they can do better than 4.5 seconds. But they can.
Appvance has worked years to get its mobile performance suite to where it shows customers how they’re actually performing. It’s new territory. Appvance gives you your actual data against your competitor’s, and you can actually see the differences. The mobile performance suite has a redacted dashboard that shows bell curves of performance characteristics of each important step as a user would see it with their own eyes, and shows it globally, whether in China, the UK, or the US. You can see how you’re doing in China versus here and how you’re doing over here versus there.
This is cool, really fascinating technology, and it’s a game changer for anyone in mobile apps. Everybody needs to use it, but most people don’t know Appvance has it. It’s the best kept secret in mobile performance history, but shouldn’t be. Let Appvance help you benchmark your mobile apps against your competitors. You can do it at every build if that’s what you want. It’s statistically relevant, and Appvance does it the only practical way, the only right way, which is visually using AI, analyzing the visual information on the phone’s actual screen.
To learn more about Appvance’s mobile testing capabilities, schedule a call with the Appvance sales team.
