My first programming job after college was for a garment maker in Slough, in the United Kingdom. We were a small team, and everyone had to do everything. My programming by day tasks were complemented by being “on call” one night per week and one weekend day per month.
Arriving at the data center in the middle of the night, the first words I said to the operations team were always the same, “What changed?” I had learned, just as Newton had predicted, that software continued in its “uniform state of motion” unless acted upon by some external force. That is, change.
Primitive though those days were, we kept detailed logs of changes made to the system’s hardware and software, and to the business software and data sources we developed. Invariably the cause of the outage was some change, causing an unexpected impact on the system.
Today we have comprehensive change control systems that audit and track us as we move fast and improve things. Nonetheless, bugs do still escape into the wild and must be hunted down and squished.
Software test automation provides us with many diagnostic tools that enable us to pinpoint where the issue lies. Here are my top 10 indispensable ones.
1: Steps to recreate
This will always be the number one request from a developer. What do I need to do to reliably reproduce the error so that I can see it happening. And so, I can turn on my debugger and watch “under the covers” the lines of code being executed, and the data vales being processed. In an automated testing environment this is your test script and test data. With this, the developer can see for themselves exactly what is happening.
2: Logs
Some bugs are intermittent and depend on external factors that mean the occur sometimes, and sometimes perform entirely as expected. The logs created by the system tell more than just the story of your application’s execution, they tell the whole story of the universe within which your application was running. Everything from the operating system to the database, to the users logged in and all the other software swirling around monitoring, tracking, and touching yours. Logs are great for telling what happened, but they don’t say how it happened.
3: Screen shots and page shots
A picture paints a thousand words, they say. Sometimes the reason for the issue occurring is simply bad data. Looking at what the user entered or what the system generated can often be the quickest way to diagnose a problem in your application. Whether you’re manually testing, or running an automated test, take screen shots (or videos) regularly and often. Sometimes a seemingly insignificant piece of data can have profound consequences such as a date in the wrong format – 12/05/2023 is very different to 05/12/2023 depending on where you live.
4: Instant replay
One very cool feature is HAR files. HAR, or Http ARchive format files contain a record of how an application transacts with the user at the bit-by-bit level. When you replay the HAR file you see exactly how the application looked to the end user. What’s even more interesting is that the playback does use the application so you can watch a HAR file from last week, see how the application behaved, then watch the one from today and determine the differences. Indeed, a simple file comparison tool will enable you to quickly identify the differences.
5: Spot the difference
Remember “What changed?” Spotting the difference and understanding how that may have caused the error you are experiencing is a key weapon in the search for a bug’s hiding place. Having built-in differencing engines in your test automation tools is critical for the bug hunter. Differences in the code should come from your repository management system. Differences in the user interface, even a pixel off, can cause testing tools to stop working. Differences in the data, or data format, will have profound effects. Using production data is always a mistake (and a topic for another blog) because is changes constantly, contains PII and PHI, and does not contain every combination of data that your system is designed to process.
OK. Halfway there. These top 5 are must haves for you to provide to the dev team. They are the minimum buy-in that enables dev to remediate the code and eliminate the bug quickly, accurately, and reliably.
6: API calls
Just like the hidden mass of the iceberg that fated night in the North Atlantic sealed the fate of the Titanic, the vast submerged network of services and APIs are just as menacing and mysterious in the world of software. A minor change to a service call payload may have only minor effects, but those effects, accumulated over millions of calls per day per day, can (literally) add up a system crashing impact. Being able to compare which calls ran, which did not, which were new, and which changed, will help the sleuthing developer narrow their focus to potential culprits. Sure, the repository lists what code was changed, but you need to know what changed at execution time. And don’t forget, many systems today rely on galaxies of third-party services whose provenance and reliability are outside your control. In 2011, I was awarded a Patent (7,934,221) on the solution to this problem.
7: Go with the flow
A much harder problem to solve is when the application does not do what it is supposed to do, does not go where it is supposed to go, and does not show what it is supposed to show. Seeing what there isn’t is much harder than seeing what there is and what is wrong. Seeing that the amount on your check is wrong is attention getting, seeing that the check is undated, is easy to overlook. One way to do this is to trace the user flows. To be able to see exactly the pathways the interactions and data entered caused to be followed, and to be able to contract and compare them against previous paths followed, will enable you to determine the point of divergence – the point at which something changed.
8: Time is an illusion
An often overlooked, or at least a late to be considered, is the impact of the application’s performance on the behavior. We have all heard of deadlock, and race conditions, and other seemingly esoteric down-in-the-engine-room system level issues, but these can directly impact the proper performance of the application. So thorough performance and load testing is essential ahead of release of the software. Let’s not forget that performance is impacted by many external factors and so constant monitoring, real-time, does provide insight if not point directly to internal or external forces that could be contributing to the issues observed.
9: Don’t get hacked off
When was the last time you had your application security tested? Is it possible that there is some vulnerability being exploited accidentally or maliciously that is causing these unplanned behaviors to occur? Regular App Pen testing is essential especially today to ensure the code you are delivering cannot be exploited. Run a Pen Test and see what it reveals. You have nothing to lose … except your customers, their money, your reputation, and, maybe, your liberty.
10: Code review
As I mentioned earlier, my first programming job was for a garment maker. The IT office was between where machines cut and sewed the garments and the design office where the garments were imagined. The IT office had been a storeroom and there were a few mannequins still in there. It was required, if you could not debug an issue, to have a code walkthrough with a colleague to get an extra set of eyes on the problem. If no colleague was available, we took to walking through our code with a mannequin. I cannot tell you how many times just describing what the code did to an inanimate, anonymous, plastic proto-programmer resulted in discovery of source of the problem. It is a practice I continue today but with my Golden Retriever, Bobby.
Conclusion
Having all these diagnostics in your debugging tool bag helps but they are no substitute for domain knowledge, experience, and knowledge of the business rules of your application. If you would like to learn more about how you can combine AI technology with your domain knowledge to help you make better applications and speed digital transformation, give me a call.
To learn more about Appvance’s web application testing capabilities, schedule a call with the Appvance sales team.
