Get Started

The AI Testing Hype Machine


The AI Testing Hype Machine

In the past year we have seen a handful of startups get funded claiming they use AI for testing. But when testers actually got their hands on the tool, they found nothing more than a first generation scriptless recorder. Some had helpers and some a new language…but in the end, humans had to write or record scripts. No different than we have all been doing for 30 years. This is not where we all want AI to take us. And it is confusing the market. A point I have heard from a dozen prospects lately.

The term AI in itself is arguably a marketing term which is all the rage. Nevertheless, in a real “AI” system, we would use any of 100+ machine learning techniques to model parts of what human testers do today. Just placing some AI in a test system isn’t the point. The goal of AI driven software testing is to replace some mundane human tasks (or augment them to the point of great productivity gains). The truth is, we are not seeing this from other vendors yet. And it may be years before we do because it is exceptionally difficult as we have learned.

While Appvance announced the first AI driven test generation in 2017, lets ignore AI for the moment and go back in time a bit. Appvance introduced the first modern scriptless testing in 2012. That was our first generation, but based on a new JS injection which was browser independent. We later rewrote that to gen 2 and again to a 3rdgeneration in 2017 based on what we had learned from the millions of tests completed on the earlier two generations. Customers speak. We listened. It is a far better scriptless system today, and likely more advanced than anything else we have seen.

Our 3rd generation Test Designer is compatible with all modern UI libraries (Angular2, ReactJS, KendoUI etc), and has access to the UI, DOM and API -full stack- at the same time. That means UX and API scripts are created at one time. Scripts are data driven with simple drag/drop, and there is a complete language behind it, plus representing the script in Javascript. Meaning a senior QA or developer can edit it fully with some JS knowledge, but manual testers can capture their work for repeating instantly with little training. In side by side timing comparisons, we found testers can create use cases about 10X faster than the same use case in Selenium. All without the need to know any code. We felt 10X was a breakthrough. But it’s not good enough.

We added our first machine learning “AI” in 2016 with self-healing scripts. This technique would learn from all possible accessors for each element the first time it created a script, and later automatically rewrite the script with more stable ones if chosen accessors failed in subsequent runs against new builds. This reduces maintenance and flakiness as it’s self-maintaining.

None of the above eliminated any QA jobs, but augmented their work and made people far more productive.

AI for the sake of AI is not helpful to our industry. AI which drives bottom line results is what we all need. Surely some startups would like to deliver true orders-of-magnitude gains in QA productivity. However, we found to do so was a 100 man-year effort, at least for us. Maybe others will get there faster. Who knows. But it isn’t a weekend project for 3 smart coders. It’s serious math, science, modeling and coding. And it’s not perfect.

My personal recommendation is “don’t believe the AI hype.” AI or not, look for real gains in productivity (for example orders-of-magnitude reduction in tasks like scripting). It is quality and speed and cost improvement you are looking for. Not AI alone. Understand where your developers and QA people spend their time today. Manual testing, scripting and maintaining scripts often accounts for 85% of activities. Every enterprise needs continuous testing today as part of their CI/CD pipeline. A true AI driven test generation system can offer a huge boost to productivity, speed, and outcomes. Certainly one or two orders of magnitude improvement in key areas. It isn’t a simple tool one downloads and tries. It’s a serious system with serious implications.

While small companies and startups may try anything new, even a new recorder labeled “AI”, high quality enterprises don’t change easily or quickly and are not easily fooled with marketing hype. They need to see real impact across hundreds of applications. And true enterprise level support to roll out AI generated tests over time, with care.

And last, there are defined levels of autonomy in many fields today, including QA, which I will cover in upcoming blog posts along with the past year of learnings from the first AI generated tests at Appvance clients. It is a fascinating journey I know you’ll enjoy and we can all learn from.

We are both humbled and excited by this journey that along with our client partners is changing the software development lifecycle forever.

Kevin Surace is CEO of, the leader in AI driven software testing. He has been featured by Businessweek, Time, Fortune, Forbes, CNN, ABC, MSNBC, FOX News, and has keynoted hundreds of events, from INC5000 to TED to the US Congress. He was INC Magazines’ Entrepreneur of the Year, a CNBC top Innovator of the Decade, World Economic Forum Tech Pioneer, Chair of Silicon Valley Forum, Planet Forward Innovator of the Year nominee, featured for 5 years on TechTV’s Silicon Spin, and inducted into RIT’s Innovation Hall of Fame. Mr. Surace led pioneering work on the first cellular data smartphone (AirCommunicator) and the first human-like AI virtual assistant (General Magic), and has been awarded 82 worldwide patents.