Get Started

One Year of AI Driven Testing – What We Learned


One Year of (real) AI Driven Testing – What We Learned

Appvance introduced the first level 4 autonomy for software testing in September 2017. By now thousands of test runs have been accomplished covering millions of test steps, all fully created by AI. No recording or scripting, just machine learning. It’s not magic…but it seems like it is. As one always learns from real customers we have as well. Tons. Today we share a few of the important take-aways from real customers generating real tests automatically…no humans involved.

  • Setting up AI requires expertise

Our goal was to create a system that replaces the mundane tasks of writing and maintaining scripts. Not with a recorder, but with AI script generation, truly automatic. And automatically finding major and minor bugs. In doing so we realized that speed is king in continuous testing and to get speed we need some human-led setup the first time. What is often called “hinting” in AI. This appears to be in line with other “real not fake” AI applications in other fields. This requires expert setup to get all the gains. So, we have adopted a model where our pro-services team sets up the AI engine for each AUT, at least to start. It’s included in our enterprise offering. And that turned out to be a positive selling point, not a negative. People want results. Our experts setting up the system relieves clients of any concern about attaining results, and new tasks and training, and lets them reap the benefits of AI driven testing faster. We have found that after the first few applications, developers and high-level QA engineers are able to be trained and set it up themselves.

  • Every application is different

We knew that of course. But it is really true. We have had to add many more setup choices to deal with the variety of applications, libraries, odd paths and more. From ReactJS to ShadowDOM architectures, the AI system must be able to learn from any library or flows and create real tests.

  • Results in an hour really count

Today, 25% of organizations are achieving at least one release a day on some applications. Those must be tested, with results generated, in an hour or so to give time to developers to address new issues. Better testing that takes a day is useless. An hour became our standard target, and that’s what we now deliver.

  • Smoke tests have real value

Our instant smoke tests, which are delivered during AI blue-printing (that’s when we machine-learn the AUT), provide real value, finding dozens of bugs quickly. All data-driven of course. While AI has created artificial pathways through the AUT, we tend to uncover serious app issues quickly that can be resolved even before regression tests are generated (automatically of course).

  • Regression tests are well understood

People get regression tests. Not always smoke tests as a term. True regression would mean “attempt to do in this QA build what real users have been doing in production.” Now think about that statement. Because in the past we never knew what users actually did, we guessed. So, our regression tests were by definition a guess. Until Appvance IQ. Our AI system generates regression tests based on knowledge of real production user usage. With nothing added to the production system whatsoever. No keystroke recorders or anything else that SecOps would say no to. We use just what they already have at the server level. With nearly 100% user flow coverage capability. And the issues found go well beyond what legacy scripts were finding. This powerful technology, which maps production user flows at the server to similar flows at the UX in the new build is a breakthrough in QA productivity and accuracy.

  • People ultimately want Level 5 autonomy

Level 5 autonomy includes validating actions which even humans cannot easily do today. It’s the holy grail of autonomy. And once achieved can mimic basically every test a human can manually do, or script. And then some. Achieving level 5 is the testers dream. Just as it is for driverless vehicles.

  • Choose clients wisely, because we will be partners for many years

We do not have a freemium model or any downloadable “tool.” For true AI, it just doesn’t work. This is not a “tool” you toss over a wall. It’s a complex system that needs to work closely with your team and the AUT to be a QA game changer. The new client requests to use Appvance IQ consist of qualified clients with a real pain point who have a budget today to engage and timing identified. This seems to align well with other real AI technologies being delivered across many industries. If we are going to dedicate resources to help a QA team improve productivity by 90% or more, then we need everyone on deck together. And that doesn’t happen for free. We believe in string long lasting partnerships. Hand in hand for strategic success.

  • Real results

We have seen very large multibillion dollar clients be able to dramatically accelerate their release schedule with continuous AI driven testing. And reduce QA costs. Taking QA time from days or even as much as 3 weeks down to an hour or so. And in many cases, reduce reliance on outsourced services. It requires dedication from all parties, just like any DevOps process transformation.

At Appvance, we keep learning from our customers. Our AI techniques are a breakthrough for many applications and not yet right for others (such as Windows desktop apps for one). There is no panacea. Every enterprise needs continuous testing today as part of their CI/CD pipeline. A true AI driven test generation system can offer a huge boost to productivity, speed, and outcomes. We keep getting better and adding more algorithms to our already extensive code base. And expand our patent filings as we were the first firm to introduce real AI driven test generation. But it has been gratifying over the past 12 months to see so many teams identify bugs they never found before, increase their speed, reduce their costs, and even improve revenue in client facing applications. The good news: CIOs get excited by these results every time.

And we have more work to do!

If your team has the need to improve and a budget to move to real AI driven test generation, request a technical consultation now to see if your needs match what our industry leading AI technology can do for today.

Kevin Surace is CEO of, the leader in AI driven software testing. He has been featured by Businessweek, Time, Fortune, Forbes, CNN, ABC, MSNBC, FOX News, and has keynoted hundreds of events, from INC5000 to TED to the US Congress. He was INC Magazines’ Entrepreneur of the Year, a CNBC top Innovator of the Decade, World Economic Forum Tech Pioneer, Chair of Silicon Valley Forum, Planet Forward Innovator of the Year nominee, featured for 5 years on TechTV’s Silicon Spin, and inducted into RIT’s Innovation Hall of Fame. Mr. Surace led pioneering work on the first cellular data smartphone (AirCommunicator) and the first human-like AI virtual assistant (General Magic), and has been awarded 82 worldwide patents.