How Appvance Used AI to Successfully Deliver Level 4 Autonomy
When Appvance began working on true testing autonomy we knew there were many directions one could take with machine learning to attack the problem. One direction would use neural nets to scan thousands of applications and build a database of how they work and are built. And there may be value in doing this. But we did not find that it would add much to our knowledge. Domain experts know how actions work in modern libraries just as testers do. While this is interesting work, it seems more interesting to an AI researcher rather to achieving true QA autonomous testing. And indeed, today many excellent research papers have been published which start with this technique.
One of the challenges of machine learning is noisy data, or data that changes frequently. When we attempt to learn from thousands of applications, we find that each is built differently, with different libraries and different developers. The similarities often end past login. Again, from a pure research perspective, totally cool. And recognizing objects with AI is obvious. But we don’t believe it moves others closer to true autonomy, since domain experts already know this information and could hint an AI system to use it. But it’s very cool work to build a neural net model that recognizes objects in mobile apps or web pages. While Appvance did not use this technique, we appreciate the effort and progress there.
Of course, others will find something that moves them forward in their own goals. But we pursued another path which was to focus on deeply learning only the application under test, using a library of known actions. That embeds your libraries, your data, your usage patterns and your developers. And that is where our breakthroughs began to appear a couple years back. We know this is a different path than a fair bit of the academic work has taken. But, we already delivered the end goal of full autonomy, suggesting there may be many methods to get there.
We also thought at first we needed one master ML technique, but that was a flawed strategy because human testers don’t use one technique, they use many to create and execute and validate tests (we identified 19 areas of ML). And if we want to have AI do the job of human testers, then we need to model their actions, intuition, experience etc. And that is what we ended up doing. Not trying to find a better way to test, but instead modeling the 30 years of software test experience humans have amassed doing this work.
Humans have already progressed over a few decades to hone their skills and time and effort with the current crop of tools. While some might argue the tools are poor, in the end testing is done, bugs are found and fixed and testing is redone. That is, nature took its course and we should model what nature did thus-far before thinking we can outsmart it.
We are not the first to do this. Automation Anywhere has been modeling what humans do for a decade and is now worth billions automating sales and support functions. The setup isn’t easy, but the results are impactful. Often a 90% reduction in costs and improvement in speed and outcomes. That is something C-level execs get and buy.
While we have a lot more work to do, modeling human activities is likely to be one major method how AI gets applied to many fields in the coming years. Of course, with image and speech recognition, we could use deep learning and find new pathways to recognize patterns. And that will eventually happen in QA as well. But since each application is different and needs to be tested differently, the problem set is harder than image recognition alone.
With the ML techniques we used, we have been able to roll out level 4 autonomy in production environments (essentially fully autonomous testing without scripting or recording of any kind). And I will provide further insight into our experiences shortly.
Every enterprise needs continuous testing today as party of their CI/CD pipeline. A true AI driven test generation system can offer a huge boost to productivity, speed, and outcomes. The good news is you can already get Level 4 autonomy today from Appvance.
Kevin Surace is CEO of Appvance.ai, the leader in AI driven software testing. He has been featured by Businessweek, Time, Fortune, Forbes, CNN, ABC, MSNBC, FOX News, and has keynoted hundreds of events, from INC5000 to TED to the US Congress. He was INC Magazines’ Entrepreneur of the Year, a CNBC top Innovator of the Decade, World Economic Forum Tech Pioneer, Chair of Silicon Valley Forum, Planet Forward Innovator of the Year nominee, featured for 5 years on TechTV’s Silicon Spin, and inducted into RIT’s Innovation Hall of Fame. Mr. Surace led pioneering work on the first cellular data smartphone (AirCommunicator) and the first human-like AI virtual assistant (General Magic), and has been awarded 82 worldwide patents.
