Application Coverage™ is the new gold standard metric of testing completeness, having supplanted the old-school metrics of Test Coverage and Code Coverage. This is because Application Coverage mimics user experience and can only be comprehensively achieved via generative-AI. Test Coverage and Code Coverage are limited because they are human dependent in terms of test conception, creation and execution. They fall short because they look for errors based on the users’ expected behavior, not how users actually experience the application-under-test (AUT), and thereby allow defects to pass into production. Those escaped defects inevitably get discovered by real users.
Old School Coverage vs. New School Coverage
Let’s drill down into each of these three coverage metrics.
Code Coverage
Code Coverage still has a place in the SDLC, as assessing it can be a valuable exercise for the development team, especially during code reviews. However, such assessments cannot anticipate how users will actually traverse the AUT and therefore how unanticipated user flows may expose users to bugs.
Test Coverage
Test coverage is measured against assumptions about how users will traverse the AUT, assumptions that tend to be trumped by actual user activity. Further, as long-lived and often complex applications grow and change over time, the actual behaviors of users may diverge significantly from the initially defined requirements set by product managers. This means that the measured test coverage often bears little relationship to the effective requirements that real users place on the AUT.
Application Coverage
Application Coverage is the new kid on the block and owes its existence to generative-AI based testing, such as from Appvance. Generative-AI is uniquely capable of mapping and then testing every possible path that users can take through an application, thus allowing comprehensive Application Coverage. Because generative-AI based testing operates as users do, but does so with superhuman speed and comprehensiveness, it needn’t make assumptions about what users may do, nor does it need to economize for limited staffing bandwidth. It tests every possible flow.
Examples abound about bugs experienced by users due to unanticipated paths. It is a universal truth that the first bugs reported after a new release are those where the user goes off script, does something unexpected, and triggers a behavior no one has seen before.
Concerns with Generative-AI Testing
One concern with generative-AI based testing and its resultant complete Application Coverage is that too many bugs will be found, thereby overwhelming the dev team with required fixes. This is understandable yet obviously shortsighted, a classic example of pay-me-now or pay-me-later. Sure, the initial usage of Appvance IQ is likely to catalog a longer list of defects than does human-based testing, but it is surely better to know about those before releasing the code to production. The scrum team can still prioritize what gets fixed and in which order. Plus, the initial bumper crop of exposed defects is just that: initial. The dev team can take comfort that they won’t be tortured by a drip, drip, drip of production bugs, especially given the much higher profile that production bugs assume.
The Path to Application Coverage
Generative-AI is the best thing to happen to software quality since, well, forever. It frees people from machine-like work even as it makes complete Application Coverage possible. The result of this new gold standard quality metric is faster release cycles, assured quality, and more rewarding work for everyone involved in the SDLC.
Want to see Application Coverage taken to the next level, request a personal live demo here.
