Software QA is undergoing a sea change due to generative AI-driven testing. Given that, this post compares and contrasts generative AI (GenAI) test creation with traditional scripting methods. It does so across half-a-dozen aspects of scripting, including the development process, efficiency, accuracy, customization and adaptability, maintainability, and ongoing improvement.
6 Points of Comparison
1. Development Process
Generative AI
- Automated Generation: GenAI automatically generates scripts for an application-under-test (AUT). Not surprisingly, automated generation of scripts saves an immense amount of time in the quality process.
- Learning Curve: Users need to learn how to work with the GenAI-based system.
Traditional Scripting
- Manual Generation: Scripts are manually created by developers or testers in a very time-consuming and often error-prone process.
- Scripting Expertise: Traditional scripting requires expertise in scripting languages, on top of domain-specific knowledge of the AUT. It also requires version control and software-change processes for the tests, not unlike the control processes used for the AUT itself.
2. Efficiency
Generative AI
- Speed: GenAI quickly generates full portfolios of scripts that can number in the thousands. This game-changing efficiency frees up test engineers to focus on analysis and remediation, and to close the gaps in coverage and cycle-time that characterized the pre-AI testing paradigm.
- Coverage: GenAI produces a full portfolio of scripts that provide complete application coverage. This goes far beyond the coverage possible from human generated scripts, and thereby helps uncover unexpected issues. My colleague Kevin Parker’s recent post Application Coverage: The New Gold Standard Quality Metric goes into detail on this.
Traditional Scripting
- Tailoring: Manually created scripts can be tailored to very specific testing requirements. Hence, manual script creation is best focused on narrow areas of the AUT that have obscure or sensitive requirements.
- Inconsistency: The manual nature of traditional script creation means that they are often inconsistent in form and function.
3. Accuracy and Reliability
Generative AI
- Accuracy: Generated scripts should not contain errors or inaccuracies if generated with the right model, e.g., from a purpose-built GenAI quality platform like AIQ.
- Training: The scripts might not initially align with intended testing goals. Additional training for the GenAI-driven testing platform may then be required.
Traditional Scripting
- Precision: Manually created scripts are aligned with the specific requirements of each test case. As noted above, the labor intensity of manual scripting means that it is best focused on narrow areas of the AUT with such specific requirements.
- Unreliable: Manually created scripts tend to become less reliable over time because they are not continually refreshed, as is done with GenAI created scripts.
4. Customization and Adaptability
Generative AI
- Adaptability: GenAI scripting easily adapts to different inputs and generates a broad portfolio of tests for a wide variety of scenarios.
- Limitations: GenAI may have limitations in understanding complex and domain-specific requirements, albeit additional training typically solves this.
Traditional Scripting
- Customization: Scripts can be highly customized to meet the specific needs of each test case, albeit at steep cost and with a short shelf-life.
- Instability: Manually created script portfolios tend to become unstable over time since they are not continually refreshed as is done with GenAI scripts.
5. Maintainability
Generative AI
- Training: Ongoing training of the system adapts it to evolving software and testing landscapes.
- Version Control: Good version control and documentation is a byproduct of system generated test scripts. Further, the constantly refreshed nature of GenAI-generated script portfolios virtually eliminates version-control overhead.
Traditional Scripting
- Control: Manual scripting offers full control over script maintenance, updates, and modifications, albeit at the cost of substantial and continued human effort
- Version Control: Good version control and documentation requires considerable and ongoing labor and discipline.
6. Learning and Improvement
Generative AI
- Learning Ability: A high-quality GenAI based platform, e.g., AIQ, learns and improves over time, adapting to new patterns and scenarios.
- Feedback Loop: A GenAI-based platform improves its model by being continuously trained on new data.
Traditional Scripting
- Static Nature: Scripts are static and do not learn or adapt, unless manually updated by developers. This leads to the overwhelming labor required for traditional scripting.
- Knowledge Transfer: Manual scripting relies on knowledge transfer and learning among team members for improvement. Such institutional knowledge tends to degrade over time.
Conclusion
GenAI and traditional scripting are not mutually exclusive. However, GenAI’s immense speed, coverage and adaptability make it a game-changer that should be used as extensively as possible. Given traditional scripting’s heavy labor requirements and the fragility of its output, it should be reserved for highly complex or sensitive features. Even there, GenAI is increasingly capable of covering edge cases.
The bottom line is that GenAI leapfrogs traditional scripting and must be exploited by software quality leaders who are dealing with the twin imperatives of limited staffing and high-quality standards.
