Test Automation Myths: Building A Better Bug Trap
Most of the formation on this page will be unpopular. It might even seem controversial to some. All of this is definitely debatable, but I believe is very important to present to organizations in creating realistic expectations.
Myth #1 Automated Testing is Testing
Automated testing is not testing. Automated “tests” are better described as “comparisons against a known state”. True testing incorporates analysis that is not accomplished in most automation test suites. This is especially true at the UI level. UI automation usually only confirms that specific elements are visible or able to be interacted with by a user. It does not mean that the interface is intuitive or user-friendly. Another concern is that automated tests often do not take into account Web Content Accessibility Guide (WCAG) requirements which may be of importance depending on the circumstances.
Myth #2 Automated Testing Will Save Money
This is not guaranteed to be true. Many factors come into play before this can be determined. They include:
- Cost to create automation
- Cost to maintain automation
- Cost to license tools
- Infrastructure costs
- Number of times the tests will be executed over the lifespan of the product (interations)
- Quality of the tests written
- Test coverage
Depending on the details of these answers above an automated test suite may not save money compared to costs. For example, if the system is still undergoing lots of design changes, maintaining and improving automation made early in the project may become very expensive.
The automation test code needs to be treated just like the platform code in production including but not limited to:
- Code repository
- Coding standards
- Code reviews
- Release processes
This will add cost but allow for higher levels of quality that prevent non-deterministic and fragile tests.
Finances alone should never be the driving factor for automation projects.
Myth #3 Everything has to be Automated
Wrong. In fact, if the goal is 100% automation the project is almost certainly doomed to fail. The reality is that some things in software testing should not be automated because they are best left to human interpretation. If you are building a system for human interaction it is best not to completely remove the human aspect of accepting its usability. Some parts of software require too much subjective analysis.
Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.
Ian Malcom (Jeff Goldblum), Jurassic Park
Yes, There is a Better Way Forward
The best path to take is to come up with a Quality Assurance Automation Strategy (QAAS) that accomplishes the following:
- Focuses on automating mundane or tedious tasks
- Focuses on automating things that are repeatable and predictable
- Focuses on automating things that are mission-critical or the business
- Champions the creation of good unit tests and incorporates them into the build process
This list above reflects fundamentals based on the Testing Pyramid.
This analysis and strategic planning are what exceptional software leaders bring to an organization.