Automation Myths

False. Not all tests should be automated.

This may seem like a very controversial statement at first but the truth is that some tests should remain manual. Blanket statements like "automate everything" are not based on analytics and do not demonstrate an engineering mindset. Even worse, these one-size-fits-all statements could demonstrate that decision makers are acting more emotionally than logically.

Companies have limited resources. This includes people, time, and money. An automation strategy needs to reflect this reality.

A better approach is to develop criteria for determining which tests should be considered eligible for automation. A scoring or prioritization process can be implemented to determine priority. If needed, this process can be cyclical (think sprints or spirals!) and repeated as often as necessary. This ensures that the process is adding value and improving software quality immediately.

Automation is perceived as too difficult for a number of reasons. Those reasons can be traced back to common root causes like:

• Poor design (project or platform)
• Poor planning
• Lack of support from management
• Poor engineering practices

Creating a realistic and sustainable plan that's well funded and contains concrete goals is the first major step to take in test automation. Engineering these tests using established protocols with small incremental progress is key to sustained progress. Early success when combined with sustainable effort will encourage management to stay committed.

Just remember, the programs being tested by these automated tests was not completed in a day, neither will the testing programs. The test code must be treated just like your production code to help ensure the project does not collapse under its own weight.

The bar chart below graphs the actual progress of implemented tests over a period of a year by a small team under my management. One data set shows API tests, the other data set shows DB tests. This is what test automation growth looks like under a sustainable engineering plan.

Test automation is not guaranteed to save money and cost savings should never be the primary reason for beginning any test automation project.

As an automation test suite grows the maintenance costs will most likely grow significantly. Test automation code does not live in a vacuum. It is subject to a changing code base it was written to test. New tests will almost certainly be required and existing tests run the risk of being stale and useless without the proper attention.

Depending on the number of people involved, their skill level (salary), and benefits, the size of the investment may be significant over time (see myth #4). Test automation needs to be sustained and funded plan to avoid the tests becoming neglected. Neglected tests not performing accurately are worse than having no tests at all because they run the risk of generating false positives.

False positives rapidly degrade confidence in an automated test suite, especially outside of the Quality Assurance Team and can be a cancer to the health of the automation project. Trust is something that is quickly lost and slowly recovered.

Test automation is not guaranteed to save time and time savings should never be the primary reason for beginning any test automation project.

An automation test suite execution time depends on many factors, including, but not limited to: resources, complexity, volume, and frequency. In addition, it takes a significant amount of time to engineer a meaningful and sustainable testing strategy and platform (See myths #2 & #3).

Depending on the codebase associated with the automated tests, it may be a significant amount of time to repair tests. If automated tests are used to gate pipelines, test pass rates could become a prerequisite for events to trigger. For example, a team may wish to enforce a 100% unit test pass rate before allowing a pipeline to install in a test environment. This may cause a halt in operations until that criteria is met.

So if an argument is being made that the primary reasons for doing test automation are not time and cost savings, what are the primary reasons for starting an automation project?

Software quality assurance, customer satisfaction and revenue protection should be the primary reasons for automation efforts.

Time and cost savings are secondary achievements and may come into play at some point in the future but should not be the primary driver.

Forgetting why software quality assurance exists in the first place (protecting investments, preserving revenue, ensuring positive reputations) leads to poor management decisions. This includes starting projects for the wrong reasons and having them focus on the wrong KPIs.