Testing, as it has traditionally been done, may no longer be a good idea.
I am talking here about end to end, usually UI based, testing. Even the automated kind. The kind that I myself have specialized in for several years!
Once simple fact cannot be ignored:
- Quality for customers is determined by application code quality
- Tests, in of themselves, do not improve the quality of application code
These simple facts have bothered me greatly in my role as an automation specialist over the past few years.
For much of our industry, End To End testing has moved from manual to automated processes and yet at company after company the approaches I encounter still reflect waterfall and command and control approaches – the UI tests are written by someone other than the developer and the feedback to the developers comes days to weeks later and is done by someone else who is then in a defensive, checking, testing role, not a quality improvement role. ‘Our QA is now called QE and is embedded in the team’ is a common refrain. Unfortunately the next comments are often about how hard it is for them to keep up with developers when they write tests. Not to mention the fact that their best QE just got “promoted” to (application) developer and now earns $45,000 more per year. Actions always speak louder than words and 2nd class citizen syndrome becomes rampant and accepted by all. “The QA person” has quite a remarkable set of assumptions and triggers implicit biases (many based on real evidence) in our industry. The other issue is that the testing code base itself will take more maintenance over time, quickly becoming an additional issue for code quality and needing tests to test the tests and even tests to test them.
There are several key approaches that need to be adopted to address this change. These approaches are well known by many organizations, however they still struggle to realize the changes that are needed in existing processes including architectural approaches.
The key new approaches are:
- CI – Continuous integration to run all tests in the cloud for all branches during development
- TDD measurements as KPIs, reporting and compliance measures
- Immediate feedback from production customers by automated means
- Canary Releases
- Blue Green Releases
- Feature Flags
- Speed – Avoiding testing suite time lengths that continually grow
- Continuous Deployment reducing MTTR (Mean Time To Recover)
- Teams that promote contributions from all members and pay equitably
It’s exceedingly hard to do the above because most organizations default to continuing the previous developed testing characteristics of
- Manually run automation and manual testing
- TDD compliance not monitored as a KPI
- Measuring bugs and focusing on speed of response to bugs
- Production customer real-time automated feedback KPIs not shown in-house on primary displays to development teams
- Test Suites that grow in length every week
- QA’s being failed or junior developers that are paid less
and doing those activities quickly leads to no time to do the previously mentioned ‘new approach’ activities that would actually be of more benefit in improving quality for customers. Change is hard, especially when it appears to mean less testing and more risk. When done correctly it can actually mean more testing (more unit, less UI) and less risk if many supporting parts are done correctly but moving to this model is very hard and the more established the company and their software development shop, the harder it is. This is one of the key reasons that small companies and startups continue to disrupt, as it is generally easier to adopt new practices than to change existing practices that were successful in the past.
To summarize, improve quality for customers with
Less End to End testing…
and more quality activities such as…
Code Linting, Code Grading, Code Reviews, Code Education and Training, Immutable Objects, Internal Iterators, TDD Measurement,Well named objects and methods, avoiding premature optimization, Short methods, Short Classes, Single Responsibility, English readable, well stubbed and mocked, SOLID principle based code that is deployed in a modern CI/CD environment that provides immediate automated feedback based on customer activity and provides the facility to revert changes with minimal efforts within seconds or minutes.