From Coding to Support - Which Tests at Which Stage

The art of testing lies in the ability to put the right checks in the right places to provide the right people with information at the right time. And, tests failing is just one form of an error.

This means that while building software we need to be questioning how this information gets surfaced.

In terms of writing code this should obviously take the form of unit tests. We automate these because the cost of running these manually all the time is ridiculous.

In terms of integrating these components we create integration tests which check integration points. We should also look at an End to End test to ensure everything can talk successfully on the infrastructure. Again, we should automate these because the cost of running these manually all the time is ridiculous.

Finally we should put monitoring in place. This is a form of testing - providing feedback to the team on whether the product/feature they own is providing the expected value to users.

We should always put the right level of testing in place at each stage (unit, monitoring, UI, integration) to provide us the confidence we need. We should always put as much effort in to the earlier testing phases, and avoid duplicating any tests; by doing this we will see a natural funnel appear (aka the Testing Pyramid), where the majority of tests are Unit, and minimal tests are UI based.

Putting the logic-checks in to earlier test stages (such as Unit tests) makes sense because tests further down the Software Development Life Cycle (i.e. UI tests) cost more in terms of time and resources to run (think of the time cost of running 1 unit test vs 1 UI test; milliseconds vs minutes).

We get one other added benefit which is that the right people are notified at the right time of breakages. We often see a delay in a code change and UI tests being run for a variety of very reasonable reasons. When Unit tests break the developer is aware of this early on, while the code is still fresh in their mind.

We need to also consider testing after a product is released though, looking at what checks are in place to provide us the "Four Golden Signals" as Google has so kindly given us; we need to monitor our apps and ensure the team is notified of failures, post deployment.

The responsibilities and roles of the QA are suddenly very twisted; where does the Quality Analyst role end and the Sys-Ops role sit. More on the Cross Functional Developer role in another post.