An area of constant debate in the software industry revolves around automation of tests and who takes part in their design, creation and maintenance. There are many blogs on the subject about who within the team should do this, and different scenarios have varying degrees of success for the team, product and company.
Overall, I believe there are core guidelines that should be considered and adhered to when approaching this topic.
The Current Approach from Industry
A common scenario I see is where a team, working diligently using Test Driven Development (TDD), has developers writing production code and building a test infrastructure at the unit and lower-level integration layers to prove their code and receive fast feedback. At the same time, and in order to complete the testing pyramid, QAs write test code to verify higher level integration scenarios that are driven via the UI, e.g. using webdriver to exercise page content and verifying that the user ends up on the right page with the right text/message/values.
This can progress well whilst the app is small and there isn't much functionality or related tests. As it grows (even past a couple of pages on the UI), very real and difficult questions that the team comes across are
"Do we have duplication of test coverage?" and
"Is our test suite structured in a cohesive manner throughout the whole stack?".
Duplication
For the former question, if there are effectively two groups of people owning some tests but not all, a natural divide evolves.
The developers trust that the QAs know what they have to test at the higher level and the QAs trust that the developers are testing 'the right stuff' at the lower level.
In theory this seems plausible, however what happens is one of two things :-
1) QAs spend most of their day reading and writing automated tests in order to fully understand what to add to the framework. While this may seem to be a good thing to some (the testers are writing tests all day after all!), it impacts the rest of the essential tasks in the role e.g. assisting in shaping stories with BAs, performing structured exploratory testing sessions, engaging with the Product Owner and ensuring their feedback is taken onboard by the team. The QA spends most of their time to the right of the wall and much less to the left. This approach to the role loses a lot of the benefit that a QA brings to a team working iteratively, described here.
2) QAs attempt to create a layer of verification without the required in depth knowledge of the lower level tests. This creates duplication and brings about the ice-cream cone anti-pattern.
An example is a standard client/server web app, with a story to add a page of account details for a customer to enter their name, address, DOB etc. Developers write unit tests to verify javascript validation of inputs in the browser (allowed number of chars, numbers, illegal chars), and integration tests to check server-side validation of content and that it is persisted. QAs write tests driving the browser for a full stack validation, checking various combinations of inputs and asserting the account details are also saved.
What would have sufficed from that written by the QAs is a single webdriver test to ensure that the client is talking to the server, not rechecking field validation, nor the data being persisted in a database.
Cohesion
Moving on to the latter question, "Is our test suite structured in a cohesive manner throughout the whole stack?", the scenario above leads to effectively two teams committing to one codebase. If both sets of people have a sufficiently high level of skill and experience in working with code, and there are other people able to take on the other duties described in 1), then problems can be minimised (they never completely go away though).
What does a sufficiently high level of skill and experience mean?
The team needs to be confident that code is being written in a maintainable format. Consideration has to be paid to existing tests, any refactoring of them based on the work in progress, the testing framework, libraries and patterns used, naming conventions etc. Referring to the example above, a more cohesive team would be aware of the need to have the single webdriver test at the top of the testing stack and that each layer has to contain different types of test.
Four Principles for Test Automation
Building a testing infrastructure is a complex process as it is continuously changing and evolving as the product under development does. This test code should be treated as a first class citizen, in the same manner as the production code.
With this in mind, and acknowledgement of the discipline that is required in software development, I believe the following four statements are guidelines the team should follow when approaching test code :-
- Only work on test automation if you have the skills to work on production code if required.
- Only work on production code if you have the skills to work on test automation if required.
- In case of 1) or 2) not being true, pair with someone who meets the requirements of 1) and 2) until it is.
- The team must own and contribute to test code, not certain individuals or disciplines within it.
Principles in Action
Applying these principles to the above example, and assuming that the QA in the team doesn’t meet 1) or 2), when the account details story is picked up the QA and Devs firstly discuss the story and what needs to be tested. This will include unit tests of the validation in javascript, integration tests of the server-side validation and data persistence, and a journey test to ensure the new page actually connects with the DOM and data flows as expected.
The developers then use these scenarios as part of their TDD work. The QA performs exploratory testing as and when functionality is made available, providing feedback until the acceptance criteria are met and the story is ready for showing to the Product Owner.
The first benefit here is that, as the developers have been involved in all of the test stack, they are able to refactor any existing tests and add new ones such that no duplication occurs. Next, the QA has had a chance to provide valuable input on what to test. Thirdly the testing infrastructure is built in a maintainable manner and is owned by the team.
A long-term career benefit for QAs with this approach is that they are regularly exposed to best practices for structuring test code - in time, they can (if they choose to) work to meet steps 1) and 2) and bring the team closer to the polyskilled model.
How do you think these principles affect the shape of your team? Do you believe that everyone creating and updating the test code has the capabilities to work on production code as well?
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.