Summary of the book Practical Guide to Testing in DevOps, Katrina Clokie

The book tells how to build testing, not just to catch bugs, but to avoid their appearance. We liked it very much, so we decided to support the tradition of notes on Habré on the rights of old-timers and lay out the most interesting theses.

How the approaches to testing organization changed:

Waterfall
It all started with testing with the Waterfall methodology. This stage is characterized by the fact that testing is the activity of only one team member (guess who). No one except the tester knows anything about the test strategy and does not have access to test cases and checklists. Tasks fly around the team like a ping-pong ball. This period is characterized by long releases.

Agile
In work on the Agile methodology, testing becomes the responsibility of the whole team. This means that not only testers, but also developers find and solve problems. This stage is characterized by frequent releases and fast feedback.

Devops
Now support, analytics, infrastructure, monitoring influence testing. Borders and character of testing change. Information now comes to the team from different channels: monitoring, applications from the support service, analytical reports. In this case, testing should be reliable, but not hinder rapid releases.

Testing in the era of DevOps and CI

The author claims that DevOps is much bigger than CI. CI focuses on technical practices that accelerate code writing (for example, version control, unit tests, frequent commits) and DevOps on organizational changes (in particular, support for closer collaboration between software vendors: analysts, support, development team).

Without Agile, there would be no DevOps culture. People need to think flexibly to come to DevOps, whose main goal is reliability and frequency of releases. In this case, the topic of testing is often overlooked in DevOps reports.

The main thesis of Katrina is that it is necessary to test always and at every stage - from the beginning of work on the task to the last release commit.

The amount of work is very large. The question arises how to organize all the testing and not to go crazy.

Where to start testing

• Understand how the testing process on the project is organized.
• Organize a retrospective of testing strategies for the whole team and answer the question of what we are testing now and why. It may turn out that the participants of one team will respond differently to questions about the process, and this is wrong.

DevOps is more than just flexible testing. The ideology of DevOps implies a well-established process of not only rapid deployment of a new version in production or rollback to the previous, but equally well-functioning communication between the dev and operation commands.

10 criteria to help check if Agile is in your testing:

1. The whole team clearly knows what needs to be tested while working on a particular user story.
2. Everyone has a common understanding of business requirements.
3. When you discuss a user story, you have the answer to the question "How are we going to test it?"
4. Everyone in the team knows how to run autotests and where to watch the result.
5. You discuss in advance what you will automate and at what level so as not to duplicate tests at different levels. (This point seemed to us the most important).
6. Your test scripts are versioned and stored with the source code, since the tests are part of the software.
7. You do not have bugs in backlog, because you correct errors as soon as you find them, and not just register them.
8. No downtime in the CI server.
9. During the rally, it is unclear who is the developer and who is the tester.
10. Your team can evaluate the quality of the product. Everyone understands how the testing process on the project.

You can check yourself by reference - here the author explains why these items are highlighted and how important they are.

DevOps cooperation practices

The more often people communicate, the more they understand what their colleagues are doing and how they can be helped. Therefore, Katrina invites them to discuss testing more often. This can be done in several ways:

• Review checklist analysts and support. Involving early testing can improve the quality of releases.
• Testing in pairs: tester and analyst, support and developer.
• Rotation of employees between the dev teams and support.
• Dojo (as part of software development) is an environment where people can learn and practice their development skills together: master classes, knowledge sharing.

Production Testing

The following tools will help facilitate production testing:

• Monitoring set up so that it can easily find and identify the problem,
• Alerts using various communication channels: for example, a message to the instant messenger, if the server fell, or email, when the memory has run out,
• Analytics (for example, Google Analytics), talking about how many users use the functionality,
• Logging made readable.
• Feedback from customers - reviews in stores, feedback forms in web applications.

PS

This summary does not include many interesting things. The book is well written, we recommend it to testers who have thought about testing on DevOps projects.

Finally, a couple of useful links:

• Book can be bought here .
• Link to a video of Katrina's performance with Agile Testing Days 2017.
• Link to the blog of Katrina Clocks.