When requirements on a project change “on the fly” and you do not have at hand means of monitoring the implementation of each individual requirement for a feature or module, you have a question: how to analyze the coverage? One such tool that our QA team uses on such projects is the traceability matrix.
At the moment we use the matrix for more than 2.5 years. During this time we were able to evaluate the benefits of this tool, as well as adapt it for our project.
What is the traceability matrix?
By definition, the traceability matrix is a two-dimensional table containing the functional requirements of the product (functional requirements) and prepared test scenarios (test cases).
At the intersection of the corresponding row and column, a mark is put indicating that this requirement is covered by the given test case.
Thus, the table gives a visual display of two parameters:
- presence in the system of requirements that are not yet covered (if the requirement does not have a single intersection with test cases (sufficient condition);
- is there any redundant testing in the system - if the requirements have several intersections (necessary condition).
In our project, we use traceability matrices not only to assess coverage, but also to determine the relationship between development tasks, requirements, and test artifacts.
Therefore, the matrix has the form of a table, each row of which contains:
- number and description of the task for development from task tracker;
- logical block to which the task belongs (optional);
- atomic requirement or acceptance criteria;
- a priority;
- number and description of the corresponding test artifact.
Since we use the Jira tracker, Zephyr by Jira for test documentation and the Confluence requirements management system, all entities are synchronized and such traceability allows us to:
- visualize the current state of implementation;
- break requirements into more atomic ones and structure them;
- keep track of whether there are requirements that are not yet planned for development (implementation pass);
- track whether the requirement is currently implemented;
- track whether the requirement is covered with a test case (test skip);
- visually display prioritization requirements.
Links to the traceability matrix
Binding requirements and test cases can be:
- 1 to 1 (atomic requirement that is covered by one test case, this test case covers only this requirement);
- 1 to n (the requirement that is covered by several test cases, these test cases cover only this requirement);
- n to n (requirement, which is covered by several test cases, these test cases cover this and other requirements).
Regarding the last point, I would like to note that
- when a single requirement in a traceability matrix is covered by several tests, this may indicate redundancy testing. In this case, it is necessary to analyze how the requirement is atomically.
- If we ensure the completeness of coverage by running all the test cases, and the test cases themselves do not duplicate each other - this will not be excessive testing.
We have such cases on a project when one requirement is covered by several tests and one test can cover several requirements (links “1 to n” and ”n to n”).
Coverage assessment specifics using traceability matrices
If we use the “ratio of the number of requirements to the number of test artifacts” metrics for coverage, then the links in the matrix should be “1 to 1”, and the requirements should be maximally decomposed.
Example. We have a non-atomic requirement: “The user must be able to change and format the letter in a text editor”. One test case will obviously not be enough, but if only one artifact is linked in the matrix, the presentation will be visually that the requirement is covered.
Therefore it is better:
- divide this requirement in the matrix into separate atomic functions of a text editor;
- for each function, write an acceptance criterion;
- for each criterion to create a test artifact;
- If several atomic requirements can be covered by a single checklist, you can avoid over crushing, saving resources.
If there are not enough resources for maximum decomposition, you can use non-atomic requirements, but you need to create several test artifacts to cover each of them.
In this case, test cases and checklists for each non-atomic requirement are compiled at a time, that is, each requirement in the matrix is either completely covered with artifacts or not covered at all.
When compiling matrices, it is advisable to adhere to the recommendation that the decomposition of each requirement in a single matrix should be approximately equal, that is, one table should not contain requirements, some of which require 5 test cases, and some - one test case.
The coverage estimate is then calculated separately for each matrix.
Since our project documentation can have a different look for each feature and even a description of one feature can contain UML, diagrams, diagram of user cases and transitions, and the project contains more than 40 volume functionalities, we decided to develop a separate matrix for each module or feature, so that Do not lose any of the advantages of this tool.
Coverage assessment is also calculated separately for each module or feature.
With this approach, we can use the metric described above: “the number of requirements for the number of test artifacts”. Even if we have connections 1 to n, n to n, we have several components, each of which can be used in several modules. Requirements and acceptance criteria are described in each matrix, and one test artifact is used.
Our matrices are also stored in the Confluence requirements management system - each matrix is located with a structure as a child page of the feature for which it was designed. Also, all the matrices are collected on one page for convenience in assessing the coverage of the entire application.
Creating and maintaining a matrix
Matrix creation is included in our workflow on analytics tasks.
When we receive information about a new feature, an analyst of our team creates a task in the task tracker and works with this product together with the product owner. In the process of collecting and structuring requirements, the whole team conducts reviews and asks additional questions. When the requirements are formulated, documented and confirmed by the customer, the team lead of the development creates tasks for the development of this feature, and the testing team can begin to create a trace matrix.
And here we can distinguish the following stages of the compilation of the Traceability Matrix:
- At the beginning, the requirements are decomposed and subject to prioritization by the QA team and \ or product-owner. The result of the stage is a structured and prioritized list of all requirements for this functionality.
- The second stage will be communication with the development team and putting tasks from the task tracker for development into a matrix to the relevant requirements. As a result, we can track the traceability of requirements and development tasks.
- The third stage is the development of test cases and checklists.
This stage is carried out either before testing or during testing of a specific task.
If the functionality is new and the interface changes, there may be cases in which the steps are best described immediately before the testing of the task begins.
If the implementation functionality is similar to one of the existing features, then we can proceed to the description of test cases with steps immediately after the review and decomposition of requirements. - Stage 4 - filling the matrix with test cases.
According to the results of the whole process, we get development tasks, test cases for testing and a traceability matrix that unites them and requirements.
The task of developing requirements is closed. - Stage 5 - maintaining the matrix up to date. Changes must be made with any modifications to the requirements. You should also take into account the integration of the relationship between the two matrices that describe different features or modules, and when changing one, be sure to check whether there is a need to edit the second.
Challenges in working with the traceability matrix
- Update
The matrix will be useful only under the condition that it will always be kept up to date. On our project with frequently changing requirements, the actualization took a lot of time, but if the matrix is not actualized, it becomes not only useless, but also causes confusion.
How to solve :
Partially solved the problem with frequent changes in requirements and moved the stage of creating the matrix at the time when the requirements have already been reviewed by the team and confirmed by the customer.
The team decided that the analyst would update the requirements not only on the page with the description of the feature, but to find and update them in the matrix, highlighting it with a different color. This helped the whole team not to lose the changes, and the QA team in particular - to see which test cases need updating. - Temporary resources
The project may have an urgent release and work with new requirements at the same time, and all QA resources are sent for testing, not work with requirements. Thus, the debt on test documentation increases.
How to solve :
If all QA specialists are busy testing priority tasks, we postpone the creation of a matrix for a specific feature. It is transferred as much as possible at the moment of testing the first task for this feature, and in this case the matrix is filled with test cases as the tasks in which the feature is implemented are tested.
QA-specialist lays in the evaluation time, not only to write the test cases themselves, but also time to develop the matrix. - Efficiency.
If the project is small and all the requirements are in the form of a structured TK, and test cases are created for each requirement immediately, the traceability matrix in our form will only duplicate information and be a waste of resources.
Therefore, you need to use the standard matrix described in the definition to estimate coverage.
Amenities
- The matrix allows you to monitor the implementation of requirements, to monitor that all requirements are developed and tested and nothing is missing.
- The matrix helps the QA team keep track of whether there is a debt on test documentation, and which specific requirements are not yet covered by test cases.
- The tool is used by the analyst and the QA team to control the changed requirements.
- On the project, the traceability matrices were used not only by us, but also by the product owner by the customer. So they made sure that all the requirements are there and they are correct, and tracked with the help of the matrix that has already been implemented. The matrices have allowed us to make the development and testing process somewhat transparent.