Agile Requirements Tracing with Teamscale

Teamscale provides a broad range of functionality to facilitate requirements tracing for agile software development teams. It reads requirements from several requirements management tools and interprets links to test code and test results. The gathered information about requirements and their connection to other artifacts can be visualized in a verification matrix or in a variety of dashboard widgets. Moreover, Teamscale provides a novel, lightweight way to keep the software specification and implementation up-to-date and in sync, by listing the relevant requirements for a code change directly in merge requests. As a result, teams get the benefits of requirements tracing without having to manually curate spreadsheets and avoid the bureaucracy of waterfall-like development processes.

Requirements Tracing: Build Confidence in Your System

If you need confidence that your software system is correct with respect to its requirements, it is necessary that you rigorously capture all these requirements and trace them to the tests that verify them. Unsurprisingly, requirements tracing is heavily used in the engineering of safety-critical systems and described in standards from various domains, including

• ISO 26262 for automotive software,
• DO-178C/ED-12C for avionic systems, or
• ISO 13485/IEC 62304 which is used for medical devices.

But requirements tracing is a useful ally in other contexts as well, especially when incorrect behavior of a software system may cause damage or financial loss.

One of the main goals of requirements tracing is to be able to demonstrate that the requirements have been tested thoroughly enough. To this end, a verification matrix is created, which forms the basis to investigate the relationship between requirements and tests:

The verification matrix allows you to answer the following important questions in a facts-driven way:

• Are there any requirements without associated tests?
• Do the tests for a given requirement provide sufficient risk mitigation?
• Have all relevant tests been executed successfully on the release candidate?
• Are there any tests that claim to verify a large number of requirements?

These questions aim to identify weaknesses in the verification activities for your software system.

The Problem: How to Keep Everything in Sync?

To be able to create a verification matrix, you need to combine the requirements of your system with the available tests and the connection between them. This already sounds non-trivial. To make things worse, code and tests are continuously modified during the evolution of the system. A software development project always entails a learning experience. Hence, the requirements are subject to change when the project progresses. Keeping requirements, tests, and the relations between them in sync is a major challenge in software engineering.

Standards that promote requirements tracing, like the ones mentioned above, often use waterfall-like processes as their underlying mental model. But although it is desirable that the documentation looks as if the product was developed in a fully rational top-down manner, in practice this is not happening. The following diagram depicts how one software engineering artifact influences the other:

Especially in challenging, innovative projects, going back-and-forth between working on requirements, architecture and design, code, and tests is inevitable. Agile software development methods even explicitly embrace change and learnings in development projects.

Traditional methods for creating verification matrices often require a lot of manual effort and time. This is in stark contrast to modern, change-based quality assurance methods such as merge-request based reviews and quality gates.

Teamscale helps to combine modern, agile software engineering with safety- and correctness-centric development - without the usual bureaucracy.

Ensure Consistency Between Requirements and Code in Your Merge Requests

Modern software quality assurance happens in merge requests. In code collaboration platforms like GitHub, Bitbucket, GitLab, Gerrit, etc., the merge request (or sometimes called "pull request") is usually where changes are peer-reviewed. This helps to avoid breaking anything or blocking other developers and keeps the main branch in a releasable quality. Thus, it is a natural fit for making sure that requirements and code are kept in sync. For instance, if a test is changed, the reviewer should make sure it still verifies the requirement. However, as requirements specifications are often lengthy, figuring out which parts of a specification are affected by a change is usually hard.

Teamscale makes this task much easier for the reviewer. It automatically creates a review checklist that contains all items of your requirements specification that are potentially impacted by the code changes in a merge request:

As prerequisite for this review checklist, you need to connect Teamscale to your requirements management system and point it to tracing information. Tracing can be expressed in form of links in the code. While Teamscale analyzes the commits on a branch, it also captures all specification items that are potentially impacted by the code changes. Teamscale then annotates the number of impacted specification items in the corresponding merge request, including a link to the detailed review checklist. This makes it easy for the reviewer to ensure everything is in sync before merging the change.

How to Get Your Requirements into Teamscale?

Requirements are usually stored in a requirements management tool that allows to capture requirements – and often other parts of a specification as well. Teamscale can be directly connected to several requirements management tools using dedicated connectors. As a fallback, specifications can be read from CSV files, e.g., stored in a version control system.

Since specifications usually do not only contain requirements but also related artifacts such as "Use Cases", "Goals", "Components" or items, all these entities are called specification items in Teamscale. Specification items can have user-defined types, such as "Requirement", "Use Case," or similar that reflect the different parts of the specification documents used. Teamscale stores the specification items in a fully historized manner and provides a query language for filtering them. Specification item queries also the basis for defining metrics such as "number of requirements in draft state" or "requirements that are in review for longer than 14 days". These metrics can be used in dashboards to provide insights about quality aspects of the specification next to quality data from code and test analyses:

How to Enable Tracing Between Tests and Requirements?

Teamscale is able to detect automated test cases written using common test frameworks such as GTest (C++) or pytest (Python). If you annotate tests with the identifier of a specification item, Teamscale recognizes that this test contributes to the verification of the mentioned requirement and will present it, e.g., in the verification matrix.

When using test frameworks that are not (yet) supported by Teamscale or if you have written a custom test framework, you can simply mark methods in your code that represent an entry point to a test using a dedicated, custom comment. This tells Teamscale that these methods are tests that contribute to the verification of requirements.

Whenever a test is modified, Teamscale lists the corresponding impacted specification items in the merge request, so that all relevant information is accessible immediately to the reviewer without the need to dig through lengthy documents.

If provided to Teamscale, test results are mapped to the detected test declarations as well. This allows you to make sure that the tests have really been executed successfully. You can even distinguish different test sources, for instance when your tests are executed on different platforms, and of course different branches in your repository.

What’s the Difference Between Requirements and Issues?

This is a question that often arises when teams start to introduce requirements-driven processes. What makes the situation extra confusing is that many times the same tools are used for requirements management and issue tracking (e.g., Jira). In Teamscale, we distinguish between two use cases:

• Requirements specifications are typically regarded to be part of the product and thus need to be updated while the software is evolving.
• Issues (or tickets, work items, or similar) are a means of project management and help with the planning and organization of tasks that need to be done.

Although an issue can be a task like "implement requirement X", there still does not have to be a 1:1 mapping. Issues might refer to multiple requirements, the implementation of a component, or a feature.

In many cases, issues are change requests: they describe a modification or extension of existing functionality. If the functionality that should be modified has been specified with requirements before, these requirements have to be modified accordingly. Furthermore, issues can be completely unrelated to requirements, e.g., modifications of the CI pipeline, documentation tasks, etc. Thus, issues often have final states (e.g., "Done" or "Discarded) and are usually not intended to be further maintained in this state. In contrast, requirements usually have a different lifecycle: they should persist and be updated as long as the software is intended to provide the specified capabilities.

Consequently, requirements and issues are separate concepts in Teamscale. We treat them differently although their data sources might overlap. For instance, Jira might be used as issue tracker and as requirements management tool. Teamscale provides separate Jira connectors for both use cases.