Setting up Test Gap Analysis on a .NET Core Application (Blogifier) ​

As a developer or tester of a .NET application, you may use Teamscale's Test Gap Analysis (TGA) to improve your testing effectiveness. This tutorial leads you through the setup of TGA on Blogifier, an open source web application that can be used to self-host a blog.

Step 1: Installing Blogifier ​

As a prerequisite for running Blogifier, install .NET Core Runtime 6.0 (or newer).

Next, download and unzip the 3.0 release of Blogifier. Note that the release zip contains not only the executable, but also the corresponding PDB (symbol) files.

Finally, start Blogifier.exe and open it in your browser, to ensure that the installation was successful.

Step 2: Installing the Teamscale Ephemeral Profiler ​

Download the latest release of the Teamscale Ephemeral Profiler and unpack it to C:\teamscale_dotnet_profiler.

To register the profiler, multiple environment variables have to be set. This can be accomplished either by creating a launch script, or by assigning the environment variables system-wide in the Control Panel. For example you can create the Blogifier.bat batch file next to the Blogifier.exe that sets the necessary environment variables and starts the executable:

set CORECLR_ENABLE_PROFILING=1
set CORECLR_PROFILER={DD0A1BB6-11CE-11DD-8EE8-3F9E55D89593}
set CORECLR_PROFILER_PATH_32=C:\teamscale_dotnet_profiler\Profiler32.dll
set CORECLR_PROFILER_PATH_64=C:\teamscale_dotnet_profiler\Profiler64.dll
Blogifier.exe

set CORECLR_ENABLE_PROFILING=1
set CORECLR_PROFILER={DD0A1BB6-11CE-11DD-8EE8-3F9E55D89593}
set CORECLR_PROFILER_PATH_32=C:\teamscale_dotnet_profiler\Profiler32.dll
set CORECLR_PROFILER_PATH_64=C:\teamscale_dotnet_profiler\Profiler64.dll
Blogifier.exe

If you want to set the environment variables system-wide, you can do that by navigating to Control Panel > System > Advanced system settings > Environment Variables and adding all the CORECLR_ variables.

Setting the CORECLR_ENABLE_PROFILING variable enables profiling on .NET Core processes. The CORECLR_PROFILER variable provides the GUID of the Teamscale Ephemeral Profiler, which the .NET Runtime uses to verify the profiler DLL on load. Finally, the CORECLR_PROFILER_PATH_32 and CORECLR_PROFILER_PATH_64 tell the .NET Runtime where to find the profiler binaries.

Next, we configure the profiler itself. Rename C:\teamscale_dotnet_profiler\Profiler.example.yml to C:\teamscale_dotnet_profiler\Profiler.yml, then edit the file.

Adjust the global configuration section, i.e., the first entry below match, as follows:

{
  # ...
  match: [
    {
    # This section matches all processes.
    # We configure shared options here but disable the profiler.
    # The later match sections will enable the profiler for specific processes.
      profiler: {
        enabled: false,
        targetdir: "C:\\blogifier_coverage",
        upload_daemon: false
      },
    # ...
    }
  ]
}

{
  # ...
  match: [
    {
    # This section matches all processes.
    # We configure shared options here but disable the profiler.
    # The later match sections will enable the profiler for specific processes.
      profiler: {
        enabled: false,
        targetdir: "C:\\blogifier_coverage",
        upload_daemon: false
      },
    # ...
    }
  ]
}

This globally disables both the profiler, to avoid profiling unnecessary processes, and the automatic upload of coverage, which we will configure later. Moreover, it instructs the profiler to write trace files to C:\blogifier_coverage.

Next, add a process-specific configuration section below the global one (delete all other sections from the template), to enable the profiler specifically for the Blogifier executable:

{
  # ...
  match: [
    {
    # This section matches all processes.
    # ...
    },
    # These sections turn on the profiler for specific executables.
    {
      executableName: "Blogifier.exe",
      profiler: {
        enabled: true
      }
    }
  ]
}

{
  # ...
  match: [
    {
    # This section matches all processes.
    # ...
    },
    # These sections turn on the profiler for specific executables.
    {
      executableName: "Blogifier.exe",
      profiler: {
        enabled: true
      }
    }
  ]
}

To test your setup, start Blogifier with the batch file you have created in Step 2. This will open a console window that shows the logs of the web application, and will start the application on the default 5000 port. Open the application in the browser, do something (e.g., create an admin user), and then stop Blogifier by pressing Ctrl+C in the console window. The target directory C:\blogifier_coverage should now contain a coverage report named like coverage_20230629_1352510131.txt (the date and time parts of the file name will differ) with some content.

Step 3: Uploading Coverage to Teamscale ​

The Teamscale Ephemeral Profiler comes with the ability to upload coverage directly to a Teamscale instance. During the upload, the profiler maps the recorded coverage to the source code of the application under test, using the respective PDB files.

Step 3.1: Preparing Teamscale ​

In Teamscale, go to Project Configuration and click New project.

Name the new project blogifier and choose the analysis profile C# Default.

In the Branching Configuration, change all start dates to 2019-09-01.

Click Source Code Repository and select Git. Create a new account named Blogifier GitHub with the URI https://github.com/blogifierdotnet/Blogifier.git and without credentials. Set the Default branch name to main.

Click on Advanced Settings and set the End revision (expert option) to 2022-02-16. This day is after the v3.0 release, so this way Teamscale will analyze all changes between v2.5 and v3.0.

Click Create project.

Step 3.2: Linking the Version Under Test to a Source Code Revision ​

Create the file C:\revision.txt and set the file's content to revision: 8e634ab4b93a60df0e216f3929b35c17cbffb1ca. This revision identifies the source code corresponding to the Blogifier 3.0 release, which we want to link the coverage information to.

Step 3.3: Uploading Coverage ​

Edit C:\teamscale_dotnet_profiler\Profiler.yml again and change the global configuration section to match the following:

{
  # ...
  match: [
    {
    # This section matches all processes.
    # We configure shared options here but disable the profiler.
    # The later match sections will enable the profiler for specific processes.
      profiler: {
        enabled: false,
        targetdir: "C:\\blogifier_coverage",
        upload_daemon: true
      },
      uploader: {
        teamscale: {
          url: "http://localhost:8080", # use your URL
          username: "admin", # use your username
          accessKey: "u7a9abc32r45r2uiig3vvv", # use your access key
          project: "blogifier",
          partition: "Manual Tests"
        },
        pdbDirectory: "C:\\path\\to\\Blogifier", # use the folder where Blogifier.pdb is located
        revisionFile: "C:\\revision.txt",
        assemblyPatterns: {
          include: [
            "Blogifier.*"
          ]
        }
      }
    },
    # These sections turn on the profiler for specific executables.
    {
      executableName: "Blogifier.exe",
      profiler: {
        enabled: true
      }
    }
  ]
}

{
  # ...
  match: [
    {
    # This section matches all processes.
    # We configure shared options here but disable the profiler.
    # The later match sections will enable the profiler for specific processes.
      profiler: {
        enabled: false,
        targetdir: "C:\\blogifier_coverage",
        upload_daemon: true
      },
      uploader: {
        teamscale: {
          url: "http://localhost:8080", # use your URL
          username: "admin", # use your username
          accessKey: "u7a9abc32r45r2uiig3vvv", # use your access key
          project: "blogifier",
          partition: "Manual Tests"
        },
        pdbDirectory: "C:\\path\\to\\Blogifier", # use the folder where Blogifier.pdb is located
        revisionFile: "C:\\revision.txt",
        assemblyPatterns: {
          include: [
            "Blogifier.*"
          ]
        }
      }
    },
    # These sections turn on the profiler for specific executables.
    {
      executableName: "Blogifier.exe",
      profiler: {
        enabled: true
      }
    }
  ]
}

This globally enables and configures automatic uploading of coverage data.

Use the URL and credentials (username and access key) of your Teamscale instance. The project option contains the ID of your Teamscale project such that the coverage data is sent to this project. The partition is an arbitrary label that you may use to distinguish coverage source, such as different test stages. Teamscale merges all coverage in the same partition and allows you to inspect coverage in different partitions individually (or in any combination).

During the upload, the Teamscale Ephemeral Profiler maps the coverage data back to a particular version of the source code, using the PDB files and the revision ID from the revision.txt. It will do so for all the assemblies of Blogifier (whose names match the regular expression Blogifier.*).

To test your setup, start Blogifier.bat, do something (e.g., create a new blog entry), and stop it again. The profiler automatically schedules uploading of coverage data from terminated processes every five minutes.

To verify that everything worked as expected, go to the Activity perspective. If you filter your commits for the Commit Type "External Analysis", and the data was processed successfully, you will see the coverage upload.

Step 4: Analyzing Coverage and Test Gaps ​

Step 4.1: Test Gaps at File Level ​

In Teamscale, go to Test Gaps > Files and set the baseline to Git Tag v2.5 at the top.

The table on the page shows the file-system structure of the Blogifier project, with Test Gaps, Execution, and Churn data aggregated for each file or directory. You may drill down into the folders and click on the colored bar charts to check what exactly has been changed or executed and where the test gaps are.

Step 4.2: Test Gaps at Method Level ​

In Teamscale, go to Dashboards. In the top bar click Add a new dashboard and then choose Test Gap Treemap from the list of widgets. Finally, click Save Dashboard and name the Dashboard Blogifier Test Gaps.

Move the mouse over the Test Gap Treemap and click on the icon that appears in the upper left, to edit the widget. As the Baseline, select the Git tag v2.5. Make sure that Manual Tests is selected under Coverage Sources, for the widget to consider the coverage you uploaded earlier. Confirm the dialog.

The widget now shows the code changes between version 2.5 and 3.0 in color. The red and yellow blocks represent methods that have been changed, but were not covered by tests, i.e., test gaps. Hover the mouse over the blocks in the treemap to see details in a tooltip. Click on the blocks to drill into the data.

Further Reading: ​

1. Test Gap Analysis
2. Test Gaps Perspective
3. How to Create a Test Gap Analysis Dashboard
4. Working with Test Gap Treemaps