Custom Check Framework

Custom checks can be used to easily define rules checked by Teamscale. The Teamscale distribution already comes with about 100 build-in checks for different languages, which are build upon the same framework.

The »Teamscale Custom Check API« allows users to extend Teamscale by writing custom analyses that create findings. These custom checks are executed within Teamscale's incremental analysis engine and, thus, provide real-time feedback to developers on every commit. Consequently, Teamscale treats the findings created by custom checks equal to all other findings, such that all finding-related features, like tolerating, flagging as false positive, visualizations, filtering etc., can be used just as usual.

A custom check is a local analysis in Teamscale, i.e. its implementation is given a single source file as an input and the check can create findings for code locations within this file. A check can use the source code of the file as plain text, the sequence of tokens or the abstract syntax tree (AST) of the file or a combination of these for its analysis. Each Teamscale distribution already contains many built-in checks, e.g., for detecting bad practices or uncovering bugs.

• Implementing a Custom Check
  • Setting up the Development Environment
• Writing a custom check
• Sample Custom Check
  • Code Artifacts in Custom Checks
  • Creating findings
• Testing Your Custom Check
• Deploying your Custom Check
• Custom Check API Evolution

Implementing a Custom Check

Setting up the Development Environment

It is recommended to use the Eclipse development environment to implement Custom Checks. To set up a development workspace, please perform the following steps:

1. Download an up-to-date version of the Eclipse IDE and unzip it.

2. Launch Eclipse and create a new workspace.

3. Download the Custom Check Example project from the following URL, create an Eclipse project as described there and import it into your Eclipse workspace.
   https://github.com/cqse/teamscale-custom-check-sample

Writing a custom check

To implement a custom check, create a new Java class deriving from CheckImplementationBase that is annotated with @Check . The class CheckImplementationBase contains the analysis context (ICheckContext) which you can use to get information about the analyzed source code file. This includes the textual representation of the code, the stream of tokens in the file (the output of the scanner/lexer), as well as the abstract syntax tree (AST, output of the parser). The only method your Custom Check needs to implement (overwrite) is execute().

If your check should be based on tokens, you can use methods from TokenStreamUtils to focus on specific tokens.

If your check requires the more structured AST, you can use methods from ShallowEntityTraversalUtils to filter specific AST entities.

In the @Check annotation you must specify meta-data about your check. This meta-data is used by Teamscale mainly for configuration of the analyses (quality profile) and the presentation of the findings your check will produce. The meta-data includes the following information:

• name: Name of the check

• description: A description of the check that explains developers why the findings of the check are a problem and how they should fix them. You can omit this meta-data in the annotation, and provide a check name.md file in the /check-descriptions folder on the class path instead. This is particularly useful if the check description is long. In case the check name contains characters that are not valid in Windows file names, they have to be substituted with dashes.

• groupName: The analysis group the check should be contained in (e.g. General Checks)

• languages: The programming language(s) for which the check can be used.

Furthermore, a Custom Check must specify on which representation of the code it performs the analyses (using the parameters-argument of Check). E.g., the check may only access the AST if the option ECheckParameter.ABSTRACT_SYNTAX_TREE is specified or type information will only be present if type resolution is explicitly switched on by using ECheckParameter.TYPE_RESOLUTION. Teamscale needs this information to cluster the checks, so that representations that are not needed are not calculated at all.

Sample Custom Check

The following code listing shows a simple Custom Check:

@Check(name = "Abstract types should not have constructors (CA1012)", 
    groupName = "Language misuse", 
    languages = { ELanguage.CS }, 
    parameters = { ECheckParameter.ABSTRACT_SYNTAX_TREE })

public class AbstractTypesShouldNotHaveConstructorsCheck extends CheckImplementationBase {

	@Override
	public void execute() throws CheckException {
		List<ShallowEntity> rootEntities = context.getAbstractSyntaxTree(getCodeViewOption());
		List<ShallowEntity> types = ShallowEntityTraversalUtils.listEntitiesOfType(rootEntities,
				EShallowEntityType.TYPE);
		for (ShallowEntity type : types) {
			if (TokenStreamUtils.contains(type.ownStartTokens(), ETokenType.PUBLIC)
					&& TokenStreamUtils.contains(type.ownStartTokens(), ETokenType.ABSTRACT)) {
				analyzeMethodsInType(type);
			}
		}
	}
	
	/**
	 * Analyzes the top-level methods that are public constructors in the given
	 * type. Since constructors can't be nested in other methods, we don't need to
	 * look inside other methods.
	 */
	private void analyzeMethodsInType(ShallowEntity type) {
		List<ShallowEntity> topLevelMethods = ShallowEntityTraversalUtils
				.listMethodsNonRecursive(Collections.singletonList(type));
		for (ShallowEntity topLevelMethod : topLevelMethods) {
			if (topLevelMethod.getSubtype().equals(SubTypeNames.CONSTRUCTOR)
					&& TokenStreamUtils.contains(type.ownStartTokens(), ETokenType.PUBLIC)) {
				buildFinding("An `abstract` type should not have a `public` constructor",
						buildLocation().forEntity(topLevelMethod)).createAndStore();
			}
		}
	}
}

The description can be provided adding the file check-descriptions/Abstract types should not have constructors (CA1012).md to the classpath:

Constructors on abstract types can be called only by derived types.
Because public constructors create instances of a type, and you cannot create instances of an abstract type, an abstract type that has a public constructor is incorrectly designed.

Code Artifacts in Custom Checks

Text

Some custom checks are do not require deep information on the code but can be based simply on the file text. For example a "code does not contain emojis" check would just take the file text and check whether any char in the text is an emoji unicode char. In your execute() method, you can obtain the text with context.getTextContent(ETextViewOption.FILTERED_CONTENT).

Tokens

The next level of structuring is tokens. Teamscale scans the code and generates tokens (e.g., for keywords and identifiers in the code). For languages like C++, we also run a C preprocessor. You can obtain the tokens with context.getTokens(ECodeViewOption.FILTERED_PREPROCESSED).

The most important information in tokens is their type (getType(), for example IDENTIFIER), their text (getText()), and their position in the code (line number, char offset). Common methods for handling tokens are implemented in TokenStreamUtils and TokenStreamTextUtils.

Abstract Syntax Tree ("Shallow Entities")

The AST consists of so-called shallow entities. These entities are the nodes of the AST (e.g., types or methods). They are called shallow because the AST is only shallow in the sense that it is constructed down to the statement level but does not have detailed information on expressions within statements. You can obtain the shallow entities with context.getAbstractSyntaxTree(ECodeViewOption.FILTERED_PREPROCESSED).

The most important information in a shallow entity are

• its type (getType(), for example METHOD)
• its subtype (getSubtype, for example "CONSTRUCTOR")
• its children entities (getChildren())
• and tokens (e.g., ownStartTokens()).

Common methods for selecting specific shallow entities to be analyzed are implemented in ShallowEntityTraversalUtils.

Creating findings

If your custom check found a code pattern that should be marked in the UI, you have to create a finding. A finding needs at least a message and a location (file path and char offset). The buildFinding() method in CheckImplementationBase implements a builder pattern that makes sure all required information is given. For example, if your check identifies that a token myToken needs to get a finding, then you could call it like this:

buildFinding("Token is bad", buildLocation().forToken(myToken)).createAndStore();

Testing Your Custom Check

The Teamscale Custom Check Framework provides a Unit-Testing framework you should use to write tests for your checks. The easiest way to test your check is using data-driven tests. This way you do not have to write any test code but only provide test data in terms of code listings that contain the findings your check should identify and a file describing the expected results. To define a data-driven test you need to perform the following steps:

1. Create a new subfolder within the src/test/resources folder using the class name of your custom check class as the folder name (e.g., MyCheck)

2. Place a test code file (e.g., MyTestClass.java) to test your check against in the newly created folder.

3. Create a second file in the same folder and use the same name followed by .expected, e.g., MyTestClass.java.expected. This file will be used to compare the findings detected by your check with the expected findings. For every finding your check should emit, put a line into this file using the following format: Offsets <fromOffset>-<toOffset> (lines <fromLine>-<toLine>): <findingMessage> The offsets here are character offsets in the source code. Please specify the following information corresponding to your src/test/resources file: fromOffset/toOffset: Offsets to the tokens the finding should be annotated. fromLine/toLine: The line number that should be marked with a finding of your check. findingMessage: The message that should be displayed for your finding.

4. To be able to execute the tests, you need to create a single test class deriving from CheckTestBase that will automatically create a test suite from the test data (this class is already available in the src/test/java folder of Custom Check Example project):

   /** Main class for testing checks */
   @RunWith(Parameterized.class)
   public class CheckTest extends CheckTestBase {
   
     /** Constructor. */
     public CheckTest(File referenceFile, Map<String, CheckInfo> checkInfoBySimpleClassName) {
       super(referenceFile, checkInfoBySimpleClassName);
     }
   
     /** Generate Test Parameters. */
     @Parameters(name = "{0}")
     public static Collection<Object[]> generateParameters() throws IOException {
       return CheckTestBase.generateParameters(new CheckTest(null, null));
     }
   }

5. To run the tests, launch the test class as a JUnit-Test in Eclipse and inspect the results (information about the actual results of your check are written to the directory test-tmp).

Deploying your Custom Check

1. Package your custom check into a JAR file using the JAR Export Wizard of Eclipse. You do not need to re-package any of the included libraries. The check classes are sufficient.
2. Place the JAR in the custom-checks subfolder of your Teamscale installation. The location of this folder can be customized in teamscale.properties.
3. Restart Teamscale. The checks become available when configuring analysis profiles.

Custom Check API Evolution

The custom check API and thus any custom check binaries are compatible between patch releases (e.g., 5.6.0 and 5.6.1). However, when updating to a new feature release (e.g., 5.6.x to 5.7.x) a rebuild of your custom checks against the latest customer check API is recommended.