Setting Up Test Coverage Profiling for Android Apps

This guide shows you how to adapt your Android mobile application to collect coverage to enable Test Gap Analysis in Teamscale. We describe the setup for a Kotlin project with a Gradle build system. The instructions can vary depending on the specific technologies in your setup. The coverage can be collected with an emulated device or a physical smartphone.

• Adapting the build.gradle.kts to Add JaCoCo
• Setting up JaCoCo in Your App
• Changing the AndroidManifest.xml
• Prerequisites for Pulling and Converting Coverage
• Generating, Pulling, and Converting Coverage
• Uploading Coverage
• Troubleshooting

Adapting the build.gradle.kts to Add JaCoCo

First, you have to enable recording code coverage in the file build.gradle.kts by adding JaCoCo to the project. Remember to check for the current version of JaCoCo and to adapt it accordingly.

android {
    compileSdk = libs.versions.compileSdk.get().toInt()
    namespace = "<your.apps.namespace>" // adapt to your app's namespace

    // Add these two lines:
    testCoverage {
        jacocoVersion = "0.8.15" // adapt to current version
    }
    ...
    }

Also, adapt the buildTypes and enable coverage for Android and Unit tests.

buildTypes {
    getByName("debug") {
        enableAndroidTestCoverage = true
        enableUnitTestCoverage = true
    }

We also recommend generating a git.properties file to later allow mapping the recorded coverage to the right build version of the software.

Setting up JaCoCo in Your App

Create two new files in the folder app/src/main/java/. The first file is TestCoverageUtils.kt. It allows fetching coverage data from the active JaCoCo agent on the Android device. The coverage is saved as numbered .exec file inside a coverage directory:

package <your package name>

import android.os.Environment
import android.util.Log
import java.io.File

object TestCoverageUtils {

    fun generateCoverageReport() {
        val jacocoTag = "jacoco"
        Log.d("StorageSt", Environment.getExternalStorageState())
        
        // Define a base directory and folder for coverage files
        val baseDir = Environment.getExternalStorageDirectory().toString()
        val coverageFolder = File(baseDir, "coverage")
        if (!coverageFolder.exists()) {
            val directoryCreated = coverageFolder.mkdirs()
            Log.d(jacocoTag, "Creating directory $coverageFolder succeeded: $directoryCreated")
        }
        
        val coverageFile = getNextCoverageFile(coverageFolder)
        val coverageFilePath = coverageFile.absolutePath
        try {
            Log.d(jacocoTag, "Generating coverage report to $coverageFilePath")
            
            // Use reflection to call JaCoCo agent to dump coverage data
            val agent = Class.forName("org.jacoco.agent.rt.RT")
                .getMethod("getAgent")
                .invoke(null)
            
            val getExecutionDataMethod = agent.javaClass.getMethod("getExecutionData", Boolean::class.javaPrimitiveType)
            val executionData = getExecutionDataMethod.invoke(agent, false) as ByteArray
            
            coverageFile.writeBytes(executionData)
            Log.d(jacocoTag, "generateCoverageReport: SUCCESS")
        } catch (e: Exception) {
            Log.e(jacocoTag, "generateCoverageReport: FAILED", e)
            throw RuntimeException("JaCoCo agent not found or failed to dump data", e)
        }
    }

    private fun getNextCoverageFile(dirPath: String): File {
        var file = File(dirPath, "coverage.exec")
        var index = 1

        // prevent that existing coverage files are overwritten
        while (file.exists()) {
            file = File(dirPath, "coverage_$index.exec") 
            index++
        }
        return file
    }
}

The second file CoverageDumpLifecycleApplication.kt creates a listener that generates the coverage report when the app is hidden, minimized or closed:

package <your package name> // adapt to your app's package

import android.app.Activity
import android.app.Application
import android.os.Bundle
import android.util.Log
import <your package name>.TestCoverageUtils.generateCoverageReport

class CoverageDumpLifecycleApplication: Application() {
    override fun onCreate() {
        super.onCreate()
        Log.d("CoverageDumpLifecycleApplication is running", "onCreate")
        registerActivityLifecycleCallbacks(object : ActivityLifecycleCallbacks {
            override fun onActivityCreated(activity: Activity, savedInstanceState: Bundle?) {
            }

            override fun onActivityStarted(activity: Activity) {
            }

            override fun onActivityResumed(activity: Activity) {
            }

            override fun onActivityPaused(activity: Activity) {
            }

            override fun onActivityStopped(activity: Activity) {
                Log.d("CoverageDumpLifecycleApplication has been ended", "onActivityStopped in $activity")
                generateCoverageReport()
            }

            override fun onActivitySaveInstanceState(activity: Activity, outState: Bundle) {
            }

            override fun onActivityDestroyed(activity: Activity) {
            }
        })
    }
}

Changing the AndroidManifest.xml

You have to adapt the AndroidManifest.xml to grant additional storage permissions. Add the following lines to the <manifest> tag:

<manifest 
    xmlns:tools="http://schemas.android.com/tools">
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.MANAGE_EXTERNAL_STORAGE"
        tools:ignore="ScopedStorage" />

In the next step, the previously created file CoverageDumpLifecycleApplication.kt has to be registered within the AndroidManifest.xml. Add android:name=".CoverageDumpLifecycleApplication" to the application tag:

<application
        android:name=".CoverageDumpLifecycleApplication"

Prerequisites for Pulling and Converting Coverage

You have to register two new gradle tasks in build.gradle.kts to:

• Conveniently pull the coverage created on the device.
• Convert the raw .exec coverage files to an .xml format so Teamscale can process it.

tasks.register<Exec>("pullCoverageData") {
    group = "Reporting"
    description = "Pulls all coverage files from the device."
    val outputDir = layout.buildDirectory.dir("outputs/code_coverage")
    
    doFirst {
        outputDir.get().asFile.mkdirs()
    }
    
    // Ensure the right path here
    commandLine("adb", "pull", "/sdcard/coverage/.", outputDir.get().asFile.absolutePath)
    isIgnoreExitValue = true
}

ADB and Path Setup

This requires adb to be on the path. Also, the coverage directory can be in /sdcard/ or /storage/emulated/0/Download/. Please make sure that this is the right path.

The coverage report conversion from .exec to .xml is performed with the following task:

tasks.register<JacocoReport>("testCoverageReport") {
    group = "Reporting"
    description = "Pulls coverage data from device and generates Jacoco report."
    dependsOn("pullCoverageData")

    reports {
        xml.required.set(true)
    }

    val buildDir = layout.buildDirectory.get().asFile
    
    // Support both Kotlin and Java classes, excluding generated code
    val kotlinClasses = fileTree("$buildDir/tmp/kotlin-classes/debug") {
        exclude("**/R.class", "**/R$*.class", "**/BuildConfig*.class", "**/Manifest*.class", "**/*Test*.class", "**/android//.*")
    }
    val javaClasses = fileTree("$buildDir/intermediates/javac/debug/classes") {
        exclude("**/R.class", "**/R$*.class", "**/BuildConfig*.class", "**/Manifest*.class", "**/*Test*.class", "**/android//.*")
    }
    
    classDirectories.setFrom(files(kotlinClasses, javaClasses))
   // Please check if this is the right place for the source files
    sourceDirectories.setFrom(files("$projectDir/src/main/java"))
    executionData.setFrom(fileTree("$buildDir/outputs/code_coverage") {
        include("**/*.exec")
    })
}

Coverage conversion requires .class files!

The .exec coverage files can only be converted when the .class files from the build are available. They cannot be converted on the Android device. To generate a readable coverage report, you have to either:

• Run the conversion in an environment where the compiled .class files are available.
• In a CI/CD Pipeline: Ensure that the .exec file is moved to an environment where the .class files of the corresponding build version are available.

Generating, Pulling, and Converting Coverage

To create coverage when running the application, you need to use the debug build via this command:

.\gradlew installDebug

You can now switch to the emulated or physical mobile phone to grant additional access rights to your app.

1. Open Settings on your emulated or physical phone.
2. Go to Apps (or Apps & Notifications).
3. Scroll to the very bottom and tap Special app access.
4. Search for All files access.
5. Find your app in that list, tap it, and toggle the switch to Allow access to manage all files.

App missing from the list?

If your app is not listed, the AndroidManifest.xml in the project probably doesn't have the MANAGE_EXTERNAL_STORAGE permission declared. You might need to add permission to the manifest and re-run the app.

You can now open your app and perform tests. When minimizing the app, coverage is written to the internal storage of the emulator or physical phone. The files are named coverage[index].exec and are either located in /sdcard/coverage/ or /storage/emulated/0/Download/coverage/.

To extract the coverage, you have to pull it from the device and then convert it into .xml format with the following Gradle command:

.\gradlew testCoverageReport

In a production setting, this step should be automated to fetch and convert coverage data after each generation.

Uploading Coverage

Teamscale offers different possibilities to upload external analysis data. For Android coverage, we suggest using the Teamscale Upload tool or the Teamscale Gradle Plugin.

Troubleshooting

Changes in build.gradle.kts have no effect

If your changes in the file build.gradle.kts have no effect, remember to sync it.

Coverage cannot be located

For detailed logs, open logcat and search for "jacoco" to discover the path where the coverage is located.