Automating multi-app functional testing against different versions of Android

Automated testing is mostly about plumbing. Don’t you think?

Context

Ghera is a repository of benchmarks that embody known vulnerabilities in Android apps. Each benchmark is composed of two apps: a benign app that exhibits a vulnerability and a malicious app that exploits the vulnerability. To help with documentation, we manually test each benchmark against different versions of Android, i.e., API versions 19 thru 25. So, for each benchmark, we perform 6 functional tests, i.e., benign-malicious combination on 6 different versions of Android.

Problem

Based on a recent feedback at Promise’17, we decided to extend each benchmark with a secure app that does not exhibit the vulnerability. Consequently, we have to manually execute 240 functional tests as we currently have 40 benchmarks. Instead, how about automating this testing?

Solution

For each benchmark, for each selected API version,

1. Fire up an emulator,
2. Install API specific APK of the benign (or secure) app,
3. Install API specific APK of the malicious app,
4. Drive the malicious apps via UI automation, and
5. Check if the observed behavior matches the expected behavior.

Here’s how I realized the above solution.

Step 1: Generate different APKs for each Android version

As is, each benchmark contains two projects: Benign and Malicious that each house an app with ids edu.ksu.cs.benign and edu.ksu.cs.malicious, respectively. So, I extended the Gradle build files of the apps by injecting sdkFlavors.gradle (see the code listing below) into each <project>/app/build.gradle files (see line 35 in build.gradle below). With these simple changes, I could use ./gradlew assembleDebug to generate Android version specific APKs.

Step 2: Create instrumented multi-app test

Since I wanted to test how edu.ksu.cs.benign app behaves when edu.ksu.cs.malicious app interacts with it, I created a project named Testing with an app module named benign_app. The purpose of benign_app was to house the instrumented tests.

Unfortunately, Android app development tools does not admit an app module without an app manifest file. So, I created an empty app manifest file in benign_app module and used edu.ksu.cs.benign as the package name in the manifest; the choice of this package name does not matter.

Next, I created a test class in Testing project to contain the instrumented UI tests. (Please refer to this Android documentation link about writing such tests.) When the tests are exercised via gradle, the test harness considers the app with the app id mentioned in the gradle build file (e.g., applicationIdon line 29 in build.gradle listed below) as the (to be) tested app. Consequently,the test harness tries to find the named app on the emulator to execute the tests against it. So, make sure the app id in gradle build file is identical to that of an app installed on the emulator/device.

In my case, this was not possible as the tests were not in the same module as tested app. Even if the tests were moved to app module in Benign project, the tests depended on edu.ksu.cs.malicious app and there was no (obvious) way to configure the test harness (via gradle) to install edu.ksu.cs.malicious app before executing the tests. This is also the reason for not being able to use gradle’s connectedAndroidTest task to run these tests.

Step 3: Orchestrate testing outside of the build system

As a solution, I crafted the following build and test orchestration shell script in Bash.

The functional-test.sh script automates the

• building the apps (lines 12–23),
• testing benign app (lines 46–63), and
• testing secure app (lines 65–78).

while assuming

• AVD names conform to a pre-defined naming scheme,
• Testing project has secure_app module to test the secure app, and
• applicationId of secure_app module/app is edu.ksu.cs.benign as this helps reuse edu.ksu.cs.malicious app without any changes.

With all of this, once we have functional tests for more benchmarks, executing them requires only few key strokes :) The only downside of this solution is the lack of pretty test reports :( Really, whom am I kidding?

The corresponding source code can be found here.

What would have been ideal?

1. Using gradle to run functional tests.
2. Modify connectedAndroidTest task (including all its product flavor specific variants) to install and uninstall APKs.
3. Having connectedAndroidTest task start and stop virtual devices.
4. Selecting and configuring the virtual device to start in each product flavor specific variant of connectedAndroidTest task.
5. Creating a instrumentation test-only Android project that does not require an app manifest and specific app ids.

Closing thoughts

Given this was my first brush with Android app development, I am impressed with Android Studio and the available tooling.

If my observations about Android plugin for Gradle are true, then I hope Android team considers and addresses them in future releases :)

All of this was done in about a day. I could have missed obvious existing solutions or approaches to do multi-app functional testing. If this is the case, then please provide pointers to existing solutions or alternative approaches (via comments below).

Cheers,