Upgrade Testing

Overview

When a user upgrades your app from an older version to a newer one, Android replaces the installed APK but keeps existing data intact — login sessions, preferences, subscribed content, stored credentials, and cached state. This is normal Android upgrade behaviour.

The challenge for QA is verifying that this data survives the upgrade correctly. You cannot run your test suite on a fresh install of the new build because the device starts with no pre-existing state. You need to:

Set up real user state on the old build — subscribe to a plan, add items to a cart, configure preferences, etc.
Upgrade the app to the new build without wiping that data
Run your tests on the new build to confirm data and flows still work correctly

RobusTest's upgrade testing feature enables exactly this. Using shell hooks in your job payload, you can run setup tests on the old build and install the new build — all as part of the automated job lifecycle.

How It Works

[1] Device is prepared
      └─ RobusTest installs your primary AUT build (the OLD version)

[2] afterInstancePrepShellCommands fires
      ├─ Command 1: am instrument  →  runs setup tests on the OLD build
      │                               (creates accounts, sets up state)
      └─ Command 2: install -r     →  installs the NEW build
                                      -r replaces the APK but PRESERVES all app data

[3] Normal test execution begins
      └─ Your test suite runs on the NEW build
         All data from step [2] is still present
         Tests verify data survived the upgrade and flows still work

The critical element is the -r flag in the install command. It tells Android to replace the application binary while leaving everything in the app's data directory untouched.

What Is an Instance?

An instance is the device preparation lifecycle for a single device in a job. When RobusTest allocates a device, it goes through a series of steps — installing the app, configuring the device, and preparing it for test execution. This entire sequence is one instance.

You can view each instance and all its steps — including shell hook command output — in the Job Report → Device Sessions section.

Shell Hooks

Shell hooks are arrays of commands placed inside the framework section of your job payload. Each hook fires at a specific point in the lifecycle, and every command in the array runs on the device in order.

Hook	When It Fires
`beforeJobShellCommands`	Once, before any device starts preparing. Use for job-level setup.
`beforeInstancePrepShellCommands`	Before each device's preparation attempt begins.
`afterInstancePrepShellCommands`	After each device is prepared and the primary build is installed. This is the hook used for upgrade testing.
`afterJobShellCommands`	Once, after all devices have finished. Use for cleanup or reporting.
`beforeTestcaseShellCommands`	Before every individual test case runs. Commonly used to reset device state between tests.
`afterTestcaseShellCommands`	After every individual test case runs.

Each hook has a paired *StopOnFailure boolean. When true, if any command fails, all remaining commands in that hook are skipped. When false (the default), execution continues regardless of failures.

Supported Commands

`install` — Install a Build from RobusTest

install <flags> <buildID>

Downloads and installs a specific build stored on the RobusTest server.

Flag	What It Does
`-r`	Replace the existing app while preserving all app data. SharedPreferences, databases, files, and account data all survive. This is the critical flag for upgrade testing — without it, the install wipes all state.
`-g`	Automatically grant all runtime permissions. Prevents permission dialogs from blocking test execution.
`-t`	Allow test APK packages to be installed. Required when installing test builds.
`-d`	Allow version downgrade. Needed if the new build has a lower `versionCode` than the installed one.

Example:

install -g -r -t <buildID>

Always use -r when installing the upgrade build. Without it, the install is a clean install and all state created by your setup tests is lost.

`am instrument` — Run Tests Directly on the Device

This Android command runs Espresso tests using Android's instrumentation framework. In upgrade testing, it is used inside the hook to run setup tests on the old build before the upgrade is applied.

Run an entire class:

am instrument -w -r -e class <testclass> <testapppackage>/<testrunner>

Run a specific method:

am instrument -w -r -e class <testclass>#<testcase> <testapppackage>/<testrunner>

Part	What It Does
`-w`	Wait. Blocks until all tests finish before moving to the next command. Without this, the `install` command runs before setup tests have completed.
`-r`	Raw output. Prints results in a format RobusTest can parse. Without this, pass/fail results cannot be read.
`-e class <value>`	Runs the specified class. Append `#methodName` to target a single method.
`<testapppackage>`	Package name of the test APK — not the app's own package name.
`/<testrunner>`	Runner class inside the test APK, separated from the package with `/`.

Targeting options for -e:

Filter	What Runs
`-e class com.example.Class1`	All methods in the class
`-e class com.example.Class1#testCase1`	A single method
`-e class com.example.Class1#testCase1,com.example.Class1#testCase2`	Multiple methods (order not guaranteed)
`-e annotation com.example.annotations.Annotation1`	All tests tagged with that annotation
`-e annotation <a1> -e annotation <a2>`	Tests that have ALL listed annotations (AND logic)
`-e notAnnotation com.example.annotations.Annotation1`	All tests that do NOT have that annotation

Upgrade Testing Patterns

All patterns use afterInstancePrepShellCommands. The shape is always the same: run setup tests on the old build, then install the new build.

Note: The am instrument command should include all job-level parameters as -e key value pairs so setup tests have access to the same environment, tokens, and feature flags as the main test run.

Pattern 1 — Run an Entire Setup Class

Use when all setup logic is in one class and every method should run. Order of methods is not guaranteed.

"framework": {
    "afterInstancePrepShellCommands": [
        "am instrument -w -r -e class com.example.suites.SetupClass com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "install -g -r -t <buildID>"
    ],
    "afterInstancePrepShellCommandsStopOnFailure": true,
    "className": "com.example.suites.MainTestSuite",
    "packageName": "com.example.app",
    "ignoreClasses": "com.example.suites.SetupClass"
}

ignoreClasses ensures the setup class does not run again during normal execution.

Pattern 2 — Run a Specific Method

Use when only one method needs to run as setup.

"framework": {
    "afterInstancePrepShellCommands": [
        "am instrument -w -r -e class com.example.suites.SetupClass#setupMethod com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "install -g -r -t <buildID>"
    ],
    "afterInstancePrepShellCommandsStopOnFailure": true,
    "className": "com.example.suites.MainTestSuite",
    "packageName": "com.example.app",
    "ignoreTestcases": "com.example.suites.SetupClass#setupMethod"
}

Pattern 3 — Multiple Methods in a Guaranteed Order (Recommended)

Use when multiple methods must run and order matters — for example, one method must complete before the next can run meaningfully. Each method gets its own am instrument command.

"framework": {
    "afterInstancePrepShellCommands": [
        "am instrument -w -r -e class com.example.suites.SetupClass#setupMethod1 com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "am instrument -w -r -e class com.example.suites.SetupClass#setupMethod2 com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "install -g -r -t <buildID>"
    ],
    "afterInstancePrepShellCommandsStopOnFailure": true,
    "className": "com.example.suites.MainTestSuite",
    "packageName": "com.example.app",
    "ignoreTestcases": "com.example.suites.SetupClass#setupMethod1,com.example.suites.SetupClass#setupMethod2"
}

With stopOnFailure: true, if setupMethod1 fails, setupMethod2 and the install are skipped — there is no point continuing setup if an earlier step did not complete.

Pattern 4 — Multiple Methods, Order Not Important

Use when multiple methods need to run but order does not matter. All methods are passed as a comma-separated list to a single am instrument call.

"framework": {
    "afterInstancePrepShellCommands": [
        "am instrument -w -r -e class com.example.suites.SetupClass#setupMethod1,com.example.suites.SetupClass#setupMethod2 com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "install -g -r -t <buildID>"
    ],
    "afterInstancePrepShellCommandsStopOnFailure": true,
    "className": "com.example.suites.MainTestSuite",
    "packageName": "com.example.app",
    "ignoreTestcases": "com.example.suites.SetupClass#setupMethod1,com.example.suites.SetupClass#setupMethod2"
}

Use Pattern 3 if order matters. JUnit decides execution order when methods are comma-separated in a single command.

Pattern 5 — Run Setup Tests by Annotation

Use when setup tests are spread across multiple classes but share a common annotation. Avoids having to list every class or method explicitly.

"framework": {
    "afterInstancePrepShellCommands": [
        "am instrument -w -r -e annotation com.example.annotations.SetupAnnotation com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "install -g -r -t <buildID>"
    ],
    "afterInstancePrepShellCommandsStopOnFailure": true,
    "className": "com.example.suites.MainTestSuite",
    "packageName": "com.example.app",
    "ignoreAnnotations": ["SetupAnnotation"]
}

Pattern 6 — Run Setup Tests Matching All of Multiple Annotations (AND Logic)

Use when setup tests must match more than one annotation.

"framework": {
    "afterInstancePrepShellCommands": [
        "am instrument -w -r -e annotation com.example.annotations.Setup -e annotation com.example.annotations.Critical com.example.test/androidx.test.runner.AndroidJUnitRunner",
        "install -g -r -t <buildID>"
    ],
    "afterInstancePrepShellCommandsStopOnFailure": true,
    "className": "com.example.suites.MainTestSuite",
    "packageName": "com.example.app",
    "ignoreAnnotations": ["Setup"]
}

Preventing Setup Tests from Running Twice

When am instrument runs setup tests in the hook, those same methods are still in the test APK. During normal execution, RobusTest will discover and run them again unless you explicitly exclude them. Always pair your hook with the matching filter.

How You Ran Setup	Filter to Use
By class	`"ignoreClasses": "com.example.suites.SetupClass"`
By method	`"ignoreTestcases": "com.example.suites.SetupClass#setupMethod"`
By annotation	`"ignoreAnnotations": ["SetupAnnotation"]`

Alternative — runOnlyAnnotations

Instead of excluding setup tests, restrict normal execution to only tests with certain annotations. Any test without a listed annotation is automatically excluded.

"runOnlyAnnotations": ["Regression", "Smoke"]

How ignore and runOnly interact:

Ignore always takes priority over runOnly, regardless of order.

Configuration	Result
`runOnlyClasses: "TestClass1,TestClass2"` + `ignoreClasses: "TestClass2"`	Only TestClass1 runs
`runOnlyClasses: "TestClass1"` + `ignoreTestcases: "TestClass1#testCase1"`	All methods in TestClass1 run except testCase1
`runOnlyAnnotations: ["Annotation1"]` + `ignoreAnnotations: ["Annotation2"]`	Tests with @Annotation1 run, but any that also have @Annotation2 are excluded

Viewing Results

Every shell hook command appears as a numbered step in the device's instance log with the command, full output, duration, and pass/fail status.

After Instance Prep Shell Commands
  ├── #1  am instrument -w -r -e class com.example.suites.SetupClass ...   ✓  18s
  └── #2  install -g -r -t <buildID>                                       ✓  31s

Navigate to Job Report → Device Sessions to view the instance log for each device.

Troubleshooting

Problem	Why It Happens	What To Do
Setup tests run a second time during normal execution	Setup tests not excluded from the normal run	Add `ignoreClasses`, `ignoreTestcases`, or `ignoreAnnotations` to match how you targeted them in the hook
App data is wiped after the upgrade install	`-r` flag missing from the `install` command	Add `-r`: `install -g -r -t <buildID>`
Install step is skipped when setup tests fail	`stopOnFailure: true` aborting the hook early	Set to `false` if the install should proceed regardless, or fix the failing setup tests
`INSTRUMENTATION_FAILED` in step output	Class name, runner, or package name does not match what is in the test APK	Verify the full class name, test APK package name, and runner against the test APK manifest
Methods run in unexpected order	Comma-separated methods in a single `am instrument` call	Use one `am instrument` command per method (Pattern 3)
Hook runs more than once on the same device	Device preparation failed and was retried	Expected behaviour — each prepare attempt runs the hook independently

Espresso Hub - Espresso Hub overview and job execution workflow
Hub Job Payload - Full job creation API reference
Run Settings - Configuring timeouts and retry behaviour
Reading Job Reports - Viewing device session logs and test results