Performance Benchmarks

A performance benchmark is a set of App Vitals values that you decide a good build should stay within. After every job that has App Vitals turned on, RobusTest compares the new build with the benchmark and gives the job a clear result: Pass, Fail, or Not compared.

Use it to catch performance regressions — more memory, more CPU, slower start, more janky frames, crashes — before a build ships, without reading every App Vitals report by hand.

In short: you create a benchmark once from good runs → every new job with App Vitals is compared with it → the result shows on the job report and in the jobs list.

Before you start

How it works

1. Create a benchmark for the project. For each test case, set the value each metric should be compared with (for example, "memory used: 180 MB").
2. Add checks. A check says how much worse than the benchmark a build may be before it fails (for example, "fail if memory is more than 10 % higher").
3. Set it as the project default. Creating a benchmark is not enough on its own: jobs are compared with the project's default benchmark, so mark yours as the default. (Optional: to use a different benchmark for one job, add its ID to the Run Setting, see below.)
4. Run a job with App Vitals on. When the job finishes, RobusTest compares each test case with the benchmark.
5. Read the result on the job report, in the jobs list, or on the Job performance page.

A job is compared with the project's default benchmark. If the job's Run Setting also names a benchmark, the one in the Run Setting is used instead.

Important — set a default benchmark. Creating a benchmark does not make it the default by itself. If the project has no default benchmark and the job's Run Setting doesn't name one, no comparison happens: the job shows Not compared ("no benchmark"). Tick Make this the project default when you create the benchmark, or click Set as default on it later.

Opening the benchmarks

On the Project Dashboard of an Android App or iOS App project, click Performance benchmarks (next to Traffic Modifier). This opens the benchmark list for the project.

The benchmark list

The list shows every benchmark in the project.

Admins can Create benchmark (button at the top), open a benchmark to Edit it, or click Set as default. (There is no delete button on these pages — see the note below.) If the project has none yet, the page says No benchmarks yet and job results show Not compared until one exists.

The line at the bottom of the page is a reminder: Every change saves a new version; jobs record the version they were compared with. Crashes, ANRs and hangs always fail a test and are not counted under Checks. The count at the top right (5 benchmarks) shows how many the project has, and Search benchmarks filters the list by name.

The Checks column summarises each benchmark's checks, for example 5 checks · all Fail. Benchmarks saved earlier may also show Warning only checks; new benchmarks only have Fail checks.

Creating a benchmark

Click + Create benchmark. The page has three tabs — Details, Test cases and benchmark values, and Checks. A red number on a tab shows how many things are still missing there. The Create benchmark button stays disabled until everything is filled in, and the first missing item is shown in red next to it (for example, Name the benchmark to save, Add at least one test case to save, Add at least one check to save). Cancel leaves without saving.

Tab 1 — Details

When you are done, click Next: Test cases and benchmark values at the bottom of the page.

Tab 2 — Test cases and benchmark values

This tab holds the test cases the benchmark covers and the value each metric is compared with. Until you add one, it says No test cases yet. The top of the tab shows + Add test cases, Calculate values from a job… (available once at least one test case is added), and a count such as 0 test cases on the right.

Adding test cases

Click + Add test cases. A pop-up lists every test case in the project.

Test cases are matched by class and test name (on iOS, also the test plan configuration). If you rename a test, it counts as a new test and will not be compared until you add it.

Setting the values

Each test case becomes a row with one column per metric. Set each value in one of three ways:

Calculate works only for test cases that have been run with App Vitals on. In the screenshot above:

• The first two test cases say No App Vitals data yet. Their calculate links are greyed out and struck through, so you can't calculate from runs. Type the values instead (or run the test with App Vitals on, then come back).
• The third test case has App Vitals data, so its calculate links are active (blue). It says No source attempts · Add because you haven't yet chosen which runs to use. Click calculate under any value, or Add, to open Attempts for … and pick the runs. Once you pick runs, every empty value of that test case is filled from them.

Calculated values, and changing them by typing

Once you pick the runs for a test case, the values are calculated for you — you don't have to work them out. Each value is the median of the runs you chose.

In the screenshot, the third test case now says 5 source attempts · Edit, and four of its values are filled in. Under each value a small label shows where it came from:

To change a calculated value: click type under it. The number becomes a box you can edit, and it keeps the calculated number as a starting point. Type the value you want, for example a stricter target. The label changes to typed, as with Janky frames in the screenshot. Use this when you know the number a build must meet, or when a calculated value looks too low or too high.

To undo: click calculate under a typed value to use the calculated value again, or click the × in the box to clear it.

A typed value is kept when you later change the source attempts or use Calculate values from a job… — only calculated values are refreshed.

Other options on each row:

Choosing the runs: Attempts for …

What it is for. A calculated benchmark value comes from real runs of the test case. Attempts for … is where you choose which runs. Every time a test case ran with App Vitals on (in any job of the project) is one attempt. Tick the attempts that represent a healthy build; RobusTest takes the median of them as the benchmark value. Choosing runs by hand lets you leave out a run you don't trust, for example one from a different device or a different build.

To open it, click Add under a test case that says No source attempts, click Attempts, or click calculate under a value.

When you open the pop-up for a test case that has no source attempts yet, RobusTest ticks the usual attempts that match the first build and device for you. Review the ticks and change them if needed.

Below the table, the pop-up previews the value each metric would get from the ticked attempts, with the lowest–highest range. If you have ticked fewer than 3 attempts it reminds you that 3 or more give a steadier value. A value you typed yourself is kept — it is not replaced.

Two ways to calculate values

You can calculate values from real runs in two ways. Use whichever suits you — you can mix them:

• From specific attempts, one test case at a time. Open Attempts on a test case and tick the runs you trust. Best when you want to choose runs by hand (for example, leave out a run on a different device).
• From one job, for many test cases at once. Click Calculate values from a job… at the top of the tab. Best when one job is your "good build" and you want to fill every test case from it in one go.

In the pop-up, type a job name or number in Find a job to narrow the list, then choose the job in Job (only jobs with App Vitals data are listed, each with its date and number of attempts). The pop-up then lists each test case in the benchmark with what will happen — values calculated from N attempts, Not in this job · unchanged, or Every attempt is unusual · unchanged. Values you typed are kept. Only test cases already in the benchmark are filled; use + Add test cases to add more. Click Calculate for N test cases to apply.

The Job drop-down lists only jobs of this project that have App Vitals data. Each entry shows the job name, its number, the date it ran, and how many attempts have App Vitals data (for example, … · 23 Oct 2026 · 40 attempts), newest first. Pick the job that ran the build you want as your benchmark. If you can't find it, type part of its name or number in Find a job. If no job has App Vitals data yet, the pop-up says so — run a job with App Vitals on first.

When you are done, click Next: Checks.

The tab also shows how many test cases you have and how many values are still missing (for example, 2 test cases · 6 values missing).

Which metrics appear

The table has one column for each metric that has a check (see Checks). A new benchmark starts with five checks, so you see these five columns:

Add a check for another metric on the Checks tab and its column appears here. All the metrics a benchmark can hold:

Janky frames and cold start are not captured on iOS yet, so iOS benchmarks do not offer them. Values can't be negative, and a value in % can't be over 100.

When you pick runs, RobusTest shows warnings so you can judge how reliable the values are:

• Few runs — the value is based on fewer than 3 runs.
• Runs differ — the picked runs were made on different phone models, RAM, chipsets, OS versions or builds.
• Low coverage — App Vitals could only read the app's CPU and memory for less than 80 % of the test.
• Unusual run — the test did not complete or pass, or the collector had a problem. Such runs are tagged unusual and are not ticked for you; tick one only if you are sure you want it.

Tab 3 — Checks

Values say what a good build looks like. Checks say how much worse a new build may be before it fails. They apply to every test case in the benchmark, and each metric can have one check.

A new benchmark starts with one row for each of the five usual metrics, with the limit left empty. You type every limit — RobusTest never fills one in for you. The red number on the Checks tab counts the limits still missing. A benchmark needs at least one check, and every check needs a limit. When all are filled in, Create benchmark becomes active (compare the earlier screenshot, where it was greyed out).

Compare with has two options:

Good to know:

• Percentage checks need a benchmark value. If a test case's value is empty (or 0), that metric is not compared for it. A fixed limit works even when the benchmark value is empty.
• Add more checks. Below the table, + Add check lists the metrics that don't have a check yet (for example App memory (95th percentile) or Threads growth), with a line metrics without a check: …. Pick one to add a row. Its column then appears on the Test cases and benchmark values tab.
• Crash or ANR is always checked. The grey, locked last row (Any, no value needed — Any crash or ANR — Can't be changed) means any crash or ANR fails the test, no matter what else you set. On iOS it reads Crash or hang.
• Checks only fail. There is no "warning only" level for new benchmarks.
• On the values tab, a ! next to a metric name means it has no check yet (its values are kept but not compared) or its check is a fixed limit (its benchmark values are ignored).

Click Create benchmark to save it as version 1.

Opening a saved benchmark

Click a benchmark's name in the list to open it. This page shows everything the benchmark holds, so you can check it before relying on it.

The Values table

Each test case is a group, with a heading such as 2 of 2 runs included. Under it is one row per metric:

Show runs (next to each test case) lists the runs behind its values with the job, build, device and date, and whether each run was Included or Left out.

A dash means no value. In the screenshot, Janky frames shows – and based on 0 runs, with the note too few frames to score: 53 frame(s) drawn, at least 60 needed. Janky frames needs at least 60 frames drawn in a run to give a reliable percentage. A test that barely draws anything on screen has no janky frames value, so a percentage check on janky frames is not compared for it (a fixed limit would still run).

Checks in this version and Version history

Scroll to the bottom of the page for the rules and the history.

Checks in this version lists the checks that apply to every test in the benchmark ("the same checks for every test"):

These are the checks as they were when this version was saved, so they may differ from a later version.

Version history lists every version of the benchmark, newest first:

The line at the very bottom (Benchmark … · calculation 2.2.8) is the benchmark's ID. Support or an API call may ask for it.

Editing a benchmark and versions

Open a benchmark from the list and click Edit. You must write a short note in What changed and why before you can save — this is how the team keeps track of why a benchmark moved.

Every save creates a new version (v1, v2, v3…). Older versions are never changed or lost. Open Version history on the benchmark page to see who saved each version, when, and the change note, and to open any earlier version.

This matters for results:

• A job is compared with the version that was current when the job started. Saving a new version later does not change the results of jobs that already started.
• Deleting a benchmark isn't available on the benchmark pages. A project admin can do it through the API (DELETE /v3/appvitals/benchmark/:id). A deleted benchmark is hidden from the list, but jobs that already used it still show their results.

Setting the default benchmark

Use Set as default on the list or on the benchmark page. A project has one default at a time; setting a new one replaces the old. A benchmark is never made the default automatically — not even the first one you create — so remember to set one. Without a default (and without a benchmark named in the Run Setting), jobs are not compared.

Choosing which benchmark a job uses

There is no benchmark picker in the Run Settings screen yet. To compare a job with a benchmark other than the default, add the benchmark's ID to the performance section of the Run Setting JSON (the ID is shown at the bottom of the benchmark page):

{
  "performance": {
    "enableAppVitals": true,
    "appVitalsBenchmark": "<benchmark id>"
  }
}

The Run Setting takes precedence over the project default. If the project has a default benchmark and the Run Setting also names one with appVitalsBenchmark, the job is compared with the benchmark named in the Run Setting, not the default. If you leave appVitalsBenchmark out, the project's default benchmark is used. The ID must belong to a benchmark of the same project that isn't deleted; otherwise the job shows Not compared with the reason.

Run Setting options that affect benchmarks

These options are in the Run Setting you use for the job. Set them before the job starts. To find them, open Run Settings from the Project Dashboard, open your Run Setting and go to the Performance section.

• App Vitals — switch it on so the job collects App Vitals. Without it, the job can't be compared with a benchmark.
• App Vitals cold start — Android (Espresso) only, and it needs App Vitals switched on. It is optional. Before each test, RobusTest force-stops the app and measures a cold launch; this is the only source of the Cold start value. Switch it on if your benchmark has a Cold start check. Each test then starts from a freshly launched app, so turn it on only where your suite is fine with that.

When the result appears

The comparison starts after the job has finished and the App Vitals data of every test has been processed. While it waits, the result shows Pending ("Comparing App Vitals with the benchmark"). This normally takes a short while. If some data is still missing after about 10 minutes, RobusTest compares what it has and marks the rest Not compared ("summary not ready").

Reading the result

Results

How the job result is decided:

• Each test gets its worst metric result. A test fails if any of its checks fails, or if it crashed. It passes only if a benchmark check was actually compared and passed — a test with no crash but nothing compared is Not compared, not Pass.
• The job fails if any test fails. Otherwise it passes if any test passed. If no test could be compared, it is Not compared.

What is counted:

• A crash, ANR or hang in any attempt of a test counts, even if the test passed on retry.
• For the value checks, RobusTest uses the last attempt of each run, and only runs that completed and passed. A failed run never decides a check.

Where to see it

On the job dashboard

Open a job from the Test Runs list. If it ran with App Vitals, a performance chip appears at the top right of the job dashboard, in line with the breadcrumb.

The counts in the tooltip are counted in tests. In this example the benchmark covers many tests but the job ran only a few of them. The ones the job didn't run count as not compared, and the one test that was compared stayed within its limits, so the job result is Pass. If you expect more tests to be compared, check that the job ran them and that they are in the benchmark.

A failed job looks like this. The chip turns red and reads Performance: Fail, and the tooltip lists how many tests failed.

Here 2 fail · 4 pass · 31 not compared means two tests were worse than a limit (or crashed), four were within their limits, and 31 could not be compared (for example, benchmark tests this job didn't run). One failed test is enough to fail the whole job, even though most tests passed. Click the chip to see which tests and metrics failed.

Two different counts on one page. The boxes below the summary (Total, Pass, Fail …) count the job's test cases and whether they passed or failed. The chip counts tests against the benchmark, so the numbers differ. A job can pass all its test cases and still show Performance: Fail, or the other way round.

The chip updates by itself: a job that has just finished shows Pending and changes to the final result when the comparison is done.

In the Test Runs list

On the project's Test Runs page, every job that ran with App Vitals shows a small performance icon next to its status icons (highlighted in red below). It lets you spot a performance problem without opening the job.

Hover over the icon for a short summary (the benchmark and version, and the test counts). Click it to open the Job performance page in a new tab.

The performance icon is separate from the test result. The Pass % column tells you whether the tests passed; the performance icon tells you whether the app's performance stayed within the benchmark. In the screenshot, the first job has a low pass rate (16.67 %) but a green performance tick, while the second job passed 83.33 % of its tests but has a red cross, because a test was worse than its limit or the app crashed. Read both.

On the Job performance page

Click the chip (or the icon in the Test Runs list) to open Performance comparison for the job. This is where you find out what went wrong. The breadcrumb links back to the job and to the benchmarks, and the JSON button opens the same data as the API returns.

Each row of the Results table:

Reading a failed result — an example

The top of the page above says Fail — 2 of 37 tests failed a check, so the Results table lists the two failed tests first:

1. MisbehaveCrashTest#appCrashes failed because of a crash. The Crashes row shows 1 for this run, its Rule check says Always decides, and its Outcome is Fail with 1 crash during the test. A crash fails a test whatever the benchmark holds, which is why the Benchmark column shows –.
2. PinTest#guestSignsInWithoutPremium failed because of Cold start. The benchmark value was 1,359 ms and this run took 1,752 ms: +393.5 ms, +29 %. The check allows at most a 25 % increase, so the Rule check says Worse and the Outcome is Fail (Cold start +29.0 % vs benchmark (limit +25 %)).

The other rows of that second test are Pass — memory and CPU are not worse than their limits, and App memory growth is within the limit (31.7 MB ≤ limit 150 MB). One failing metric is enough to fail the test.

The last row, Janky frames, is greyed out and Not compared: too few frames to score: 26 frame(s) drawn, at least 60 needed. A metric that isn't compared never counts as a pass or a fail.

To investigate a failed test, read its metric rows to see which metric crossed its limit and by how much, then click open run to see the App Vitals report for that run (graphs of memory, CPU, frames and so on).

If the job has no result, the page says No performance result for this job: App Vitals was off for it, or it ran before benchmarks existed.

Why a test is not compared

A test shows Not compared, with the reason, when:

Some metrics are skipped on purpose and listed in the notes, for example janky frames and cold start on iOS (not captured yet), and hangs on iOS unless the device log stream is on (crashes are still checked).

Good practice

• Build the benchmark from a build you trust, ideally from 3 or more runs of the same test on the same device model.
• Use percentage checks for noisy metrics (memory, CPU) and fixed limits for targets (cold start).
• Start with a loose limit and tighten it as you learn how much your app normally varies. Every change is a new version, so you can always see what changed.
• Keep one default per project and name other benchmarks (for example per release) in the Run Setting only when needed.
• Read the warnings on a benchmark. Few runs, runs from different devices, or low coverage mean the value may not be reliable.

API reference

All endpoints need a logged-in user or access key. Changing a benchmark needs a project admin.

The job performance page is at /appvitals/job/<job id>/performance.

Related documentation

• App Vitals
• App Vitals — Android metrics
• App Vitals — iOS metrics
• Run Settings
• Reading Job Reports