Two-phase QA for metric algorithms and app behaviour

I split the process at the point where the required expertise changes. Phase one happens in AxLib before app integration: engineers replay a candidate against de-identified production traces, compare it with the baseline and prepare the changed or uncertain cases for domain review. Phase two begins after that calculation is accepted and integrated: app testers verify observable behaviour in one exact build through SBN Testing.

To see how a new calculation behaves on real clinic work, each check reruns it on about 1 in 100 of the saved force recordings it includes. Personal details are already removed. That is a sample plan, not a review of every test the app has saved. Each run keeps the exact number of recordings used, which recordings were left out, which version of the calculation ran, and the outcomes, so a reviewer can see what the sample covers.

I designed AxLib as both a development environment and a calculation-review system. Each candidate runs against frozen, de-identified traces without replacing the baseline. The comparison records the input identity, algorithm identity, metric values, units, calculation status and intermediate evidence needed to explain why a result changed.

The important measure is not only ‘did the code run?’ AxLib calculates baseline and candidate success rates over the selected production cohort and surfaces recovered, lost, changed, unchanged and unavailable outcomes. Domain experts can then spend their time on the delta: the cases where behaviour changed, evidence is ambiguous or the calculation needs a judgement only they can make. This is possible because those calculations come from one shared metrics Kotlin Multiplatform project.

Version comparisons remain reviewable and append-only. A later candidate can improve the algorithm without rewriting the evidence used to approve an earlier one, and disagreement can be recorded as correct, incorrect, uncertain or still unreviewed instead of being compressed into a misleading green status.

Once an algorithm has passed the first phase and is integrated into the app, SBN Testing gives testers a smaller, clearer responsibility: verify what a user can observe. That includes metric availability, names, phases, units, selections, save and reopen behaviour, synchronisation and the feature-flag combinations that control the experience. They do not need to re-derive the maths.

The dependency map is where a tester reads that relationship for one exact build: the flag combination first, then the features and suites that combination covers. Progress is shared live, results retain their evidence and history, and the same case can be picked up in a short pocket of time without losing who tested what. This turns fragmented availability into useful parallel work instead of duplicated or context-free checks.

The steps stay easy to follow when a screen is unfamiliar or new. Hovering a named screen or button in a step opens a popover with that step’s reference screenshot, so the tester can see the part of the app the instruction means without leaving the case. The same preview opens from the keyboard or a tap. A step with no screenshot stays ordinary text.

Feedback stays close to the evidence. A tester can mark a case Passed, Failed, Blocked or Don’t understand, attach a screenshot and start a discussion on that exact result. Questions and replies can optionally notify the relevant people in Slack with a protected link back to the build and case; the durable conversation remains in SBN Testing rather than being lost in chat.

That live record also removes the wait between testing and fixing. A developer does not sit idle until the whole pass is finished and handed over as a batch. As soon as a case fails, the exact build, configuration, screenshot and discussion are already attached, so the developer can start the fix while testers continue the remaining cases. The overlap cuts the time between finding a defect and correcting it.

I introduced a typed feature-flag contract for AxIT iOS. Every flag is one Boolean, named feature_flag_ios_<name>, registered in a single catalogue and backed by Firebase Remote Config. New product behaviour defaults to off. While it is off, its entry points and side effects stay out of the product. Turning it on reveals that behaviour in the build already installed, so testers compare the two states without waiting for a second binary. If Remote Config is missing or the fetch fails, a new flag stays off rather than turning itself on.

That removed a release wait. The implementation can travel inside the exact build under test while users still see the previous behaviour. Enabling it for testers is a flag change and a relaunch, not another app. The rest of the pass can continue on that same build with the new behaviour still hidden.

The dependency map stops the combinations from becoming guesswork. It does not ask anyone to try every on/off permutation. For one exact build it groups cases that can share a setup. All ON turns on the flags under test and still lists the safeguards that must stay off. Cases that need contradictory settings stay in separate configurations. Each configuration is drawn through to the features and suites that depend on it, so a tester sees both the switch positions and the product behaviour those positions cover. Choosing a configuration never changes the flags on the device. It tells the tester what to set, and then what to confirm, before the related features are tested.

Switching to tests that need a different combination does not keep the previous setup. SBN Testing clears that confirmation and asks the tester to check the required on and off positions, then click to confirm them, before those cases open. The click records that the tester checked the app. It does not change the flags. If a later step itself changes a flag, the steps after it stay closed until that change is checked and confirmed too, so the pass cannot continue on the wrong setting.