AI debugging: from plausible patches to reproducible evidence

On Mapify, the strongest early debugging tools were close to the system: inspect the PostgreSQL schema, query records, count migration states, read migration errors, check missing search vectors and compare backend logs with the code that wrote the data. That was already much better than editing from a symptom alone. Read operations were safe by default; any database write required a preview of affected records and explicit approval.

The limitation was not the quality of those checks. It was that the loop was still mostly task-specific. A database investigation could be excellent without a reusable contract for the exact browser state, account, input file, feature flag and visible before/after behaviour. The next debugging session depended on remembering which evidence had mattered last time.

That experience shaped the later system: keep the direct access to source, logs and data, but wrap it in a workflow that can be repeated by another agent and audited by another engineer.

Henri's debugging skill begins by classifying the failure. A UI or layout problem starts with a device screenshot and view hierarchy. A data or state problem starts with captured logs, state and read-only database queries. An API problem adds network and backend logs. A crash starts with the stack trace. The category does not solve the bug; it selects the observations that can disprove the first theory.

The workflow reuses captured artefacts before asking for another reproduction: iOS logs, backend logs, the current device screenshot and saved test results. If that evidence is insufficient, the agent instruments the suspected boundary and runs the case again. It is explicitly told not to guess. For visual ambiguity, temporary coloured borders can reveal which container owns unexpected spacing; for data ambiguity, a read-only query can show whether the wrong value was stored or merely rendered incorrectly.

The fix also has to respect the surrounding system. A layout correction checks the shared component and design tokens before patching one screen. A schema or persisted-contract regression audits older onboarding, resume, library, dashboard and reset paths that read the same state. After implementation, the agent self-reviews for duplicated components, inconsistent neighbouring states and new constants that belong in the shared design system.

Different evidence answers different questions. The visible screenshot says what the person saw. The view hierarchy says which element owned the geometry. Runtime logs show ordering and state transitions. A network trace separates a client timeout from a server error. A backend log follows the request beyond the mobile boundary. A database read shows what persisted. A crash stack locates the failing path. An Instruments trace aligns hangs, hitches, CPU work, signposts and application logs on one timeline.

The agent's job is to connect those signals without treating them as interchangeable. If the database row is correct but the screen is wrong, changing the write path is noise. If a request fails before reaching the server, backend retry logic is not the first owner. If an Instruments trace shows the stall before a named log event, the investigation can focus on the preceding work rather than broadly ‘optimising SwiftUI’.

This layered approach also produces better handovers. The root-cause explanation can point from observed symptom to state transition, request, record or hot path. Another engineer can challenge that chain instead of reviewing a patch whose only justification is that it appeared to work once.

For reported bugs, I now create a reproduction data contract before implementation: where the evidence came from, the exact visible values, any values inferred from them, the route and selected tab, the app build and backend, feature flags, locale, device or viewport, and how the evidence was captured.

Original attachments or replay data are preferable to screenshots of a ticket page. Concrete values in a report should become the fixture or controlled test data; easier placeholder numbers can hide the calculation, formatting or threshold that caused the problem. If the report implies a value rather than stating it, the inference is recorded instead of silently becoming fact.

This contract is what makes before and after meaningful. The pair is only comparable when it uses the same data, real screen, relevant state and viewport. If that exact rerun is impossible, the report says so and keeps the original ticket evidence separate instead of presenting mismatched pictures as proof.

Computer use is valuable because the agent can operate the product as a person would: open the app or browser, navigate through the normal route, select the reported state, reproduce the interaction, inspect the visible result and repeat the path after the fix. It turns visual verification from ‘please check this manually later’ into part of the engineering loop.

The boundary is important. A desktop screenshot of a Simulator window includes computer chrome and may hide or scale device pixels. A generated SwiftUI capture screen, preview wrapper, Storybook page or reconstructed HTML can resemble the product while bypassing navigation, state, native payload construction or backend behaviour. Those views are useful diagnostics; they are not authoritative evidence that the product path works.

For final mobile evidence, the workflow captures the complete simulator or emulator framebuffer directly—`simctl` on iOS or the Android equivalent—then checks orientation and readability. Computer use drives the journey and inspects the result; the device capture records what the mobile runtime actually rendered. For browser products, the real browser route and loaded state remain the authoritative surface.

Henri's full test workflow makes the order explicit. First comes a compile gate. Launch-sensitive changes then require a launch smoke test because an app can compile and still crash while creating its first model or predicate. Unit tests run next because they are fast and isolate logic. Targeted UI tests verify the changed journey fresh. Full UI coverage can then run in isolated parallel shards, followed by a single-stack database verification pass when aggregate persisted state matters.

Each UI-test stack owns its backend, PostgreSQL database, Redis, object storage, simulator and build output. Isolation matters because a green parallel run is meaningless if shards silently share a backend URL or skip database-backed cases. The runner checks enumerated tests against executed tests and cleans leaked containers and simulators after interrupted runs.

A failure is classified before anything is edited. A deterministic failure that repeats in isolation may be a product defect or stale expectation. A failure that passes on retry may expose shared state or ordering. A multi-minute test may indicate an overloaded simulator or hung backend; its results are not trusted. A backend that never started is an environment failure, not evidence that the application regressed. The aim is to repair the right system, not simply return the dashboard to green.

AxIT uses a managed simulator matrix rather than whichever device happens to be open: a small phone, a large phone and a representative iPad, with additional tablet coverage when the layout or resizable window requires it. The real app is installed, the affected screen is reached through normal navigation, and complete before/after framebuffers are inspected for clipping, safe areas, keyboard behaviour, Dynamic Type and readable hierarchy.

That matrix is intentionally the default even when physical devices are nearby. Simulators are reproducible, resettable and suitable for parallel automated coverage. They do not prove Bluetooth discovery, connection ownership, sensor sampling, physical-device performance or an install on the target hardware.

When the boundary is hardware-only, the claim stays pending until the exact signed build is installed and exercised on a paired device. The physical-device workflow records the model, bundle, marketing version and build, source revision, dirty state, installation path and launch result. Performance profiling also respects the boundary: some SwiftUI Instruments data is available only from a real device, while Simulator runs use the profiler lanes they can actually populate.

After the root-cause change and regression test, the agent reruns the original steps through the same real product path. It uses the saved data or fixture, account, environment, feature flags, selected state and viewport whenever possible. A multi-state bug repeats the same state matrix; a multi-step workflow captures every meaningful stage and the final persisted result.

The resulting record separates what is visible from what proves the transition. A screenshot can show the corrected loaded state. A test or log can prove an invisible lifecycle event. A physical-device observation can prove Bluetooth behaviour. A human test result can approve the build. Keeping those claims separate makes the report more useful, not less confident.

On AxIT, SBN Testing extends that loop across the team. Exact-build plans, dependency maps, shared evidence, live progress and questions let testers contribute in small windows while developers respond to failures before the whole pass ends. Existing results and private evidence are preserved when a plan changes; a new automated run never rewrites a person's acceptance history.