INSIGHTS / ROBOT SOFTWARE

AI Writes the Code. How Do You Know It Works?

A coding agent has implemented a feature and its tests pass. Before you accept the change, there is one question to answer: did the tests check the behaviour you asked for, or only the code the agent wrote?

We found a way to answer this question for robotics software projects. This article uses a simplified localisation example, but is applicable to all robotics software components.

Explore the workflow
Engineers reviewing robot software beside a robot test setup
BEFORE YOU START

A new role not yet in your org chart

You certainly have these roles already: the product owner, the dev team, the customer, the project lead, the scrum master...you name it. This does not work the same anymore in the new paradigm, since a new role shifts the other: we assign a specific role to the agent, and two other roles are severely impacted:

  • The Product Owner (PO). In 'classic' agentic workflows, a Product Owner will be overloaded with features to check, decide on and scope. Where previously the dev team was slowed down by 'feature creep', agents make it 'feature explosion'. The PO will want to A/B test lots of combinations, manage backlog ('to do') and inventory ('to merge'), keep sanity over the tremendous pace the dev team acquires.
  • The Robotics Software Engineer (Devs/Ops). Their role shifts from 'code' to 'architecture'. The Devs turn the feature request into modules, interfaces, behaviors, limits and exceptions/errors. A true specification, in human language that adds detail to the PO's feature description. The Ops create the 'Harness', which is the tooling the agents will use. They prepare input/evaluate output of the agents.
  • The Agent (AI). It's role is to work within the Harness created by the Devs/Ops. It will create mutable assets (code and tests) and its output/work is evaluated with immutable assets (the specifications) and immutable tooling (which was created using the same workflow).

If you are a software engineer, you may see the 'Catch 22': if the tooling which evaluates Agents is also made by Coding Agents, the trust moves to that tooling. Who then validates that code/test stack? A Dev? Yes :-) How? Let's explore that below!

THE RUNNING EXAMPLE

Start with one behaviour

Specification, observable robot behaviour and implementation code

Imagine a robot's localisation component loses the sensor input it needs. Other components use its position estimate to plan and move. What should they receive next?

“Handle missing sensor data” is not specific enough for a user story. The component might keep returning the last position, report an error or stop producing output. Each choice has a different effect on the rest of the robot.

The PO starts at the product level: a robot must not plan from a stale position, and it must recover predictably when the sensor returns. That behaviour becomes a backlog work item. Before asking for code, decide what another component must be able to observe.

STEP 01

Turn the backlog item into a specification people can read

The PO's work item says what the robot needs to do. The engineers turn it into a specification for the current architecture. For localisation, that means identifying the node that owns estimate validity, the interface through which planning receives it, the modules that must change and the modules that must not.

The localisation specification might say: when required input stops arriving, mark the estimate as unavailable within an agreed time. Do not present an old estimate as current and valid. When input returns, recover in an agreed way. It also defines a boundary: localisation says whether its estimate is usable; motion decides how to respond. The engineers agree the interface, limits and error cases with the owners of both components.

Write down these timing requirements, recovery conditions, affected interfaces, expected error handling and what lies outside the change. The result is more detailed than the PO's feature request, because it must be precise enough to evaluate an implementation on this architecture. Here, an agent can help expose gaps; the engineers and PO resolve them.

The PO stays at the higher level: which behaviour matters, which feature combinations belong together and which trade-offs are acceptable. The engineers own architectural quality and implementation checks.

STEP 02

Build the tooling that turns specifications into repeatable work

The Devs/Ops develop tooling, ie the 'harness', that reads the approved specification, prepares the right harness and starts the agents in a controlled environment. For the localisation item, the harness assembles the component, the relevant interfaces, a replay or simulation input and the expected observations. It also records the software branch and exact revision, scenario and configuration so the result can be reproduced.

The acceptance scenario for our localisation example is simple:

  • Supply valid sensor input and confirm that the component produces a usable estimate.
  • Interrupt that input and check the reported validity and elapsed time against the requirement.
  • Restore the input and check the agreed recovery behaviour.

Put this scenario in a protected harness: the environment, inputs and checks used to assess the change. The implementation agent can run it and inspect failures, but it cannot change the acceptance conditions to make its own work pass. If contradictory requirements block progress, engineers resolve the contradiction with the PO where product intent is affected, update the specification and restart the agent loop.

The first useful scenario takes setup. Engineers can use coding assistants to build scenario runners, reports and protection checks, then review that tooling separately from the feature implementation. With an existing build and replay setup, allow weeks to months for this work to mature. A new simulator or hardware rig needs its own schedule. Once built, your setup will serve many changes.

STEP 03

Give agents a protected harness and a repeatable environment

Give the coding agent the architectural specification, relevant interfaces and repeatable environment. It writes code and supporting tests, runs the protected checks and uses the results to improve the change. The tooling collects each run's inputs, outputs and evaluation. Engineers resolve technical blockers; questions that change the requested behaviour go back to the PO.

We don't trust that the agent writes correct tests - they are part of the process but not part of the proof. Agents will create them in order to 'anchor' assumptions of the written code and they will in most cases flag when assumptions are broken by a code change.

THE TWO LOOPS

One controlled flow, from request to decision

Workflow phases and the agentic and human review loops

We have now reached phase 3 in the diagram: the implementation runs through the immutable test harness. Phase 4 returns evidence to the agent, closing the inner loop. The agent can improve the code and run the same checks again. The wider human loop uses independent validation to decide whether the result meets the request or whether the specification needs to change.

STEP 04

The agent loops; engineers verify

After a test run, the agent receives evidence: which checks ran, the recorded inputs and outputs, the plots or robot visualisation, and what remains unverified or broken. That evidence drives its next iteration and is available to engineers throughout the loop.

For the localisation example, the evidence report should show the sensor-input trace, the estimate-validity signal, the elapsed time to unavailable and the recovery trace. If the behaviour has a spatial effect, a 3D robot visualisation can show the estimate and the robot's response.

This is where the Devs monitor the agent. Engineers check that the specification was interpreted correctly, that the harness measured the right thing and that the plots make technical sense. They verify what went in and what came out, including the claims in the agent's report. A separate validation agent can compare the change with the original request and evidence, but engineers check its conclusion against the data too.

These views also help engineers check the tooling itself. Did the scenario actually interrupt the sensor input? Does the report show when the estimate became unavailable and when it recovered? Plot the recorded samples against a common time axis, with units and the specified limits visible. Compare the plot with the source data and try a deliberately broken case to check that the evaluator detects it.

And code reviews? Our view is that review effort should shift towards specifications, architecture and the harness. Reading generated code alone gives uneven assurance about robot behaviour. Engineers need to examine whether the checks could expose a wrong implementation, then inspect the observed behaviour. They can still inspect code where the evidence raises questions. The PO adds another level of scrutiny by exercising the resulting features together.

STEP 05

Combine features and visualise behaviour at every level

A robot combines localisation, perception, planning, control, drivers and many other components. You need a graphical interface at every level: the module, the architecture and the full application. Each view answers the same question: what was the input, and what did the software produce, both in isolation and in combination with other components?

For localisation alone, plot sensor arrivals, estimate validity and recovery. Across the architecture, align those signals with the planner's decisions and motion commands. For the full application, show the robot's trajectory and mission progress in a 3D replay alongside the measured timings. Each view should let the reviewer trace what they see back to the recorded numbers.

Once engineers have checked a feature, it enters the PO's inventory of candidates to combine and merge. The PO uses a view tuned to product decisions and can launch the same workflow for different feature combinations. For example, combine the localisation change with a new planner recovery strategy: does the robot resume its task smoothly, wait indefinitely or repeatedly stop and restart? A module can meet its specification while the combined behaviour reveals a side effect or an unforeseen error mode.

The PO decides which combinations deliver the intended behaviour and which discoveries become new backlog items. Engineers trace problems through the architecture and improve the specification, harness or implementation. This closes the wider human loop in the diagram.

GETTING STARTED

Start small, then use the evidence to scale

Start with one behaviour and one meaningful scenario: sensor input is interrupted, the estimate becomes unavailable and the system recovers as specified. Let the PO create the work item, let engineers make the architectural specification and harness, and let the tooling return plots and results through the engineering gate before the PO reviews the behaviour.

Treat the first component as setup followed by several real change-and-review cycles. Record the human effort, test runtime, plots that made a decision easy and unresolved questions. That gives a better basis for the next component than assuming every team needs the same preparation.

So the answer to the tooling question at the start is a graphical interface for verification at every level. Engineers use plots and robot visualisations to check implementations and the tools that evaluate them. The PO uses views of the same evidence to judge combined product behaviour. Every accepted change should answer what was requested, what went in, what came out, how it was checked and who owns the decision.

CONTINUE THE CONVERSATION

Explore our AI-Assisted Software Development approach, or talk to us about a first workflow.