Running tests
What a result means
TestFlow is built to be unable to tell you a test passed when it did not. This page is the specific list of mechanisms, so you can decide how much to trust it.
Every AI tool claims to be careful. What follows is checkable: each item below is enforced by code, not by an instruction in a prompt, and each one exists because the failure it prevents is one we found rather than one we imagined.
The organising principle
A behavioural guarantee is never a sentence in a prompt.
Prompt text is advisory. A model under pressure can misread it, and a model that has just been told three times to be helpful is exactly a model under pressure. So prompts are used for wording, and the guarantees are physical: a capability that is absent, a change that is rejected, a claim that is refused, an outcome that does not exist.
The five guarantees
- 1
The agent cannot invent an instrument or a command
Workflow steps are validated against your configured bench and against the real action catalog of each configured model. An action name that is not in a catalog is rejected, and the true list is returned so the agent corrects itself in the same turn. - 2
The agent cannot write a measured value
The Report's structure has no field capable of holding a measurement. It can ask for a column called load current with a band of 1.9 to 2.1 A. It cannot say the load current was 2.03 A, because there is nowhere to put that sentence. Values come from the run's results file and nowhere else. - 3
The agent cannot claim a pass it does not hold
Stating that a test passed is a separate, gated operation. It refuses unless the turn is holding a passing outcome from a run of the current workflow, and its refusal hands back the honest sentence to use instead. - 4
A stale result stops counting
Each run's outcome carries a fingerprint of the workflow structure that produced it. Edit a step and the previous green result no longer backs any claim, because the sequence it belonged to no longer exists. You cannot change a test and then point at the old run. - 5
A failing limit is a result, not a bug to fix
When a reading falls outside its band, the run is reported and stopped. The agent does not re-run it, and does not edit the test until it passes. Re-running cannot change the reading, and widening the limit is fabricating a pass.
What the product does after a run
The decision is policy, not judgement. It runs in this order, and the reason each guard exists has nothing to do with who pressed the button.
| Condition | Action | Reason |
|---|---|---|
| The run passed | Report it. | Nothing to repair. |
| You pressed stop | Report it. | You said stop. |
| A limit failed | Report it. Never retry. | The test worked and the part is out of spec. That is the answer. |
| Three attempts already spent | Report it. | A human’s number, not a model’s. |
| The same failure signature again | Report it. | The same error after a fix means the diagnosis was wrong. More attempts will not help. |
| An instrument failed | Ask you. | Re-running does not reconnect a cable. |
| The run took five minutes or more | Report it. | Bench time is yours, not ours. |
| An error, or no outcome at all | Retry. | This one is the software’s fault, and this is what retries are for. |
Unknown is not a pass
A run reports its outcome as one explicit line at the end. A run that ends without that line is unknown. Unknown is treated as a failure to produce a result, never as a success, and “no run has reported in this session” is never read as “the last run was fine”.
What is still on you
- The limits are your limits. TestFlow evaluates the band you stated. If the band is wrong, the verdict is wrong.
- The wiring is your wiring. A correct sequence on a miswired bench measures the wrong node perfectly.
- Simulation is not evidence. Only a bench run tells you about your part.
- Read the generated driver. For a special instrument you are shown the driver before it first runs. That approval step exists because it is worth using.