The problem
A model can generate code and say it worked. That sentence establishes almost nothing. It does not show that the code executed, that the result was measured against criteria anyone agreed on beforehand, or that the evidence still exists and can be read by somebody who was not in the room when it was produced.
The gap is between model-generated output and an evidence-backed execution verdict. Closing it is harder than it sounds, because every convenient shortcut — trust the model, run it on the host, accept the summary — removes precisely the evidence that would have made the result checkable.
Frontier Lab treats “it said it worked” as a claim to be tested, rather than a result to be accepted.
What it does
- Candidate submission — Accepts candidate submissions against defined missions and profiles.
- Sealed binding — Binds an execution to sealed source artifacts and a specific authorised profile.
- Disposable execution — Runs qualified candidates inside disposable VirtualBox guests rather than falling back to arbitrary host execution.
- Evidence capture — Records execution outputs, exit states and supporting artifacts.
- Evaluation pipeline — Applies criteria to actual execution evidence, not to the model’s account of it.
- Replayable records — Retains records that can be inspected and replayed offline.
- Shared interfaces — Exposes the same system through CLI, Web and MCP.
- Authorisation checks — Requires independent, current authorisation for execution; invalid, expired or mismatched qualification blocks execution rather than silently lowering the safety boundary.
How it is built
Two contracts carry the design. The execution contract says what may run and under whose authority. The verification contract says what counts as proof and who is permitted to produce it. Everything else — the guest lifecycle, the evaluation pipeline, the retention layer — exists to serve those two.
The security boundary is deliberately unforgiving. Execution runs in disposable VirtualBox guests rather than on the host, and authority is explicit, scoped and independently checked. An invalid, expired or mismatched qualification stops the run. It does not degrade into a more permissive mode, because a fallback that quietly lowers the safety boundary is precisely the failure this system exists to prevent.
Evidence is retained rather than summarised, which is what makes offline replay possible: the same verdict can be reproduced without rerunning the model, the virtual machines or the workload. The system is exposed through CLI, Web and MCP interfaces, and the qualified workflow holds consistent state across all three.
The evidence
A Run B qualification receipt dated 9 October 2026 recorded package version 1.0.0 with a RELEASE_QUALIFIED decision for the specifically qualified environment and mission profile. The run used a real Codestral model request and a real candidate execution — not a synthetic stand-in.
It recorded twelve actual disposable guest executions, with the recorded candidate outcome passing 30 evaluation criteria. Offline replay then reproduced the same PASS verdict and the same 30 criteria without rerunning the model, the virtual machines or the workload. CLI, Web and MCP state were consistent across the qualified workflow, and a final installed-package regression suite finished with 567 tests passing and zero failures, errors or skips.
These results establish qualification for the stated run and environment. They are not a universal claim about every model, candidate, workload or installation.
My role
I set the system architecture and product direction, and wrote the execution and verification contracts the rest of the system implements. The security boundary design — disposable guests, explicit authority, no silent fallback — was mine, as were the qualification criteria and the adversarial review that pushed against them.
I also coordinated implementation through coding agents and owned the release verification: deciding what had to be demonstrated before 1.0.0 could be called qualified. The distinction the system is built on is also how I worked on it. A passing model response was never enough to mark anything done.
Technical notes & claim boundaries
- Qualification applies only to the pinned Windows/VirtualBox environment and the
cwd-resource-repair:1.0.0candidate profile. It does not authorise arbitrary executables, general host execution or other mission profiles. - Clean-machine installation and backup/restore were not verified in the installation documentation. There is no claim of universal containment, unrestricted model evaluation or production readiness across environments.
- The evidence above is scoped to one qualification run. It is not a statement about all models, all candidate programs or arbitrary workloads.