Don't Trust the Claim, Verify the Artifact: Verification-Gated Orchestration for Coding Agents

Read this first. The claim here is about trustworthiness, not capability. Verification-Gated Orchestration (VGO) does not make a worker solve more tasks. It makes a worker's reported success independently checkable, so that an unsupported "done" claim cannot by itself cause an unsafe merge. The central quantities are therefore false-acceptance metrics, not task-resolution rate.

Autonomous coding agents increasingly report that tasks are complete, tests pass, or gates are satisfied. When an orchestration system accepts these claims without independent verification, a false completion claim becomes an unsafe merge. VGO removes acceptance authority from the worker agent and assigns it to an orchestrator-owned verifier: it routes work through gated phases and accepts an artifact only when the orchestrator runs the task's grader and records verifiable evidence bound to the attempt.

Because VGO targets trustworthiness rather than capability, the work introduces two false-acceptance metrics — unsafe-merge incidence (per attempted task) and false-acceptance rate among merges (conditional on a done or merge) — alongside task-resolution rate. In a seeded substrate simulation with 200,000 tasks per condition, a claim-trusting baseline produces unsafe merges at the expected per-task rate f·(1−c), with a false-acceptance-among-merges rate up to about 0.70, while VGO produces zero unsafe merges under the modeled threat assumptions.

The earliest phase is doing more than it looks. The gates that define "done" are encoded up front, during harness design, before any artifact exists — so they cannot be bent to fit the result, and they are the same programmatic checks the orchestrator runs later. The harness-design phase is where verification is grounded; everything downstream redeems a commitment made there.

This is a working draft. The system, the evaluation protocol, and the substrate simulation are real, reproducible measurements, framed as invariant validation rather than as evidence of real-agent prevalence or end-to-end benchmark efficacy. A measured claim-trusting baseline and an at-scale software-repair study remain future work. The thesis is architectural: autonomous agents should not adjudicate their own success; verification of completion claims belongs in the substrate.

The dominant narrative around autonomous coding agents is capability: how many issues an agent resolves end-to-end. Deployment surfaces a second, orthogonal problem that capability benchmarks do not measure — can you believe the agent when it says it is done?

Agents routinely emit confident completion claims — "all tests pass," "the gate is satisfied," "task complete" — that are not independently verified, and that are sometimes simply wrong. When such a claim is trusted, unverified or broken work is accepted and merged. The risk grows precisely as agents are given more autonomy and less human review, because the human who would have caught the discrepancy is the resource autonomy is meant to save.

This is a failure of the substrate, not the model, and it must be fixed there. An agent asked to grade its own work is structurally unreliable for the same reason self-reported test results are: the entity producing the work also adjudicates it. The fix is to externalize verification — move the authority that decides "did this pass?" out of the worker and into an orchestrator that independently grades the real artifact and refuses to accept unsupported claims.

The contributions are six:

1. A verification-gated orchestration architecture that separates artifact production from acceptance.
2. Up-front gate encoding: acceptance criteria are committed during harness design, before any artifact exists, as the programmatic grounding that later verification redeems.
3. An evidence-envelope mechanism that binds each accepted attempt to an orchestrator-owned grader result.
4. Anti-false-acceptance mechanisms: no auto-pass, route-to-verification, and unsupported-claim detection.
5. A metric suite for harness trustworthiness: unsafe-merge incidence, false-acceptance among merges, task resolution, and restart-safety invariants.
6. A seeded substrate simulation and ablation study validating the no-unverified-merge invariant and isolating verification from repair.

Throughout, "fabricated success" and "false acceptance" are used operationally: a worker emits a PASS or done claim for an artifact that fails the grader, regardless of whether the false claim is intentional, stale, or caused by a tool or state error.

The worker may produce code, edit files, run tools, and emit natural-language or structured claims that a gate has passed; it may also attach stale, incomplete, or invalid evidence. The orchestrator is assumed to control the acceptance rule, the grader invocation, and the evidence log.

VGO does not defend against a compromised grader, malicious changes to hidden tests, a flaky test suite, or an orchestrator whose verification environment is itself corrupted. Within this model, the property sought is precise: a worker's unsupported PASS claim cannot, by itself, cause a merge.

There is also something the worker structurally cannot do: redefine the gate after the fact. Because the acceptance criteria are encoded during harness design — before the artifact exists — and frozen thereafter, the worker can fail a check but cannot quietly move it. Self-assessment is bounded by a target the worker did not get to choose once it had a result to defend. The narrow window in which a model is most prone to motivated reasoning is precisely the window in which it has nothing to encode.

VGO wraps a worker agent and governs acceptance rather than patch generation. The worker still produces the artifact and may still claim success; what changes is who decides whether that claim is accepted. The orchestrator runs the task's grader against the real artifact and merges only on a grader pass.

Each task is a lifecycle instance progressing through phases — planning, harness design, building, a spec-drift check, a regression gate, triage, a fix-loop, review-ready, and completed — with an escalation off-ramp. Gates sit between phases; an agent cannot advance past a gate it has not satisfied. The spec-drift check guards against scope creep; the regression gate guards correctness.

The acceptance gates are not discovered at the end; they are encoded at the beginning, in the harness-design phase, before any artifact exists. That ordering does more work than it appears. At design time the worker has produced nothing, so there is nothing yet to claim "done" about — the criteria for success are authored in a regime with no incentive to fabricate, because there is no result to defend. Once a gate is committed it is frozen: the worker cannot later redefine "pass" to fit whatever it happened to produce.

The plan artifact and its encoded gates then condition everything downstream. The worker generates tokens against a fixed target rather than toward a post-hoc rationalization, so the build stays statistically in line with the plan. And the encoding is not a separate document: the programmatic check authored up front is the very check the orchestrator runs later at the regression gate. Verification is therefore not bolted on at the end — it is the redemption of a commitment made before the work began, which is what keeps the later grader ground truth rather than circular.

This grounding only reaches what can be defined and proven at design time. Requirements that emerge mid-build, or properties no one can yet specify, cannot be grounded this way; they fall back to the weaker runtime mechanisms — routing, escalation, and human review. The converse, though, is a design rule rather than an option: if a property is provable, it must be encoded in the initial harness as programmatic grounding. Leaving a provable check to be asserted later, or by the worker itself, reintroduces exactly the bias the architecture exists to remove. (This is also why the spec-drift check earns its place: it holds the build to the committed plan, so the grounding established at design time is not quietly eroded as work proceeds.)

The defining mechanism sits at the regression gate: the orchestrator — not the worker — loads the task's grader, runs it against the produced artifact, and attaches an evidence envelope to that attempt. The envelope binds the acceptance decision to verifiable facts: task id, attempt index, artifact hash, grader id and version, the scorer result, and a timestamp.

First-pass and repair-assisted attempts are recorded as distinct rows, so improvement is attributable and no accepted attempt is missing its evidence. The worker's own assertion of success is input to triage, never the basis for acceptance.

A self-healing escalation resolver matches failure and escalation events to recovery patterns. Three design points make it anti-false-acceptance:

• No auto-pass. An earlier behavior that auto-satisfied stuck gates was deliberately removed; it is the canonical way a harness admits a false accept.
• Route, don't fabricate. Unsatisfied gates are routed back to genuine verification (or to the orchestrator) rather than advanced.
• Unsupported-claim detection. A check flags any worker that reports a gate as PASSED without valid supporting evidence, converting a silent false accept into an explicit, logged escalation.

Terminal-phase and recently-resolved de-duplication guards prevent loops. Supporting components include a TTL'd known-error registry that injects previously-seen failure signatures into agent context; a composable verification harness with pluggable graders, so each task's official grader is loaded and run uniformly; and a provider abstraction that keeps the worker model swappable behind a uniform interface — essential for an evaluation protocol that holds the model fixed and varies only the harness.

The acceptance rule is merge ⇔ orchestrator_grader_pass; the worker's claim never appears in the accept condition.

Algorithm 1: Verification-gated acceptance
Input: task τ, worker W, orchestrator grader G_τ, max repair attempts k
attempts ← []
for i in 0..k:
    artifact, worker_claim, worker_evidence ← W(τ, attempts)
    scorer_result ← orchestrator.run(G_τ, artifact)          # acceptance authority is here
    envelope ← { task_id: τ.id, attempt: i, artifact_hash: hash(artifact),
                 grader_id: G_τ.id, grader_version: G_τ.version,
                 scorer_result: scorer_result, ts: now() }
    attempts.append(envelope)
    if scorer_result.pass:
        return ACCEPT(artifact, envelope)                    # merge iff grader pass
    if worker_claim == PASS and not valid(worker_evidence):
        log_unsupported_success_claim(worker_claim, worker_evidence)
    τ ← triage_and_repair_context(τ, scorer_result)
return ESCALATE(attempts)

Evaluating a harness, rather than a model, requires controlling the model. The protocol holds the worker model fixed — a single open-weight model served locally, with no per-token API charges — and runs two conditions on the same tasks: a claim-trusting baseline (accept on the worker's claim) and VGO (accept iff the orchestrator's grader passes), gating on a canonical prompt hash so both conditions present each task identically.

Two false-acceptance metrics are reported because they answer different questions:

• Unsafe-merge incidence = (merged artifacts that fail the grader) / (attempted tasks). This is the operational risk per task attempted.
• False-acceptance rate among merges = (merged artifacts that fail the grader) / (merged or done-claimed artifacts). This is whether a "done" claim is trustworthy.

Task-resolution rate (correctly accepted — the artifact truly passes — over attempted) is always compared at equal repair budget, alongside restart-safety invariants. Proportions carry Wilson 95% confidence intervals, which behave better than the normal approximation for rates near 0 and 1.

A small harnessed pilot over five instances validated plumbing: evidence logging, first-pass/repair row separation, and restart behavior. The pilot resolved 1 of 5 instances. This sample is far too small for efficacy inference, and the comparison baseline available at the time was a hardcoded fixture, not a measurement. The pilot is therefore treated only as an end-to-end integration check, with no resolution or uplift claim.

On empirical status, three honest caveats: the "published baseline" referenced in the codebase is a hardcoded test fixture, not a measurement; an earlier harnessed-vs-baseline run set failed on authentication or quota and produced empty records, so no numbers from it are cited; and a separate self-improvement loop has only a couple of real cycles with no measured behavior change and is excluded from all claims.

To measure the false-acceptance metrics under controlled conditions, a seeded simulation isolates the verification substrate, treating worker capability c and false-claim propensity f as parameters. All code and raw results are released, and the run is byte-for-byte reproducible.

Because VGO's acceptance rule is, by design, accept iff the orchestrator's grader passes, the expected unsafe-merge rate under the modeled assumptions is zero. The simulation is best read as an executable invariant test of the substrate: it verifies that the implementation preserves the no-unverified-merge invariant under repair, routing, unsupported evidence, and restart conditions, across a wide range of c and f. It is not an estimate of how often real agents emit false claims, nor evidence that VGO improves real deployments. Measuring real false-claim prevalence is the key pending experiment.

Design: each of T tasks is attempted by a worker; an attempt's artifact truly passes the grader with probability c. The worker emits a PASS claim that is truthful when the artifact passes and, with propensity f, unsupported when it fails (a fraction of unsupported claims attach a plausible-but-invalid evidence token). The claim-trusting baseline merges on the claim; VGO merges only on an orchestrator grader pass, with route-to-verification, an unsupported-claim detector, and a repair loop (k = 2). Results pool 40 seeds × 5,000 tasks = 200,000 tasks per cell, with Wilson 95% CIs.

Table 1 reports both metrics. The claim-trusting incidence tracks the analytic f·(1−c); the among-merges rate is substantially higher — up to 0.70 — because it conditions on the smaller set of done-claims. That among-merges number says how often a "done" is wrong. VGO is 0.000 on both metrics in every cell (Wilson upper bound 1.9×10⁻⁵).

The VGO zero is an invariant of the acceptance rule; the table's value is confirming that the implementation preserves it across the (c, f) grid.

To separate verification from repair, a 2×2 runs at a fixed operating point (c = 0.4, f = 0.6), comparing conditions at equal repair budget.

At equal repair budget, verification changes true-resolution by nothing in the no-repair row (0.402 vs. 0.402) while removing all unsafe merges (0.360 → 0). Repair — not verification — is what raises resolution (0.402 → 0.523 with claim-trusting; 0.402 → 0.786 with VGO). Verification itself does not increase capability; its contribution is reducing false acceptance.

One additional, honest finding: with repair, VGO's true-resolution (0.786) exceeds claim-trusting+repair's (0.523) at the same budget, because accepting a false claim terminates the repair loop early and wastes the remaining attempts. So trusting claims also costs genuine resolutions. This is reported as an observation, not as the headline.

At the same operating point (c = 0.4, f = 0.6), disabling each VGO mechanism isolates its contribution.

Route-to-verification is load-bearing: with it present (full, and minus the detector), unsafe-merge incidence is 0. The unsupported-claim detector is defense-in-depth: when routing is broken, it still removes about 85% of the residual leakage (0.131 → 0.020). The grader alone is insufficient (0.131) because auto-passing stuck gates re-admits false accepts. True-resolution is identical (about 0.784) across all VGO variants, so the anti-false-acceptance mechanisms cost nothing in resolution. (Independent runs differ from Table 2 by Monte-Carlo noise below 0.003.)

Crashing the run at its midpoint and replaying the durable first-pass log, all architectural invariants held: zero duplicate first-pass rows on replay, no lost evidence, and complete coverage after restart — the simulation analogue of the "no duplicate parent re-runs across restart" acceptance criterion.

Two steps remain to move from substrate-invariant validation to real-world evidence, with the analysis fixed in advance.

1. Real claim-audit (the key missing measurement). Run the fixed worker on a modest number of real tasks (about 30–50). For each instance, log the worker's done-claim, whether it attached evidence, and the orchestrator grader result, then report P(grader fail | worker says done) — the real-world false-claim prevalence the simulation only parameterizes. A ready-to-run harness (released) drives a real LLM worker via an external agent CLI on self-contained bug-fix tasks with unit-test graders; it was validated offline but needs the LLM CLI to execute.
2. At-scale software-repair study with a fair repair-budget design. Measure a real claim-trusting baseline (replacing the fixture), then run all four cells — {claim-trusting, VGO} × {no-repair, k=2} — at 30 then about 300 instances with the model fixed, reporting both false-acceptance metrics and resolution with Wilson CIs.

Decision rule, pre-specified: the primary outcome is the false-acceptance delta; the result does not depend on a capability uplift; an inconclusive resolution interval is reported as inconclusive — the system escalates such cases by design — rather than over-claimed.

Verification belongs in the substrate. The accept/reject decision must be made by something other than the worker that produced the artifact. Orchestrator-owned scoring with a bound evidence envelope makes "done" mean "independently graded as done."

It is cheapest and least corruptible when the gate is defined before the work. Encoding acceptance criteria during harness design is a precommitment: the worker bonds itself to a definition of success it cannot later edit, and its output is pulled toward that committed target rather than toward a convenient story. The orchestrator-owned grader at the regression gate is then simply the execution of that up-front encoding. Self-critique can still help repair; it just never gets to define the bar. The practical discipline that follows is to push as much of "done" as can be proven into the initial harness, precisely because that is the one moment the worker has no result to protect.

The honest reading of the simulation: under VGO's acceptance rule, zero unsafe merges is close to an invariant. The simulation's job is to confirm the implementation preserves it under repair, routing, unsupported evidence, and restarts, and to quantify what claim-trusting costs across the (c, f) grid — including that false acceptance is much more frequent among done-claims than per task, and that trusting claims wastes repair budget. None of this estimates how often real agents fabricate; that is the pending claim-audit.

False acceptance is distinct from low capability. A weak agent that honestly fails is safe to automate around; a confident agent that falsely claims success is not. Measuring and eliminating false acceptance is a prerequisite for autonomy, separate from raising capability.

The defensible claim: VGO does not make coding agents more capable. It makes acceptance decisions auditable — under a verification-gated substrate, a worker's unsupported success claim cannot by itself cause a merge, because acceptance requires independently produced grader evidence bound to the artifact.

• Simulation validates an invariant, not real prevalence. It varies false-claim propensity as a parameter; it does not estimate how often real coding agents emit unsupported success claims.
• No end-to-end efficacy yet. The pilot is N=5 and a plumbing check; there is no measured baseline or at-scale comparison.
• Verification-environment trust. VGO assumes the orchestrator controls a trustworthy grader invocation and evidence log; it does not address compromised runners, flaky tests, poisoned graders, or artifacts that exploit the grader.
• Latency and cost. Independent verification adds wall-clock time, compute, and operational complexity, not quantified here.
• Benchmark-to-production gap. Passing a benchmark grader is not production safety; VGO prevents unverified acceptance relative to a grader, not relative to the full intended behavior.
• Single benchmark and single model class. Generality across benchmarks and model classes is unestablished; the provider abstraction makes it testable, but it is not yet tested.

As coding agents gain autonomy, the binding constraint shifts from "can it do the task?" to "can we believe it when it says it did?" Verification-Gated Orchestration answers the second question by moving acceptance authority out of the worker: a gated lifecycle whose regression gate independently grades the real artifact and binds an evidence envelope to each accepted attempt, plus mechanisms that route — not fabricate — unsatisfied gates and flag unsupported pass claims.

Two false-acceptance metrics, the no-unverified-merge invariant validated across a wide range of capability and false-claim propensity, and a verification × repair ablation showing that verification removes false accepts at zero capability cost while repair moves resolution — together they support one architectural thesis: trustworthiness is a substrate property. Don't trust the agent's claim; verify the artifact.

Software-engineering benchmarks. SWE-bench evaluates whether models resolve real GitHub issues by editing repositories so the project's tests pass; SWE-bench Verified provides a 500-instance human-filtered subset. VGO uses this benchmark family as a target substrate but studies a different question: whether accepted success claims are independently verified, not whether more instances are resolved.

Coding-agent scaffolds. SWE-agent shows that agent–computer interface design materially affects autonomous software-engineering performance; AutoCodeRover combines LLMs with code search and program-analysis techniques. VGO is complementary: it can wrap such agents and governs acceptance rather than patch generation.

Self-feedback and iterative repair. ReAct interleaves reasoning and acting; Self-Refine and Reflexion show that self-feedback and verbal memory can improve task behavior. VGO treats self-critique as useful for repair but insufficient for acceptance: the accept/reject decision is made by an external verifier, not by the worker that produced the artifact.

Positioning. Existing benchmark reporting emphasizes resolution, and some comparisons already control the harness to compare models under a shared scaffold. The distinctive contribution here is a substrate-level trustworthiness axis — whether accepted or merged artifacts actually pass independent verification — and a mechanism that enforces it.

• Jimenez, Yang, et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770.
• OpenAI. Introducing SWE-bench Verified. Technical report, 2024 (500-instance human-filtered subset).
• Yang, Jimenez, et al. SWE-agent: Agent–Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. arXiv:2405.15793.
• Zhang, et al. AutoCodeRover: Autonomous Program Improvement. ISSTA 2024. arXiv:2404.05427.
• Yao, et al. ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023. arXiv:2210.03629.
• Madaan, et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. arXiv:2303.17651.
• Shinn, et al. Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arXiv:2303.11366.
• Wilson. Probable Inference, the Law of Succession, and Statistical Inference. JASA, 1927.
• Brown, Cai, DasGupta. Interval Estimation for a Binomial Proportion. Statistical Science, 2001.
• NIST/SEMATECH. e-Handbook of Statistical Methods, §7 (Wilson method for proportions).

The substrate simulation and the verification × repair ablation use master seed 20260615; 40 seeds × 5,000 tasks = 200,000 tasks per cell; Wilson 95% CIs. Released code includes the simulation, the dual-metric and 2×2 script, and the offline-validated external-agent-CLI bridge for the real claim-audit.

Per-instance acceptance assertions: exactly one first-pass canonical row; zero-or-more repair-assisted rows; an official-scorer evidence envelope on every accepted attempt; zero duplicate parent re-runs across an induced orchestrator restart; and inconclusive resolution intervals that escalate with the interval and baseline stated rather than auto-passing.