Make an agent's result inspectable.

The problem

A green unit suite can demonstrate runner behavior without proving that a model completed a task. A verifier can also give misleading results if the candidate can access its reference solution or if missing output and provider failures disappear from the denominator.

My contribution

I built a Harbor-based evaluation platform with versioned tasks, immutable campaigns, conservative request budgets, protected grading and evidence-linked reports. I then exercised the actual terminal workflow with four named free models through OpenRouter.

How it works

1. Validate and freeze the task
2. Run the isolated terminal agent
3. Export source and output
4. Grade in a separate verifier
5. Retain outcome and usage receipts

Decisions that shaped the implementation

• The agent image excludes the reference solution and hidden tests. The separate verifier checks the candidate's exported artifacts. Real Docker controls must pass before model evaluation.
• Task bytes, prompt/parser hashes, harness settings and repeats are frozen. Flint reuses the original calibration tasks rather than receiving regenerated versions.
• The provider credential stays in the host controller. Named free routes, zero-price ceilings and disabled paid fallbacks are checked before dispatch; actual provider usage must report exactly zero cost.
• Provider requests and whole agent trials have separate counts. Missing usage remains unknown, and automatic retries are disabled for this registered calibration.

Actual model results

24 calibration trials across four free models: 14 passes and 10 token/output-limit stops, all at $0. Each model attempted the same three synthetic task templates twice: scientific ETL, geospatial raster processing and repository debugging.

138 trial requests and four separate availability probes reported $0 in model charges. Two earlier protocol-development requests are excluded from those counts. The matched harness used Harbor 0.13.2, temperature zero, low reasoning and 8,192 output tokens per call.

A stop is not a verifier rejection

The stored category AGENT_TIMEOUT includes token and output-limit stops. Those count as unsuccessful completions. If the verifier never ran, the record has no independent correctness verdict; it does not establish that a submitted raster or program was graded wrong. This prevents a resource-limit outcome from becoming an unsupported claim about a model's scientific reasoning.

A public workflow you can inspect

The total is right, but the interval union is wrong.

The archived public repository-debug fixture demonstrates the grading boundary. Its starter keeps touching intervals separate. The existing authored reference repair changes < to <=, merging them as the contract requires.

Before: {"intervals":[[-11,-6],[-4,0],[0,3],[5,9],[9,14]],"covered":21}
After:  {"intervals":[[-11,-6],[-4,3],[5,14]],"covered":21}

Both totals are 21. The verifier rejects the starter and accepts the repair because it checks the complete output and API behavior. Six public behavioral controls pass.

This sample is a local author/reference replay with zero new model calls, not a Flint-generated answer or a new Docker eligibility result. The archived fixture predates the current separate-verifier Docker package contract. Raw calibration tasks and model traces stay private.

Limits and next questions

• Three calibration task templates and two repeats per model cannot establish a broad capability ranking or independent holdout performance.
• Flint ran on a different date. Matching tasks and harness settings does not eliminate provider or model-version drift.
• Token/output limits are part of the protocol; they were not expanded after observing these failures.
• Matched parent/child hardening and a separately developed frozen holdout remain future measurements.