Make a benchmark measure the submitted program.

The problem

The generated harness ran the reference solution while grading a candidate. A passing report could therefore hide a broken submission. Optional 0x prefixes were also decoded incorrectly, and reports inserted success claims instead of deriving them from attempts.

My contribution

I built a packed-decimal benchmark with independent fixtures, bounded candidate execution, versioned result records, and reports generated from those records. The release narrows its supported task so the measurement foundation can be checked directly.

How it works

1. Known answers + seeded cases
2. Candidate process
3. Independent verifier
4. Per-case result records

Decisions that shaped the release

• Expected answers come from known integers and independent fixtures; the grader never runs the reference to construct a candidate answer.
• Every declared case runs. Incorrect output, execution errors and skipped cases keep separate counts.
• Untrusted candidates run inside a dedicated Linux container with protected truth data and resource limits. Trusted host controls are labeled separately.

Measured results

Deterministic controls establish grader behavior. These are not language-model performance measurements.

A failure worth inspecting

An unsigned decoder returns the right digits with the wrong sign.

The candidate exits successfully, but 123D must decode to -123. The verifier records a failed case rather than borrowing the reference output.

{
  "id": "negative-d",
  "input": "123D",
  "expected": "-123",
  "actual": "123",
  "status": "failed"
}

Limits and next questions

• Frozen cases establish regression behavior; they do not establish generalization or resistance to memorization.
• No frontier-model or coding-agent quality was measured. Harbor integration is separately pinned and smoke-tested.
• The implementation remains private. This page contains sanitized control evidence.