Skip to main content

Acceptance receipts

A grader awarded full marks to code that did nothing.

On a public Terminal-Bench task, code with no implementation passed all 60 required tests. The harness counted skipped tests as passed.

The project's own collaborators documented the skip-as-pass defect in Terminal-Bench issue #2086. It was fixed upstream on 2026-10-22.

Northset reproduced it and verified a fix.

The task's current version rejects that submission.

The kept counterexample is useful as a regression check rather than as an open defect.

Terminal-Bench issue #2086
See the receipt Request a scope

What a receipt checks

For every task in your delivery, run through the Harbor harness your receiver already uses.

Identity.

Every file is hashed. The receipt names exactly what was checked, and in which configuration.

Reference.

Your solution passes, twice.

Null.

A submission that does nothing fails.

Inventory.

The tests that decide the reward actually ran. A skipped or erroring test never counts as a pass.

Known shortcuts.

Candidates that should fail are rejected. Correct alternatives you name are accepted.

Repair.

After you fix something, the same checks run again and the receipt shows what changed.

What you receive

Read the sample receipt

This sample came from the Terminal-Bench task html-js-filter.

This sample is inconclusive because the reference run wrote no reward file on this host.

Published measurement: verifier regression protection on vf2-speedup-networkx

The run-scoped temporary paths and generated job name were collapsed to <run> and <job>.

The raw evidence files named in the sample are not published here.

Acceptance receipt

1 items submitted: 0 CHECKS_PASSED, 0 FINDING, 1 INCONCLUSIVE, 0 UNSUPPORTED, 0 INTAKE_ERROR.

Harbor 0.24.0; tool 0.1.0; Darwin arm64. Unreviewed local evidence.

html-js-filter: INCONCLUSIVE

Submitted: {"git_url": "https://github.com/harbor-framework/terminal-bench.git", "commit": "e67a2cb7db4d9795c212909b3395d8c22ad60aec", "path": "tasks/html-js-filter"}.

Subject SHA-256: b5ebbdb5e8eb272b123b984cde6c295ec8340e5636387d3664e3f7bad4575f79.

Checks: intake completed; reference blocked; null completed; inventory blocked; candidates not_applicable; claims blocked.

Rewards: reference [None, None]; null [0.0, 0.0].

  • reference: Harbor trial error: RewardFileNotFoundError: No reward file found at <run>/jobs-reference/<job>-html-js-filter-reference/task__GNosB3W/verifier/reward.txt or <run>/jobs-reference/<job>-html-js-filter-reference/task__GNosB3W/verifie
  • reference: Harbor trial error: RewardFileNotFoundError: No reward file found at <run>/jobs-reference/<job>-html-js-filter-reference/task__MG8LRxY/verifier/reward.txt or <run>/jobs-reference/<job>-html-js-filter-reference/task__MG8LRxY/verifie
  • Claim could not be checked: oracle_passes

Limits: This receipt checks only the submitted task bytes, supplied controls and recorded conditions. It does not establish full task correctness, difficulty, privacy, protected verdict channels or behavior on another platform. Candidate verdicts are compared with the submitter's stated expectations; no independent behavioral witness is run. Test reports and rewards come from the task's verifier and can themselves be influenced by submitted code. No independent review, signature, dependency pinning audit or on-chain acceptance is established. Some report formats do not distinguish xfailed or xpassed; unavailable counts are null. No known-wrong or alternative candidates were replayed.

Evidence: item record, intake.

Order without a call

Send five answers to aeziz@northset.ai.

  1. Which task or tasks, at which revision?
  2. What decision is waiting on the result?
  3. When do you need it?
  4. Who is buying?
  5. Can you share a representative task under suitable access terms?

Within one business day you get a fixed scope, fee and delivery date, or a prompt decline.

Payment is by invoice. The fee is the same whatever the result.

Email five answers