Published finding
The verifier's private log directory does not enforce its permissions on a host bind mount
Measured by Northset. Docker 29.7.2 on macOS, arm64, Docker Desktop. Harbor 0.24.0.
What happened
Running the public Terminal-Bench task html-js-filter through the official harness, two controls behaved differently.
Terminal-Bench commit: e67a2cb7db4d9795c212909b3395d8c22ad60aec
| Submission | Attempts | Result |
|---|---|---|
| The task's own reference solution (-a oracle) | 2 of 2 | No reward file written. Harbor raised RewardFileNotFoundError. |
| A submission that does nothing (-a nop) | 2 of 2 | Reward 0.0, written cleanly, with a complete test report. |
The reference failure comes from RewardKit, verbatim:
RuntimeError: Reward 'checks' failed: Criterion 'filter_blocks_xss' failed:
Worker could not install required protections: RuntimeError: Worker can access private path: /logs/verifier
That assertion is in the rewardkit package, in src/rewardkit/_worker_bootstrap.py.
The worker drops to an unprivileged uid before running submissions.
It requires that it cannot read, write or execute the verifier's private paths.
It is a correct check and it fails closed.
The controls differ because of when the worker starts. Without a submitted file, the criterion stops at: filter.py does not exist.
It never starts a worker, so it records 0.0. With a file present, the worker starts and the assertion fails.
The task produces no score at all.
Why the assertion fails
Harbor bind-mounts the verifier log directory from a host path, as mounts of type bind.
The code is in compose_env.py, under environments/docker, in legacy_log_mount_env_vars.
On this host, permission bits on a bind mount are reported but not enforced. The same chmod on any other backing works.
| /logs backing | Mode reported | Unprivileged read | Unprivileged write |
|---|---|---|---|
| Host bind mount | 700, owner root | yes | yes |
| Named Docker volume | 700, owner root | no | no |
| tmpfs | 700, owner root | no | no |
| Container-local directory | 700, owner root | no | no |
On the bind mount, the probe overwrites reward.txt as nobody, changing it from 1 to 999.
Meanwhile, stat reports the file as mode 600 owned by root.
The same result appeared on three host directories: under /tmp, under the user's home, and under the harness's own temporary directory.
Reproduce with Docker
Run this script in Bash with Docker running. It needs only Docker and about 30 seconds.
#!/bin/bash
# Does the verifier's private log directory actually enforce its permissions?
# Harbor bind-mounts the verifier log directory from a host path
# (harbor/environments/docker/compose_env.py, legacy_log_mount_env_vars, type "bind").
# RewardKit's worker asserts it cannot reach that path before it runs a submission
# (packages/rewardkit/src/rewardkit/_worker_bootstrap.py, "Worker can access private path").
# This script checks whether the assertion can hold, on four backings for /logs.
# It needs Docker and about 30 seconds. It writes nothing outside its own temp dir.
set -u
IMAGE=ubuntu:24.04
PROBE='mkdir -p /logs/verifier
chmod 700 /logs/verifier
echo 1 > /logs/verifier/reward.txt
chmod 600 /logs/verifier/reward.txt
printf " reported mode %s owner %s\n" "$(stat -c %a /logs/verifier)" "$(stat -c %U /logs/verifier)"
if su -s /bin/sh nobody -c "cat /logs/verifier/reward.txt" >/dev/null 2>&1
then echo " unprivileged read: YES"; else echo " unprivileged read: no"; fi
if su -s /bin/sh nobody -c "echo 999 > /logs/verifier/reward.txt" 2>/dev/null
then echo " unprivileged write: YES (reward.txt now $(cat /logs/verifier/reward.txt))"
else echo " unprivileged write: no"; fi'
host=$(mktemp -d)
echo "host bind mount ($host)"
docker run --rm -v "$host":/logs $IMAGE bash -c "$PROBE"
rm -rf "$host"
echo "named volume"
docker run --rm -v harbor-logs-probe:/logs $IMAGE bash -c "$PROBE"
docker volume rm harbor-logs-probe >/dev/null 2>&1
echo "tmpfs"
docker run --rm --tmpfs /logs $IMAGE bash -c "$PROBE"
echo "container-local directory (no mount)"
docker run --rm $IMAGE bash -c "$PROBE"
What this means
Tasks whose verifier now runs submissions in a RewardKit worker
About ten task verifiers moved onto RewardKit on 2026-10-22 for the 4.1 release.
On a host like this one, those tasks cannot grade any submission that contains the solution file.
The failure is asymmetric and quiet. An empty submission scores 0 and looks healthy, because that path never starts a worker.
Only the reference, or any real attempt, errors.
Tasks whose verifier still runs submitted code in a forked child
Those have no equivalent precondition check.
On a host like this one, the unprivileged child can read and overwrite /logs/verifier/reward.txt. This is vector V3, Reward, in issue #2086.
The task's chmod 700 and chmod 600 give no protection here. They report success, so nothing in the run says so.
Measured: the capability at the mount level, as the table above shows.
Not measured: an end-to-end run of such a task on this host writing its own reward. Northset did not attempt that.
For a vendor or a reviewer
The same task, harness, commit and submission produce a score on one host and an error or unprotected reward path on another.
Nothing in the output names the cause.
That is the practical acceptance question: was the intended artifact graded, under the agreed conditions?
What this does not establish
-
Nothing about Linux hosts. Bind mounts there enforce permission bits in the ordinary way, so the assertion should hold.
Northset did not measure a Linux host. This should not be read as affecting Terminal-Bench's CI or any published leaderboard result.
-
Nothing about any task other than html-js-filter, which is the one run end to end.
The mount behaviour is independent of the task. The per-task consequence depends on when each verifier starts a worker.
-
No claim that RewardKit is wrong. It detected an unsafe condition and refused to proceed, which is what it should do.
-
No claim about which Docker Desktop file-sharing setting is responsible. Northset did not vary that setting.
Two things that would fix it
-
Make the condition visible before the run.
The check already exists in _worker_bootstrap.py. Run the same os.access probe during verifier setup, before any submission is staged.
That would turn a late asymmetric failure into one clear message.
The message: this host does not enforce permissions on the private log path, so the task cannot be graded here.
That also answers part of issue #2135. A regression check for verifier hardening needs a host precondition.
Otherwise it silently measures the host instead of the fix.
-
Do not put the private reward path on a host bind mount.
A named volume or a tmpfs enforces the same chmod correctly, as measured above.
Promote the reward file to the host after verification. This keeps output available without exposing the path while submitted code runs.
Provenance
Northset's acceptance receipt tool produced the end-to-end run. It drives Harbor's own harness.
It records identity, reference and null controls, the reported test inventory and rewards.
Raw job output, including both controls and the full tracebacks, is kept with this finding.
Northset has no affiliation with Terminal-Bench, Harbor or any task author, and was not paid for this work.