Skip to content

test: no port or environment races in the test harness - #377

Merged
jrosskopf merged 1 commit into
mainfrom
fix/test-harness-races
Nov 18, 2026
Merged

jrosskopf merged 1 commit into
mainfrom
fix/test-harness-races

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

Under load, the pre-push gate (scripts/check.sh) failed 3 of 4 times, each time on a different test unrelated to the change being pushed (embedded_denylist, mcp_era, twilio_whatsapp_template). That tempts --no-verify, which CLAUDE.md forbids. There were two causes.

What

  • Port race. free_tcp_port used to bind 127.0.0.1:0, release the port, and let the spawned triton bind it later. Under load, another socket took the port in between, often an outgoing connection, since the kernel gives those ports from the same range. Ports now come from 20000–32767, below Linux's ephemeral range (32768–60999), through one pid-seeded counter per process with a bind check. The spawn retry loop stays as the backstop. A new unit test checks that 300 ports all come from the range with no repeats; it failed before this change.
  • Environment race. embedded_denylist must prove that an embedded host reads TRITON_DENIED_PRINCIPALS with no wiring, so it calls set_var. Every triton that the shared it binary spawned meanwhile inherited a revoked principal or a pairing restriction. The test is now its own [[test]] binary, and its tests share a tokio mutex so their env changes don't overlap.
  • doc/realizations.md §7: the port-race entry is updated, and there is a new entry on env-mutating tests.

Evidence

  • embedded_denylist: 7/7, three runs in a row. Clippy is clean.
  • it: 757/757, three runs in a row.
  • Two copies of it at once: no AddrInUse and no denylist leak. The only failure is rasterizer::concurrent_requests_dont_deadlock, a CPU-bound check that 20 renders finish within 10 s. It slips under doubled load and is out of scope here.
  • Pre-push scripts/check.sh: 9/9 steps at default test parallelism, where it used to need RUST_TEST_THREADS=4.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

- free_tcp_port hands out ports from 20000-32767, below Linux's ephemeral
  range (where every outgoing connection also takes a port), through one
  pid-seeded counter per process with a bind check; the spawn retry loop
  stays as the backstop. Under a loaded machine (load ~175) the old
  probe-release-rebind on 127.0.0.1:0 failed a random spawned-binary test
  with AddrInUse after its 5 retries.
- embedded_denylist is its own test binary: its set_var of
  TRITON_DENIED_PRINCIPALS / TRITON_PAIRING_TOOLS /
  TRITON_AUDIT_REJECT_WINDOW_SECS leaked into every triton the shared 'it'
  binary spawned meanwhile (random mcp_era / twilio failures). Its tests
  share a tokio mutex so their env changes do not overlap.

Stress: the full 'it' binary 3x green; two copies at once show no
AddrInUse and no denylist leak (only the CPU-bound rasterizer
20-renders-in-10s check slips under doubled load).
@jrosskopf
jrosskopf merged commit cbc01b2 into main Nov 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant