Skip to content

test: fix five recurring CI flakes - #65780

Open
codebytere wants to merge 5 commits into
nodejs:mainfrom
codebytere:test-deflake
Open

test: fix five recurring CI flakes#65780
codebytere wants to merge 5 commits into
nodejs:mainfrom
codebytere:test-deflake

Conversation

@codebytere

Copy link
Copy Markdown
Member

Five tests that keep showing up in the nodejs/reliability reports for main over the last two weeks, each root-caused and fixed in its own commit. Picked by counting distinct main-targeting PRs they failed in since 2026-08-21 and dropping everything that has since been fixed on main (the SEA, recursive fs.watch, Utf8Stream, http2 and dgram link-local ones) or already has a fix in review (#65755, #65767).

test where why it fails
parallel/test-child-process-fork-closed-channel-segfault AIX, Linux containers the second send() races the worker's exit; Linux/AIX report that as EPIPE on the write, which wasn't in the ignore list
client-proxy/test-https-proxy-request-invalid-char-in-url macOS only client-proxy test asserting zero proxy socket errors; the client resets the tunnel while the proxy is still relaying the upstream's TLS shutdown, logged as ECONNRESET
parallel/test-external-memory-reasonable-size SmartOS the child aborts with 1.2 GB resident and writes a core file, which outlasts the 300 s timeout; now runs under ulimit -c 0 like the other abort tests
test-runner/test-run-watch-emit-restarted macOS x64 (marked flaky) FSEvents reports the fixture setup writes after the first run has started, so the intentional write is the second restart; now only restarts after the write count, flaky mark dropped
wasi/test-wasi-pthread all (marked flaky on three platforms) the fixture's thread-spawn waits on the same value the worker stores on success, so an early notify is lost, and gives the worker only 1 s to instantiate; either way pthread_create() fails. Flaky marks dropped

Looked at and left alone: parallel/test-runner-run ("should support timeout" reporting uncaughtException on macos15-x64) doesn't reproduce under load on Linux and I couldn't pin it down from the code; sequential/test-debugger-pid and test-run-watch-cwd-isolation-none* need a Windows/macOS box; pummel/test-fs-watch-non-recursive on AIX is the host running out of AHAFS watchers.

Tests: each changed test passes locally (Linux x64), 16-48 repeats at -j16 for the watch and WASI ones, and the two parallel/ ones also under tools/test.py --worker.

Fixes: #64226
Refs: #59146
Refs: #54534


Disclosure: the code and this description were written by Claude Code, directed and reviewed by @codebytere.

test-child-process-fork-closed-channel-segfault sends a second handle
to a worker that exits on the first message and already ignores the
errors that race produces (ERR_IPC_CHANNEL_CLOSED, ECONNRESET,
ECONNREFUSED, EMFILE). When the worker is gone before the write reaches
the IPC pipe, Linux and AIX report the write itself as EPIPE instead,
which the callback rethrew. Treat it like the other "worker already
exited" errors.

Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
test-https-proxy-request-invalid-char-in-url is the only client-proxy
test that asserts the proxy logged no socket errors at all. Once the
last response has been read the client destroys its tunnel, and if the
proxy is still relaying the upstream's TLS close_notify at that point
the client answers with a reset, which the proxy records as ECONNRESET
on the CONNECT socket. That has been failing the test on macOS even
though every request was routed to the sanitized URL. Keep asserting
on other errors but leave connection resets out.

Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
test-external-memory-reasonable-size makes a child allocate 1.2 GB of
external memory so that V8's --external-memory-max-reasonable-size
check fires and the process aborts. The abort raises SIGABRT with all
of that memory resident, and on hosts that write core files (the
SmartOS CI machines in particular) the dump takes longer than the test
timeout, so the test has been timing out there since it was added.

Run the child under `ulimit -c 0` on POSIX, the same way
test-abort-fatal-error and common.childShouldThrowAndAbort() handle
their aborting children.

Refs: nodejs#65589
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
test-run-watch-emit-restarted expected exactly one test:watch:restarted
event, but it starts run({ watch: true }) right after writing the
fixtures into the watched directory. Watch backends that deliver events
with some latency, FSEvents on macOS most visibly, can still report
those setup writes once the first run is under way, which restarts it
and makes the later, intentional write the second restart. The test has
been marked flaky on macOS x64 for that reason.

Wait for the first drain, then require that the write is followed by a
restart and a drain, ignoring whatever the setup produced before it, and
drop the flaky marker.

Refs: nodejs#54534
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
test-wasi-pthread fails now and then on every platform with
"Assertion failed: r == 0 (c/pthread.c: main: 17)", i.e.
pthread_create() itself reporting an error. The fixture implements
`thread-spawn` by starting a Worker and blocking in
Atomics.wait(result, 0, 0, 1000) until the worker signals that it has
instantiated the module. Two things go wrong there: the worker signals
success by storing 0, the value the main thread is already waiting on,
so when the worker is quicker than the main thread its notify is lost
and the wait runs into the timeout; and one second is not always enough
for a Worker to start and instantiate a threads build on the slower CI
hosts (arm debug, Windows, macOS). Either way spawn() returns -6 and
wasi-libc turns that into a pthread_create() failure.

Wait on a sentinel value that neither outcome writes, and give the
worker a platform-scaled 30 seconds. Drop the flaky markers.

Fixes: nodejs#64226
Refs: nodejs#59146
Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/wasi

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to Node.js core tests and test infrastructure. labels Sep 4, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@codebytere
codebytere requested review from panva and trivikr September 4, 2026 08:29
@panva panva added flaky-test Issues and PRs involving tests that fail intermittently in CI. commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. labels Sep 4, 2026
@panva panva added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Sep 4, 2026
@codecov

codecov Bot commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.07%. Comparing base (7551e1d) to head (ce80b32).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #65780      +/-   ##
==========================================
+ Coverage   90.05%   90.07%   +0.02%     
==========================================
  Files         769      769              
  Lines      261396   261396              
  Branches    49631    49631              
==========================================
+ Hits       235399   235460      +61     
+ Misses      17034    16982      -52     
+ Partials     8963     8954       -9     

see 30 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. flaky-test Issues and PRs involving tests that fail intermittently in CI. needs-ci PRs that need a full CI run. test Issues and PRs related to Node.js core tests and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RuntimeError: unreachable

3 participants