Serialize PUN lifecycle operations and prevent concurrent startup and shutdown races

[GlazerMann]

I recommend merging #5816 first, then #5815, followed by #5813, and finally #5809. #5816 fixes the asynchronous FilesTest cleanup race that can otherwise make unrelated CI runs fail; #5815 fixes the MutationObserver timing race in the accessibility tests; and #5813 fixes the detached-DOM-node race in BatchConnect tests. Once those test-infrastructure races are merged, #5809 can be rebased and rerun against a more stable master, making any remaining failures much easier to attribute to the PUN lifecycle changes themselves.

nginx_stage has several operations that can start, stop, regenerate, restart, or remove the same user's per-user NGINX (PUN). These operations were not serialized, allowing concurrent requests to race over the PUN configuration, PID file, and passenger.sock.

One observed failure occurs when nginx_stage app stops a PUN and immediately starts its replacement. nginx -s stop can return before the old process has finished removing its Unix socket, causing the replacement process to fail with:

bind() to unix:/var/run/ondemand-nginx/<user>/passenger.sock failed (98: Address already in use)

Concurrent initial PUN requests can also race. Multiple Apache requests can observe that passenger.sock is absent and invoke nginx_stage pun for the same user at nearly the same time, causing duplicate initialization, pre-hook execution, configuration generation, and nginx startup attempts.

This change adds a common per-user lifecycle lock and keeps the relevant state checks and shutdown completion waits inside that serialized lifecycle.

Serialize PUN lifecycle operations

NginxStage::Generator now provides a common per-user lifecycle lock used by operations that can start, stop, replace, or remove a PUN:

• nginx_stage pun
• nginx_stage app
• nginx_stage nginx
• nginx_stage nginx_clean

The lock:

• uses an advisory exclusive flock;
• uses non-blocking polling;
• is bounded to 60 seconds;
• uses a monotonic clock;
• uses a stable lock file beside the PUN configuration.

The lock file is intentionally separate from the PUN configuration. Some lifecycle paths remove or recreate the configuration file; locking the configuration inode itself could allow concurrent processes to synchronize on different old and new inodes.

Prevent duplicate PUN initialization

PunConfigGenerator now holds the lifecycle lock across the complete initialization sequence.

After acquiring the lock, it checks whether the PUN is already running. A running PUN requires both:

• an existing Unix socket; and
• a PID file whose process is still running.

If another request has already completed startup, the later nginx_stage pun invocation returns without repeating configuration generation, pre-hooks, or nginx startup.

Performing this check after acquiring the lifecycle lock closes the scheduling window where two requests can both decide that the PUN is absent before either startup completes.

--skip-nginx remains config-generation-only behavior and is not suppressed by the running-PUN check.

Wait for shutdown before replacement startup

Before AppConfigGenerator starts a replacement PUN, it waits for the previous passenger.sock pathname to disappear while still holding the lifecycle lock.

The socket wait:

• is bounded to 30 seconds;
• uses a monotonic clock;
• polls at short intervals;
• treats dangling symlinks as existing socket paths;
• runs even when the PID file has already disappeared;
• fails instead of attempting an unsafe replacement start if cleanup does not complete in time.

When a PID file exists, the replacement sequence is:

1. acquire the per-user lifecycle lock;
2. verify that the PUN configuration still exists;
3. send nginx -s stop;
4. wait for passenger.sock to disappear;
5. start the replacement PUN;
6. release the lifecycle lock.

If no PID file remains, the same lock is held while checking for a lingering socket before startup.

Keep shutdown serialized through socket cleanup

Lifecycle serialization now extends through asynchronous socket cleanup for other shutdown paths as well.

For nginx_stage nginx:

• stop waits for the PUN socket to disappear before releasing the lifecycle lock;
• quit does the same;
• non-shutdown signals such as reload do not wait for socket removal.

For nginx_stage nginx_clean:

• active-user PUN shutdowns retain the lifecycle lock until successful nginx shutdown removes the socket;
• disabled-user cleanup sends SIGTERM while holding the lifecycle lock and waits for socket removal when the signal succeeds.

This prevents another lifecycle operation from starting a replacement PUN during the interval between signaling the old process and completion of its Unix-socket cleanup.

Failure behavior

If another lifecycle operation holds the lock longer than the configured timeout, the command fails rather than proceeding concurrently.

If a previous PUN socket remains beyond the shutdown timeout, replacement startup is not attempted.

nginx_stage app also verifies the PUN configuration after acquiring the lifecycle lock and fails before invoking nginx if another operation removed it.

Tests

Regression coverage includes:

• exclusive per-user lifecycle locking;
• bounded lifecycle-lock timeout;
• propagation of errors from protected operations;
• waiting for an existing socket to disappear;
• dangling socket symlinks;
• bounded socket-cleanup timeout;
• startup with no PID file but a lingering socket;
• prevention of startup after socket-cleanup timeout;
• missing PUN configuration during restart;
• stop → socket cleanup → start ordering;
• direct stop and quit shutdown completion;
• non-shutdown signals avoiding unnecessary socket waits;
• active-user and disabled-user nginx_clean shutdown completion;
• holding the lifecycle lock across complete PUN initialization;
• skipping initialization when a PUN is already running;
• retrying initialization when no running PUN is present;
• preserving --skip-nginx configuration generation.

Together these changes serialize PUN lifecycle operations, prevent replacement startup while an old Unix socket is still being removed, and prevent duplicate concurrent PUN initialization.