Skip to content

feat(attachmux): add the stdio multiplexing package for multi-session attach - #5168

Draft
ekalinin wants to merge 7 commits into
containerd:mainfrom
ekalinin:feat/attach-multiplexer
Draft

feat(attachmux): add the stdio multiplexing package for multi-session attach#5168
ekalinin wants to merge 7 commits into
containerd:mainfrom
ekalinin:feat/attach-multiplexer

Conversation

@ekalinin

@ekalinin ekalinin commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

First step towards multi-session nerdctl attach (#3570, #4374). Design discussion is in #3570 (comment).

This PR adds the pkg/attachmux package and nothing else.

Why

A container's stdio FIFOs cannot be shared: a FIFO has one queue, so two attach sessions each receive a random subset of the container's output. The fix is to give each container a single owner of its stdio and make every CLI session a client of it over a unix socket. This package is both halves of that conversation.

What is here

file
proto.go frame format: one stream byte, three reserved, big-endian length, payload
broker.go server side: fan-out to sessions, fan-in of stdin, bounded per-session queues
client.go client side, used later by attach, run -it and start -a
socket_supported.go unix socket transport, built for linux and freebsd
socket_unsupported.go stubs returning ErrUnsupported elsewhere
attachmux.go package doc, ErrUnsupported, SocketPath

The invariants this package exists to guarantee

Each of these is a bug in attach today, and each has a test:

  • The container never blocks on a session. Every session gets a bounded queue and its own writer goroutine; one that falls behind is disconnected. The goroutine feeding the broker never waits on a consumer. This is the structural fix for the deadlock class in nerdctl run with attached stdio stops draining container stdout at ~72 KiB, deadlocking the container (nerdctl 2.3.5, containerd 2.3.3, Kata runtime-rs) #5137 / Fix foreground stdio deadlock when the internal logging process stops consuming #5151.
  • With no sessions attached the broker keeps draining, so detaching leaves the container running normally.
  • Close(exited bool) only claims an exit when the caller knows there was one. The owner's stdio reaching EOF is not evidence: the same happens when the logging process is shut down while the container keeps running, and a client treats the exit frame as proof and stops streaming.
  • The exit frame does not travel through the session queue. The writer delivers it after the queue drains, so a session whose queue is exactly full still learns the container exited rather than seeing a bare EOF.
  • Close never waits on a blocked stdin write. It takes the descriptor out under the state lock and closes it after the sessions have drained; a write blocked on a FIFO the container stopped reading is evicted by the runtime poller rather than holding up shutdown.

Testing

go test -race ./pkg/attachmux/..., 23 tests. Run repeatedly (-count=5, several rounds) on linux/arm64 because a good half of them are about concurrency: slow-session eviction, Close racing with a detaching session, SetStdin racing with Close.

golangci-lint is clean for linux, windows, freebsd and darwin.

Notes for review

  • The transport is built for linux || freebsd only, matching where the owner process will be built in a later step. Probe returning success where no broker can exist would be a capability check that lies.
  • SocketPath is a pure function of the data store, namespace and container ID, deliberately with nothing recorded on disk. The reasoning, including why it cannot live in $XDG_RUNTIME_DIR, is in the issue comment linked above.
  • Left as a draft while the design question in Multiple attach should be supported #3570 is open: if the stdio owner ends up somewhere other than the internal logging process, this package stays as it is, but the steps after it change.

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
A failed write to the container's stdin returned from readLoop, whose
deferred dropSession took the whole session with it. A container that
closed its stdin and kept printing lost its terminal on the next
keystroke, and one that had just exited was reported as a broken session
instead of a clean exit.

Also from the same review pass:

- Write splits a chunk larger than one frame instead of dropping it, and
  refuses the control stream, which is the broker's own.
- EncodeFrame moved out of b.mu.
- Dial stops retrying a refused connection, which unlike a missing socket
  means nothing is listening.
- Session.Close is no longer reported by Stream as a lost connection.
- The stdin pump stops forwarding once Stream returns, so it no longer
  swallows the next keystroke.
- Serve's context watchdog no longer leaks when Serve returns first.
- Probe trims an over-long name instead of probing a longer path than a
  real socket.
- ErrUnsupported wraps errors.ErrUnsupported.
- SocketPath tests no longer sit behind a build tag, and the unsupported
  stubs are covered.
- RemoveSocket moves here from the logging step: it is package API.

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
…dropped

Four decisions from the review that are cheapest to settle while the
package still has no consumers.

A session's queue is bounded by bytes rather than frames. A TTY echoes
single keystrokes, so a frame count evicted a session that was a couple
of kilobytes behind, while a frame can be up to maxPayload, so a frame
count also put no real ceiling on memory in the logging process.

An evicted session is told so with a control frame before it is
disconnected. Otherwise falling behind is indistinguishable from the
broker dying, and the client reports both as a lost connection.

Listen removes the socket against its inode instead of its path. Two
brokers can briefly overlap for one container while the restart monitor
swaps tasks; the new one taking the path over is correct, but the old one
exiting afterwards would unlink the live socket and leave the new broker
on an inode nobody can reach.

A client now refuses only a broker newer than itself. A broker lives as
long as its container, so a client that is newer is the normal case after
an upgrade, and it has nowhere to fall back to.

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
The assertion spelled the separator out, so it failed on windows once the
test moved out from behind the build tag. That move was the point: these
cases never needed a socket, and windows is where the package is only the
unsupported stubs.

Signed-off-by: Eugene Kalinin <e.v.kalinin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant