Repository navigation
Conversation
live777 can now act as a headless Wayland display server: GUI applications connect to a Wayland socket owned by the source and render straight into a WebRTC stream (H.264), no physical display or capture loopback in between. - smithay headless compositor on a dedicated calloop thread, SHM-only (wl_compositor/wl_subcompositor/wl_shm/xdg-shell/ wl_output/wl_seat/wl_data_device), CPU alpha-over compositing onto a fixed-size canvas, damage-driven pacing (idle desktops encode nothing) - test pattern until the first client frame, then composited frames permanently (frame_mux over an mpsc ArgbFrame seam) - OpenH264 encode (screen-content mode, in-band SPS/PPS, PLI/FIR forces an IDR, set_bitrate recreates the encoder), published through the existing MediaPacket::RtpPacket bridge path - config: url = "wayland://<socket>?width=&height=&fps=&bitrate=" behind a new Linux-only source-wayland feature (not in source-all) - tests: protocol-level compositor tests plus a WHEP end-to-end that decodes a client's solid-red frame (rsmpeg); docs (en/zh), config sample, CHANGELOG and AGENTS.md updated
Found by driving the compositor with real weston clients: after a static desktop's single commit no new frames were encoded, so a subscriber joining (or resyncing after loss) waited forever for a decodable keyframe. - PLI/FIR now also nudges the compositor to repaint and re-emit the current canvas, not just the encoder: the RTCP task sets a repaint_request flag consumed by the compositor's frame timer - once the compositor owns the stream, an idle keepalive repaints the canvas every 2 s - both paths mark the emitted ArgbFrame force_keyframe so the encoder turns it into an IDR instead of an undecodable P-frame - regression test: compositor_repaints_on_request_and_keepalive Verified live: weston-terminal attached to WAYLAND_DISPLAY=live777-demo, WHEP-pulled with whepfrom and decoded with ffmpeg — the terminal window lands in the WebRTC stream.
The wayland compositor was SHM-only, so every GPU-accelerated client
(Qt Quick, browsers, mpv, Gazebo GUI) failed: their EGL stack needs
zwp_linux_dmabuf_v1 to submit frames.
- new gl.rs: GlPipeline brings up GBM (/dev/dri/renderD128, then card0,
LIVE777_WAYLAND_DRM_DEVICE overrides) -> EGL -> smithay GlesRenderer
and proves render+readback with a probe frame; any failure logs and
keeps the CPU blit path. The renderer is !Send, so it lives on the
compositor's event-loop thread inside the frame timer.
- produce_frame() renders with GL when the pipeline is up: toplevel
buffers import as textures (ImportDma for dmabufs, ImportMemWl for
SHM with per-commit damage), draw bottom-to-top into one persistent
offscreen target (Offscreen<GlesTexture>), read back via
ExportMem::copy_framebuffer as BGRA (matches the canvas byte order);
the frame is rendered Flipped180 so glReadPixels' bottom-up rows come
out top-down. Buffer positions share window_buffer_pos() with the CPU
blit, and the existing pixel tests pass through the GL path on
GPU-equipped hosts.
- zwp_linux_dmabuf_v1 is advertised only when the renderer initialized
(DmabufState + default feedback built from the render node and the
renderer's import formats); dmabuf_imported sanity-checks against the
advertised set (the renderer is unreachable from protocol dispatch),
the real import happens per frame. wl_buffer.release keeps flowing
from surface-state replacement; the renderer's dmabuf texture cache
is dropped when a toplevel or buffer dies.
- smithay gains backend_egl/backend_gbm/renderer_gl features.
Verified on i915/Mesa: compositor_composites_real_clients runs through
the GL path ('GL compositing active on /dev/dri/renderD128').
- dmabuf_global_is_advertised_exactly_when_gl_is_up: the registry shows zwp_linux_dmabuf_v1 iff the compositor reports GL active (a client's first registry roundtrip only completes after the event loop runs, which is after GL init, so both are final); with an accessible DRM device GL must initialize, without one the compositor must fall back to CPU compositing. Safe on GPU-less CI. - dmabuf_client_reaches_the_canvas_and_buffers_cycle: an in-process client allocates two linear XRGB8888 GBM buffers on the same DRM device, imports them via zwp_linux_dmabuf_v1 params/create_immed and double-buffers commits; asserts canvas placement matches the CPU path (centered window, correct borders), content switches land (red -> green -> blue), wl_buffer.release flows (replaced buffers are released, no double-buffer starvation), and disconnect restores the background. Skips cleanly without a dmabuf global or DRM device. - CompositorHandle exposes gl_active() for the tests; the event-loop thread sets it right after renderer init. - gbm 0.18 joins the Linux dev-dependencies for the test client.
The wayland source is no longer SHM-only: with a working GPU/EGL driver the compositor advertises zwp_linux_dmabuf_v1 and GL clients render straight into the stream; without one it falls back to CPU compositing (SHM clients only). Update docs/guide/wayland.md (+ zh twin), the AGENTS.md architecture paragraph (gl.rs pipeline, import/readback design, !Send placement), and amend the unreleased CHANGELOG entry.
The GL compositing path rendered with dst_transform = Flipped180 on the assumption that the PBO readback (ExportMem::copy_framebuffer) produces bottom-up rows that need the pre-flip. It does not: the projection into the offscreen target and the readback are orientation-consistent, so the extra flip made every client — dmabuf (Gazebo GUI) and SHM alike — show up vertically flipped. Render with Transform::Normal instead. It slipped through because every existing content check was vertical-flip-invariant (solid colors, horizontal splits). Lock the orientation down with asymmetric top/bottom band patterns: - shm_buffer_vertical_orientation: a red-top/blue-bottom SHM buffer must land unflipped (runs through whichever path is active). - dmabuf_client_reaches_the_canvas_and_buffers_cycle: the double-buffer test now fills the first buffer with the band pattern and asserts the top band stays on top at exact positions, again after a repaint. - dmabuf_orientation_matches_y_invert_flag: a 4-phase matrix over (content order x y_invert flag) asserting the protocol semantics — output flips iff the client set y_invert. Verified live on i915/Mesa: weston-simple-egl's triangle renders upright (apex up, red bottom-left) and centered, over WHEP -> whepfrom -> ffmpeg.
…+ configs) A self-contained handoff for the PX4 SITL + Gazebo Jetty on Ubuntu 26.04 setup used to validate the wayland source: the px4-jetty image Dockerfile, the one-command run script (forest world, x500_mono_cam, headless camera rendering via GstCameraSystem RTP -> live777 sdp source, Gazebo GUI as a wayland client of the compositor), the live777 config and static SDP, and a README covering runbook, interfaces, tuning knobs and every pitfall encountered during bring-up.
PX4's pxh shell spins at full CPU on a dead/closed stdin, redrawing the prompt as one never-newlined line at ~11 MB/s (26 GiB in 40 min on tmpfs). Feed px4 a held-open fifo instead: no spam, and a command channel stays open for 'commander takeoff' & friends.
cmake's default generator is Unix Makefiles; the plain `cmake -S . -B` call only produced build.ninja because the persisted build dir already pinned Ninja. On a cold checkout (e.g. after /tmp tmpfs wipe) ninja then fails with "loading 'build.ninja': No such file or directory".
The wayland compositor source now spawns an XWayland server by default (xwayland=1, xdisplay=N pins DISPLAY) and acts as its X11 window manager, so X11 applications render into the stream alongside native Wayland clients. Without an Xwayland binary on PATH the source logs a warning and continues Wayland-only. The built-in XWM is deliberately minimal for a headless virtual monitor: configure requests are granted as-is, fullscreen/maximize map to the canvas, last-mapped paints on top, and only the top window is activated (no input injection yet). X11 windows draw 1:1 at their root-relative position; the unified canvas stack is xdg toplevels, managed X11 windows, then override-redirect windows (menus/tooltips) on top. Buffers ride the existing SHM/dmabuf paths unchanged. Two gotchas worth noting: - CompositorHandler::client_compositor_state must special-case the XWayland client's XWaylandClientData (it is not our ClientState). - smithay's X11Source closure captures a calloop LoopHandle, Rc-cycling the event-loop registry, so run() removes every source it inserted at shutdown: the listener drop frees the socket file and the XWayland drop kills the server. One registry skeleton still leaks per compositor lifetime (smithay-0.7 limitation, bounded in practice). Verified end to end: xeyes + xterm over DISPLAY=:N, streamed through WHEP/RTP to a screenshot at the requested window positions, plus a compositor-level x11rb test (skipped without an Xwayland binary). X11 clients must paint on Expose — damage issued before Xwayland realizes a window is dropped server-side, as with any compositor.
The wayland source now also publishes the desktop's sound: with audio=1 (default) live777 taps the monitor ports of a PipeWire Audio/Sink node (configurable name filter via audio_sink=) and publishes a 48 kHz stereo Opus track (audio_bitrate=, default 64k). No daemon at $XDG_RUNTIME_DIR/pipewire-0 (or any handshake failure) degrades the stream to video-only with a warning; a sink appearing later is picked up on the fly. Design notes: - One thread owns a PipeWire main loop; the connect handshake runs the loop for a single roundtrip inside start() (5 s cap) so daemon availability is known synchronously, then the loop runs for good. Cancel is forwarded over a pipewire channel (quitting a foreign main loop is only safe that way); the watcher is a std thread because the audio thread has no tokio runtime. - Samples accumulate as F32LE and are cut into 20 ms Opus frames, one RTP packet each (PT 111), riding a new MediaPacket::RtpAudioPacket variant so the bridge injects into the audio track without payload sniffing; PeerForward gains inject_audio_rtp_packet. - get_audio_codec is gated on the video codec being known: the bridge adds tracks in a single pass once any codec is ready, and audio must not win that race or the video track would be missed. - Encoding uses the 'opus' crate (0.4, bundled libopus via cmake — the older opus-sys 0.3 line can no longer be downloaded from crates.io). Verified end to end: a 440 Hz tone played into a null sink decodes at the expected amplitude (RMS -21 dB) after live777 -> WHEP -> RTP -> ffmpeg, and graceful shutdown stops the capture thread cleanly.
Brings in the webrtc/rtc 0.21.1 upgrade (packetizer gained a now- parameter; the wayland video and Opus audio pipelines pass std::time::Instant::now() like the native encoded source) and the datachannel cable infrastructure (#500), which the upcoming input passthrough will ride. Conflicts resolved: AGENTS.md (kept both new entries), vitepress navs (kept Wayland Source and Cable), .config/nextest.toml (union of the binary filters), Cargo.lock (re-resolved).
a-wing
force-pushed
the
feat/wayland-source
branch
from
October 23, 2026 13:03
f47e298 to
b4bf83b
Compare
Opt-in per source (wayland://...?input=1, off by default): viewer input arrives as JSON on a datachannel labeled 'input' (undeclared cables fall back to the stream's anonymous default bus, so no cable config is needed), the bridge pumps the cable's to_publisher bus into the source's new StreamSource::input_sink(), and the compositor injects events into the seat over a calloop channel — hit-tested pointer focus across the unified xdg/X11 window stack, focus-follows-click keyboard focus, click-to-raise for X11 windows, wheel as wl_pointer axis frames. DomCode→evdev is a static table; xkb keycodes are evdev+8 at the smithay boundary. Browser side: player-core gains an 'input' option (creates the channel) plus attachInputCapture() (pointer/keyboard capture with canvas-coordinate mapping and per-animation-frame move coalescing); WhepPlayer exposes an 'input' prop and the standalone player an '&input' query param. Verified end to end: xterm on the compositor, a whepfrom subscriber with --cable-label input bridging UDP to the channel, and a scripted click+'hello'+Enter sequence visible in the streamed frame. A compositor-level test drives pointer enter/button/key through the seat into a recording wayland client.
Member
Author
|
Update: the branch now also has X11 client support via XWayland ( |
- workflows: install the wayland compositor source's system deps (libwayland-dev, libxkbcommon-dev, libpipewire-0.3-dev) for the ubuntu build and clippy jobs; the test job additionally runs Linux with source-wayland enabled and xwayland installed, so the compositor, X11, input and audio unit tests run in CI (they are headless: CPU compositing, no PipeWire daemon needed) — this also feeds codecov/patch. - clippy (rust 1.99 chunks_exact_to_as_chunks): switch the constant-size pixel loops to as_chunks_mut::<4>(). - biome: sort the player-core import in whep-player.vue. - tests: audio Opus packetizer cadence (20 ms frames, +960 per RTP timestamp), URL parsing for the audio_sink/input params.
- build(ubuntu): link needs libgbm-dev/libegl-dev/libgles2-mesa-dev (smithay backend_gbm + EGL), on the test job too (test binaries link). - clippy: three more chunks_exact_to_as_chunks sites in tests/wayland_source.rs (only compiled with source-wayland, which the all-features clippy leg covers). Verified locally: clippy with the no-webui+wayland feature set passes -D warnings; the CI test job from the previous push ran the wayland suite (compositor/dmabuf/X11/input/audio) green.
- CI test job (Linux): start a PipeWire daemon + wireplumber + a null sink before the coverage run, so the wayland source's audio capture path is exercised (the audio test skips cleanly without a daemon). - New unit tests: pattern generator (scrolling/counter/task lifecycle), XRGB→I420 converter (BT.601 red reference values), Opus over RTP cadence, and a daemon-gated PipeWire capture flow test. - tests/wayland_source.rs: the e2e subscriber now also opens an 'input' datachannel and, with input=1 on the source URL, drives the full cable→pump→seat-injection path (media decode asserted undisturbed).
Without it the sink node is destroyed as soon as pw-cli disconnects and the audio capture test finds no Audio/Sink. Verified the full chain locally: pipewire + lingering null sink + wireplumber -> Opus packets flow into the test.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft: Wayland compositor source (
wayland://)Summary
live777 can now run as a headless Wayland display server. A stream configured with a
wayland://source creates a real Wayland socket; GUI applications connect to it and render straight into a WebRTC stream (H.264), with no physical display, no X11/Xvfb, and no capture loopback in between.Driving use case: PX4 / Gazebo simulation streaming
The main motivation for this PR is streaming a PX4 SITL + Gazebo simulation out of a container to a browser. Two complementary paths were validated end-to-end on this branch:
gz-sim-gui-client, Qt6) connects to live777's compositor overzwp_linux_dmabuf_v1and renders through our EGL compositing path (i915 tested; llvmpipe works). GUI chrome, panels and entity tree all stream correctly.GstCameraSystem(recommended for real use): the camera sensor renders server-side (gz sim -s --headless-rendering+ PX4'sserver.config), the plugin encodes with GStreamer (x264enc,rtph264paywithconfig-interval=1) and pushes plain RTP/UDP H.264 to127.0.0.1:5600. live777 ingests it with the existing SDP-file source — to live777, the simulator looks exactly like an IP camera. FPV footage from the forest world verified through WHEP.The interesting finding for the project: with the compositor source plus the existing RTP/SDP sources, live777 covers both the "GUI as a stream" and "simulator as an IP camera" patterns with zero code on the PX4/Gazebo side.
What lands in this PR
liveion/src/stream/source/wayland/— new source module (featuresource-wayland, Linux-only, deliberately not insource-all):compositor.rs— smithay headless compositor on a dedicated calloop thread: wl_compositor / wl_subcompositor / wl_shm / xdg-shell / wl_output / wl_seat / wl_data_device, CPU alpha-over compositing onto a fixed canvas, damage-driven pacing (idle desktops encode nothing), PLI-triggered canvas repaint, 2 s idle keepalive IDR so late joiners always sync.gl.rs— EGL/GBM renderer (render node probing with env override, probe-composite fallback to the CPU path),zwp_linux_dmabuf_v1advertised only when GL is up, dmabuf import + SHM import through one uniform GL path, PBO readback into the existingArgbFrameseam.mod.rs/encoder.rs/convert.rs/pattern.rs— OpenH264 encode (screen-content mode, in-band SPS/PPS, PLI/FIR→IDR,set_bitratevia cheap encoder recreation), yuvutils BGRA→I420, RTP packetization through the existingMediaPacket::RtpPacketbridge path, test pattern until the first client frame (frame_mux).wayland-clientover the real socket), dmabuf client tests incl.y_invertorientation matrix, SHM↔GL parity by construction, plus the WHEP end-to-end test that decodes a client's solid-red frame with FFmpeg.Findings worth knowing (for follow-up work)
EglPBufferSupport::setConfigOptionthrows on the "Full Screen" option (present in both Garden/ogre-next 2.3.1 and Jetty/2.3.3). Gazebo's own troubleshooting docs recommend XWayland. GUI chrome works fine; the 3D scene needs either an upstream Ogre fix or the camera-sensor path (which is the better fit for live777 anyway).docs/note if wanted.docs/guide/wayland.md.Test plan
cargo nextest run -p liveion --features source-wayland— 151/151 (incl. new dmabuf tests)cargo nextest run -p live777 --test wayland_source --features source-wayland[,rsmpeg]cargo clippy --all-targets --features source-wayland -- -D warningsclean;cargo fmt --checkcleancargo check --workspace --features source-all,webui,recorder,cascadeunaffectedFollow-ups (not in this PR)