Repository navigation
Testing harness #3
Copy link
Copy link
Open
9 / 109 of 10 issues completedLabels
wayfinder:mapWayfinder map issueWayfinder map issue
Description
Destination
Open-ended by design (dtinth): this map has no fixed endpoint — the direction is get the Jamulus codebase under test, and we chart more as we explore ("the mapping part of wayfinding is more useful than reaching a fixed destination"). Both unit tests and smoke tests are wanted eventually — different benefits, limitations, and confidence levels; separate PRs. Milestones along the way, not destinations:
- First upstream PR: opened — jamulussoftware/jamulus#3828, in upstream review.
- Smoke testing explored and eventually upstreamed (see its ticket).
Notes
- Tooling decided: qmake + QtTest, staying in-family with
Jamulus.pro(no CMake/Catch2 for now). - This fork is a free experimentation space; upstream PRs must stay very small.
- All upstream-facing activity (issues, PRs, review replies) is posted by @dtinth personally; the agent prepares code and draft text in this fork only.
- There are informal signals that upstream would welcome an automated testing harness.
- This effort carries execution: building the harness in the fork and preparing the upstream PR are in scope, not just decisions.
- Repo hygiene: no agent/tooling scratch files in the working tree.
- Skills for ticket sessions:
/tdd(test implementation),/prototype(harness spikes),/grilling+/domain-modeling(decisions). - Background: initial assessment and evidence + churn analysis in this issue's comments. Pre-map decisions recorded there: upstreaming is the goal, qmake+QtTest, first targets
protocol.cpp→util→buffer.
Decisions so far
- Scope the first upstream PR — PR opened: jamulussoftware/jamulus#3828 (harness + 19-case protocol suite + 6-job CI, +1029 lines); scope/design settled through comment-by-comment review on this map; milestone 1 reached pending upstream review
- Split test helpers into a shared header —
protocoltester.hprecedent set;GenTestFrameretired for golden-hex snapshots (space-separated) + mutation of production frames - Easy-to-read DSL: CProtocolTester fluent driver — kept; chains wrapped in
QVERIFY2+ richlastError()so failure messages identify the exact mismatch (anti-debugging-frenzy) - Linux coverage job — gcovr HTML artifact + summary table; baseline: protocol.cpp 34.3% lines
- CI test reports — junitxml from every job + zero-permission step-summary rendering; checks-write action documented as opt-in alternative
- CI design for the test suite — 4-job matrix green in 26–66s (Linux Qt 5.15 + Qt 6, macOS Qt 6.10.2 w/ Xcode 16.3.0 pin, Windows MSVC), hard
Totals: 41 passedgate on every platform; verified run 29926662716; triggers still need reshaping for upstream - Prove the harness: QtTest seed suite builds and passes in the fork — proven:
src/test/harness linksprotocol.cpp+util.cppwith no stubs/production changes, 41/41 green incl. ASan/UBSan; dtinth: keep (structure review deferred), wire up CI on all three platforms next - Upstream contribution requirements for a test/CI PR — anchor on still-open upstream Automatic Client/Server Smoke Test in CI jamulussoftware/jamulus#2428 (smoke-test spec, pre-agreed); AGPL 3.0+ short headers, Qt ≥ 5.12.2 + C++11, clang-format v14, test files under
src/, one tiny workflow modeled oncheck-json-rpcs-docs.yml, targetmain; small CI PRs merge friction-free - Track upstream #3819/#3821: GetValFromStream out-of-bounds guards — riskier to ride: fix semantics unsettled upstream; lead with
ParseMessageFramecontract tests + a Add bounds check iChanID before array access jamulussoftware/jamulus#3810 regression test instead, cite GetValFromStream uses ASSERT which is likely compiled out during release builds jamulussoftware/jamulus#3819 only as context
Not yet specified
- Exact contents and narrative of the first upstream PR (sharpens after the prototype and research tickets resolve).
- How upstream review feedback gets worked once the PR is open.
- Later suites and their order: buffer/jitter regressions, wider
utilcoverage, recorder golden-file tests, JSON-RPC surface tests. - Smoke-test upstreaming (parked side-quest per dtinth, 2026-07-23): the headless build fixes (three one-liners — verification uncovered a third, link-stage bug the old workaround masked) are upstream as issue #3825 and PR #3826 (fronted by dtinth, first upstream PR of this effort). Once merged, the eventual smoke PR ships zero production changes.
- Fault-injection integration tier on top of the smoke test (TCP fallback per upstream PR Support TCP for protocol messages jamulussoftware/jamulus#3636 — pljones asked there for test cases): needs CAP_NET_ADMIN (netns/iptables/tc) so local/privileged-only, not per-PR. A third party (mcfnord) reports having such a netns harness and offered to share scripts — unverified, to be assessed when this graduates.
- Whether to revive fuzzing (afl-fuzz found the 2020 protocol bugs) on top of the harness, and what becomes of
src/testbench.h.
Out of scope
- GUI and audio-backend (sound card) testing.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
wayfinder:mapWayfinder map issueWayfinder map issue
Activity
dtinth-claw commentedon Jul 22, 2026
I surveyed the codebase (as of upstream
main@ 49f4f8d) to see what "get it under test" would build on. Summary and a proposed starting plan below.Current state: no automated tests at all
Jamulus.pro). All*test*hits are vendored deps (libs/opus,libs/oboe).autobuild.yml) is build/package/release only. The only quality gates are clang-format/shellcheck/pylint (coding-style-check.yml) and a JSON-RPC docs consistency check.src/testbench.h(CTestbench): a manual fuzz tool that fires random protocol messages at a UDP port on a 1 ms timer. Not wired into anything (one commented-out line inmain.cpp), no assertions — but it provesCProtocolruns standalone.CONTRIBUTING.md's "Testing" section literally means "build it and try the feature."What's in our favor
QT += testlib(QtTest) costs nothing.headless,serveronly/SERVER_ONLY), so core code compiles without widgets.protocol.cpp(~3000 lines) —Create*Mes/Eval*Mespairs are pure byte-buffer round-trips, anddocs/JAMULUS_PROTOCOL.mdis a spec to test against. Highest value: this is the wire format, and where malformed-input bugs live.buffer.h/.cpp— jitter/ring buffers (CNetBuf,CNetBufWithStats,CConvBuf), pure algorithmic.util.h/.cpp—CVector,CFIFO,CCRC,NetworkUtil,CHostAddress, etc.src/recorder/— WAV + Reaper-project writers → golden-file tests.Main friction: monolithic qmake project with heavy header coupling (
global.heverywhere,SERVER_ONLYifdefs), so a test target has to pull in a cluster of core.cppfiles rather than link a tidy library.Proposed first increment
tests/with a qmaketests.pro(stay in-family with the existing build; no CMake migration) that compilesprotocol.cpp,buffer.cpp,util.cpp+ deps withSERVER_ONLY-ish config and links QtTest.CTestbench's generators, but with assertions),CNetBufput/get/overflow behavior,CCRC/NetworkUtilbasics.test.ymlCI job (ubuntu, Qt from apt) running the suite on PRs — separate fromautobuild.ymlso it stays fast.Open questions
@dtinth — blocked on question 1 (and 2 if you have a preference); happy to start on the harness + protocol round-trip suite on a branch here in the meantime.
dtinth commentedon Jul 22, 2026
Anything to back this up? Past issues, commits? Can you also check the issue tracker and past work on what has a large amount of churn, that would be a good first target?
Re. questions:
Would /wayfinder be a good fit for this? If /setup-matt-pocock-skills is needed, please remember information in local memory so it (1) persists between Git branches (2) keeps the Git repo clean of agent files (for now).
dtinth-claw commentedon Jul 22, 2026
Evidence for "malformed-input bugs live in protocol parsing"
Yes, this class has a documented track record here — and one member of it is open right now:
GetValFromStreaminprotocol.cppvalidates its arguments only withQ_ASSERT, which compiles out in release builds;CVectorusesstd::vector's uncheckedoperator[], so a truncated/malformed network message can read past the end of the buffer in the builds people actually ship.024ebb47(2020) — an ACKN message with a valid checksum but no payload caused an out-of-bounds read (every otherEval*checked size first; the ACKN special case didn't). Commit message: "Found by fuzzing with afl-fuzz."d545dbce(3.12.1, this year) — headless server crash from an out-of-bounds vector access inbuffer.cpp.iChanIDbefore array indexing.So both the 2020 fuzzing wave and the 2026 bugs are the same shape: network-derived values used as sizes/indices without release-mode guards. None of those fixes came with a regression test, because there's nowhere to put one — which is precisely this issue.
Also relevant precedent: upstream's own jamulussoftware#1413 "Run tests as part of build/CI" (2021) proposed running
testbench.hin CI; it produced a useful explanation from atsampson (it's a template fuzzer generating valid-but-random messages) but was closed in 2022 without any tests landing.Churn analysis (commits touching each file since 2021-07, excluding sweeps that touched >10 files)
util.cpp/util.hmain.cppclientdlg.cpp/serverdlg.cppserver.cpp/client.cppsocket.cppserverlist.cppsettings.cppprotocol.cppInteresting split: churn and bug-severity point at different files.
util.cpp/his the churn champion (it's a grab-bag everyone touches), whileprotocol.cppis low-churn but is where the security-class bugs recur — low churn there also means tests won't rot.Proposed first target
protocol.cppmalformed-input + round-trip suite — everyCreate*/Eval*pair round-trips, and truncated/corrupted variants must be rejected without OOB access. This directly gives GetValFromStream uses ASSERT which is likely compiled out during release builds jamulussoftware/jamulus#3819/Guard GetValFromStream against out-of-bounds reads in release builds jamulussoftware/jamulus#3821 the regression test it currently can't have, which also makes a great small first upstream PR (fix + guard + test + harness in one story).util.hdata-structure suite (CVector,CFIFO,CCRC,NetworkUtil) — covers the churn champion.buffer.cppjitter-buffer suite (regression for jamulus 3.12.1 headless crash with out-of-bounds vector access jamulussoftware/jamulus#3747's class) as the third.Next step on my side: harness branch in this fork (qmake + QtTest as agreed) with target 1 as the seed suite.
Re
/wayfinder//setup-matt-pocock-skills: neither is installed in my current environment, so I can't evaluate wayfinder's fit from here — if you install it on this box I'll pick it up. Your constraints are saved to local memory either way: agent/skill scratch files stay out of the git repo, and cross-branch state lives in local memory, not the working tree.dtinth commentedon Jul 22, 2026
I updated the skills in the dev box, please check.
dtinth commentedon Jul 22, 2026
I read this comment and this one already. Regarding behaviors that open PRs would fix but did not land in main yet, it is fine to add skipped tests for behaviors that proved failing and needs fixing, although do not do this needlessly. Just as a tool in your belt.
dtinth commentedon Jul 22, 2026
FYI, I see this in runner log, please take care of that too.
dtinth-claw commentedon Jul 22, 2026
Node 20 deprecation warning: handled. Verified run 29928172118 — all 4 jobs green,
Totals: 41 passedgate intact on every platform, and zero deprecation annotations (checked the full run log and the check-run annotations API).How:
ilammy/msvc-dev-cmdhas no Node 24 release (even its unreleased main still declaresruns.using: node20), so the action was dropped entirely and the Windows job now bootstraps MSVC directly in PowerShell — runningvcvarsall.bat x64in a subshell, diffing the environment before/after, and applying the changed variables (the same technique the action used internally). A plain inlinecall vcvarsall.batwas tried first and does not work on the currentwindows-latestimage (it ships a preview VS 2026 toolchain that leavesVCToolsInstallDirunset when called from a nested script) — that dead end and the workaround are commented in the workflow.Side benefit for the eventual upstream PR: one fewer third-party action to justify. Caveat noted in the workflow: if a future runner image goes back to a stable VS, the block may be simplifiable.
dtinth commentedon Jul 22, 2026
I have a question about the Node 20 deprecation issue. Is the action that we use and stop using... like... is it an action that the upstream is already using or we just start using it here and then migrated away?
135 remaining items
dtinth-claw commentedon Jul 24, 2026
Upstream checks ran on #3828 (someone approved the workflows) — 18 of 19 green, including all six unit-test jobs on upstream's own runners, every style check, and notably the macOS/Windows autobuild jobs, which exercise our modified
mac.sh/windows.ps1in their full production context — the strongest validation those script changes can get.The one failure,
Build for Linux .deb armhf (artifacts): infrastructure flake, not us. Evidence:linux_deb.shboth passed.Action needed is a re-run, which needs upstream write access — PR authors can't re-run workflows. Suggested reply for the PR (or ask ann0see directly):
dtinth commentedon Jul 24, 2026
As you recommend.
dtinth-claw commentedon Jul 24, 2026
Item 1 shipped as (a): commit
3b4612ae— a step parsesQT_VERSIONout of autobuild.yml's macOS main entry (legacy excluded viaARTIFACT_SUFFIX, ios can't match) and exports it; the duplicated pin is gone, and the parse fails loudly if it ever yields anything but exactly one plausible version (zero-match and multi-match both verified to hit the error path). Gate findings applied pre-push (pipefail-safe failure message, portablesed -E); bonus discovered during review: the old pin was invisible to bump-dependencies' Qt regex, so this also removes a latent silent-drift risk. Verification run 30079938743 in flight. Draft:jamulussoftware#3828 (comment)
dtinth-claw commentedon Jul 24, 2026
Item-1 fix verified: run 30079938743 all 6 jobs green, with the live log showing
Using Qt version 6.10.2 (from autobuild.yml)— parsed, not pinned. Tip3b4612ae, 15 commits. Every review item on jamulussoftware#3828 is now addressed; the draft above awaits your paste, then it's armhf re-run + re-review on the maintainers' side.dtinth commentedon Jul 24, 2026
Can you rebase our working branch (not PR branch) based on the current PR's branch?
Also please spike a refactor of the actions. This way we can try explore both at once: (1) exactly what the follow-up PR will look like, and (2) explore how "doing it as a sibling PR that autobuild.yml adopts first, then this PR consumes the action" might look like (in which we extract the common action and submit as a sibling PR).
dtinth-claw commentedon Jul 24, 2026
Both explorations dispatched in parallel (isolated worktrees):
archive/testing-harness-dsl, resetstesting-harnessontounit-test-harness(3b4612ae), and re-expresses the old suite's extra coverage (~27 more cases: the full round-trip family, CLPing incl. new CL-side tester wiring, extra goldens, invalid-body rejection) in the reviewed style. CI-verified before it reports.spike/setup-qt-action: commit A models the sibling PR (action + autobuild.yml migration, legacy-mac override, the windows.ps1 env question), commit B models this PR's consumption (parse step and per-workflow setup deleted). Verification via the unit-test workflow plus theautobuild/<branch>trick for autobuild's mac/windows jobs, with cache-key continuity analysis (does the shared cache survive the transition?) in a writeup, so both PR shapes land as concrete diffs + stats for you to compare.Reports here when each is green.
dtinth-claw commentedon Jul 24, 2026
Working branch rebuilt —
testing-harnessis nowd5e587f0: one commit on top of the PR branch (3b4612ae) porting the old suite's extra coverage into the reviewed architecture. Old DSL-era head preserved atarchive/testing-harness-dsl.ReceivedLog()one-liners, goldens for ClientID and CLPing,RejectInvalidMessageBodyviaSendRawBytes(accepted-at-frame-level + zero signals, same shape as the ACKN regression).WireCLMessages()+CLSentFrames()) — the CL path (CLMessReadyForSending/ParseConnectionLessMessageBody) is separate plumbing the tester had never needed before; wired only as far as the two CLPing tests require.ParseValidFrame/RejectTruncatedFrame— already covered byAcceptValidFrameand the truncation row inRejectInvalidFrame_data; porting them would duplicate, not add.FormatFloatprecision was empirically confirmed to absorb pan/gain's 1/32768 wire quantization — checked against actual production output, not assumed.Net effect: the working branch is again strictly ahead of the PR branch (base + 26 extra cases), ready to become the follow-up coverage PR after jamulussoftware#3828 merges. Composite-action spike still in flight.
dtinth-claw commentedon Jul 24, 2026
Composite-action spike done — both PR shapes exist as concrete, CI-verified commits on
spike/setup-qt-action(tip57e5bf1c), full writeup inresearch/setup-qt-action-spike.md.The two shapes, in numbers:
27732846):.github/actions/setup-qt+ autobuild.yml's mac/windows jobs migrated —+154/−18across 3 files.639d9657): unit-tests.yml drops its parse/cache/install/PATH steps for oneuses:—+14/−72. The version literal ends up in exactly one place: the action's default input.The spike earned its keep — two real bugs found, one nearly invisible:
powershell.exesilently drops an empty-BuildOption ''arg (caught by the unit-tests run).autobuild/*push running the full build/package matrix (failing run → fixed, 10/10 green). The fast workflow_dispatch loop never saw it. Fix: PATH export is opt-in (add-to-path, default false) — only unit-tests' bareqmakeneeds it; the build scripts use absolute paths.Key design decisions (details in the writeup): action-owned cache key (hashes action+scripts+version instead of hand-mirrored workflow keys — one deliberate cache repopulation at transition); two mutually-exclusive macOS setup steps because composite-action defaults are unreachable once
with:sets the key at all; Xcode pinning stays outside (iOS shares the job template with a different version).Top-priority companion finding:
bump-dependencies.yml's Qt6 job can neither reachaction.yml(file glob) nor match itskey: valuesyntax (regex — verified directly). Post-refactor, the version literal that matters would silently go stale on the next Qt bump unless that job is updated in the same PR. Flagged as must-ship-together, not deferrable.Recommendation: sibling-PR-first, with the
add-to-pathfix and the bump-dependencies update included. Verification loop lesson for that PR: dispatch runs aren't enough — it needs at least one realautobuild/*push before review.(Also:
autobuild/spike-setup-qtexists for re-running the full matrix; quirk noted — a NEW autobuild/* branch's first push doesn't trigger the workflow, only subsequent pushes do.)dtinth-claw commentedon Jul 25, 2026
New question from ann0see on PR jamulussoftware#3828 (not a finding — a placement question, so no fix pushed; your call):
jamulussoftware#3828 (comment)
dtinth commentedon Jul 25, 2026
I already made a reply (before I saw your message). What's your take?
dtinth-claw commentedon Jul 25, 2026
My take (full detail in the draft one comment up, verified against current main): your guess is close but the build system itself is actually indifferent —
test.proreaches the sources by relative path, which works the same fromsrc/test/or/test/. What does care is the sidecar tooling, which coverssrc/but not a new top-level dir:tools/update-copyright-notices.shline 56'sfindlist (android ios linux mac src windows tools .github) would skip/test/.make clang_format—Jamulus.proline 1194 filters to^(android|ios|mac|linux|src|windows)/. CI's style check scans everywhere regardless, so/test/files would still get flagged — just with no local formatting target to fix them.paths:filters usesrc/**.Plus mild precedent:
src/testbench.halready lives undersrc/. All three are one-line changes, so no strong preference here —/test/is fine if ann0see wants it; it just costs three extra config touches in the PR.Follow-up draft for the PR thread, picking up from your "will get back":
dtinth commentedon Jul 25, 2026
Let's move.
dtinth-claw commentedon Jul 25, 2026
On it — moving
src/test/→/test/onunit-test-harness(append-only), extending the three config spots, then shepherding CI. Will post the draft reply when green.dtinth-claw commentedon Jul 25, 2026
Move done and verified.
bc226653(append-only):src/test/→test/, +13/−12 across 7 files —test.prorelative paths (..→../src), runner-script.propaths, workflowpaths:filters gaintest/**, gcovr's now-dead--exclude src/test/dropped, and the two sidecar configs extended (update-copyright-notices.shfind list,Jamulus.proclang_format regex).coding-style-check.yml/.clang-format-ignoreverified as needing nothing (extension-glob / exclude-only).../srcresolution with a real out-of-tree qmake run.Draft reply to ann0see's comment:
@dtinth done — paste when ready?
dtinth-claw commentedon Jul 25, 2026
Copilot finding on the moved harness — convincing (TruncateBy documents that an oversize truncation yields an empty frame, so chaining into CorruptCRC is a legal path to indexing
[-1]), fixed under the standing protocol. Matrix 6/6 green.jamulussoftware#3828 (comment)