Repository navigation
feat: remote files parity (fetch-on-miss, peer Files index, precache, Library) - #3872
damianvtran wants to merge 22 commits into
Conversation
The owner authorises every phase of a mesh file pull by the session's referenced-digest set; each phase re-read the whole transcript (65 reads of a 64 MiB journal for one 32 MiB pull). Memoise the set per transcript (path, size, mtime_ns, inode) in an LRU of 8 and scan in 4 MiB windows with a 64-byte overlap instead of read_bytes(). docs/design/remote-files-parity.md C1.
A peer-held session's attachment, waveform, supplement document and a remote subagent's (child route) image now resolve through one lookup order: this device's store, the session's replica (closes G1), the new bounded remote-files cache, then ONE pull from the owner over the relay (session_file_pull land=cache; closes G2). The owner still applies S1 confinement on every phase. - remote_files/cache.py: <config>/cache/remote-files in the store layout, LRU on .bin mtime under flock, budget desktop.remote_files_cache_bytes (LIVE, default 1 GiB; 0 = staging only); I4-1 keeps the file just landed. - remote_files/fetch.py: in-process single-flight per (owner, member, key), Semaphore(2) per owner, 20 s shielded wait (loopback max 6.0 s x 3) after which the pull keeps running and the route answers pending. - Statuses: 504 attachment_owner_unreachable / attachment_fetch_pending (+Retry-After: 2) are retryable; 409 attachment_peer_outdated and attachment_on_peer are permanent. Never 503 (reads as the plane closing). - Peer bytes are STREAMED from an fd opened before answering, with Range (206), via server/utils/byte_range.py lifted from the Library route. - i18n: the PR's nine new keys (incl. C4's Files copy) in en + fr, es, zh-CN, ru, vi, ur, hi; ledger audit rows pending an independent auditor. docs/design/remote-files-parity.md C2.
The relay must list a session's family for net_session_files (C4) without importing server.utils.desktop_sessions (+27 MiB RSS, +0.8 s, fastapi). Pure move of _persisted_children, _contained_child_dir, _prove_child, contained_descendants, FAMILY_MAX_* and SubagentChildUnavailable into session_files/family.py, re-exported under the same names; the files route's _family is lifted as family.family_for. Two route tests now patch the names where they live. docs/design/remote-files-parity.md C3.
A peer session's Files panel is the OWNER's own index, asked over the mesh:
- net_session_files (view, owner) runs session_files.service.list_files over
its journal with the shared family definition; cursors/revisions pass
through opaquely, paths leave only in display form, credential-shaped files
are never listed. Each page row adds remote_read / remote_unreadable under
S1 confinement (advisory: net_file_get re-checks every byte). A page over
one link record is trimmed and next_cursor re-minted (an oversize record
closed the link: 504 in the negative control).
- peer_session_files (local) forwards one phase after checking the owner's
mesh-file-index-v1; an older owner reads as peer_update_required.
- Viewer routes: …/files and …/user-attachments answer peers with remote,
origin_device (from the catalogue row), unsupported_reason, device_name;
new peer-only GET …/files/{id}/content (streamed, Range) and POST
…/files/{id}/download (~/Downloads or <config>/downloads, never overwrites,
O_EXCL temp + 1 MiB streamed copy + fsync + content-address check + link).
- pull_file land=cache skips the transfer when the bytes are already cached.
- features.remote_files = 1; DESKTOP_API.md 'Remote files'.
docs/design/remote-files-parity.md C4.
The catalogue poll (DesktopSessions.list with peers) hands its remote rows to remote_files.precache: active = reachable, unarchived, moved within 2 h, plus the last 8 remote sessions opened here (files_target's peer branch), at most 16 per pass, each re-listed at most once per 60 s and only when its mtime moved. A pass lists user attachments + image/audio/video file rows through the owner's index and pulls each readable file at most 32 MiB (the UI's MAX_DOWNLOAD_BYTES) into the bounded cache through the same fetch-on-miss; one precache pull per owner inside the owner's Semaphore(2), so an on-screen request always has a slot (negative control: with one slot it waited 1.54 s). A pass stops before exceeding the budget; budget 0 disables precache. docs/design/remote-files-parity.md C5.
The Library indexer's reconcile now runs library/remote.py: for each reachable
remote session whose catalogue mtime moved, the owner's Files index is listed
(peer_session_files, pages of 200) and its rows become occurrence rows in the
SAME tables, so grid/facets/search/dedup need no second path. Metadata only.
- schema v3: idx_assets.remote/remote_device/remote_device_name,
idx_occurrences.origin_device/origin_device_name/remote_file_id,
idx_sessions.origin_device/misses (a v2 index rebuilds once).
- identity.remote_key r:<device>:<session>:<display path> with path NULL, so a
remote asset never resolves to a file on this disk (I5-1); digest rows keep
a:<digest>, so bytes held locally and remotely are ONE asset.
- resolve_content answers content_remote for peer-only bytes; ONLY the
…/content route fetches (remote_session_files.library_content). List,
detail and thumbnail never dial (I5-2). thumbnail_source reads the remote
cache; a digest asset's liveness stamp is a:<digest>, so evicting the cached
source keeps its thumbnail.
- Sessions absent from two ANSWERED catalogue reads are dropped; an
unreachable device's rows are kept; a session that became local is left to
the local scanner. LibraryRow.remote = {device_id, device_name}.
docs/design/remote-files-parity.md C6.
Agent review — round 1Reviewer: reviewer (lopdev team, independent; served model anthropic/claude-opus-5-5, no fallback) Verdict: changes requested. 2 MAJOR, 3 MINOR, 1 NIT. The authority reuse, streaming and status mapping are solid; both majors have a small fix. What I verified (no finding)
FindingsR1-1 · MAJOR · security: an owner-supplied digest and member id become viewer filesystem paths
R1-2 · MAJOR · precache passes are not gated when opened through
R1-3 · MINOR · the eviction race is handled only on the transcript route
R1-4 · MINOR · a long CJK file name cannot be downloaded
R1-5 · MINOR · remote Library content is served inline with a peer-supplied type
R1-6 · NIT — single-flight is keyed Tests (run here, bounded, isolated HOME/config, CMUX stripped)
Not reviewed (other streams): i18n catalogues, shipped.json, UI #1880. |
i18n audit — round 1Reviewer: i18n auditor plus English copy review (lopdev reviewer role). Instance Verdict: clean. There is no BLOCKER or MAJOR. Five MINOR and two NIT findings are below. Six pairs can be bound now. Eight bind once the auditor-supplied corrections in I1-4 are applied byte-exactly and get a second independent read, as in the #3440 precedent. The en findings (I1-1, I1-2, I1-3, I1-5) can be deferred. If any of them is taken, the 7 Mechanical checks (all 14 pairs, at head and at the corrected bytes): 0 problems
FindingsI1-1 — MINOR — en: the 504 offline sentences promise an automatic load, and they don't match the UI's shared offline sentence.
I1-2 — MINOR — en: I1-3 — MINOR — en: I1-4 — MINOR — locales: glossary and style-guide deviations, measured against the existing catalogue. I give byte-exact corrected values (full table below).
I1-5 — MINOR — en: the N1 — NIT — en The rest of the en copy is honest and actionable. The 409 sentences say why and are final. Out of scope, pre-existing: the English Ledger: 14 receipts and the bind commands
I1-4 corrected values (17), byte-exact
|
QA report — round 1Verdict: PASS. 40 of 40 matrix rows passed. There are no BLOCKER or MAJOR findings, and the two MINORs below don't gate. Rig (real path, all on loopback):
Diff's own tests. All 7 new or changed test files ran under run_bounded with an isolated HOME: 63 passed in 45 s. FindingsQ1-1 — MINOR (copy): the permanent 409
Q1-2 — MINOR (scope gap, matches the design): remote PAD images never get a Library thumbnail, even when their bytes are precached.
Out of scope — pre-existing (non-gating):
Untested gaps:
|
R1-1: an owner-supplied digest and member id are shape-checked before they become viewer paths (409 session_files_remote_unreadable, nothing opened); cache.locations/lookup refuse malformed names as a backstop. R1-2: precache.opened_session goes through the catalogue poll's 60 s / only-if-moved gate, and a pass's per-path row asks sit behind the budget check and inside the precache slot. R1-3: content, Library content and download re-resolve once when an eviction wins the race to the open (then 504, never 500 or a permanent 409); fetch keeps every landing still on its way to its lookup. R1-4: the download temp file has a fixed short name (long CJK names land). R1-5: remote Library content uses the session-files MIME allowlist and a sandbox CSP. Q1-1: a permanent owner refusal answers a keyed sentence carrying the owner's reason (attachment_owner_refused), code still attachment_on_peer. i18n: I1-1/I1-2/I1-3/I1-5/N1 English copy, retranslated into all 7 locales; the auditor's 17 I1-4 corrections applied byte-exactly.
Remediation — round 1Implementer: coder (lopdev team, session Per-id dispositionAll tests named below are in
i18n: pairs edited (scope for the round-2 auditor)
Rows boundNone. The guardrail blocked running the auditor's Open, needs a decision (not fixed here): red check
|
Agent review — round 2Reviewer: reviewer (lopdev team, session Fix verification
Findings (none gate)MINOR-1 · the Files-list refusal now says "This file cannot be shown here"
MINOR-2 · a local landing failure is blamed on the owner
Tests (bounded,
|
i18n audit — round 2Reviewer: i18n auditor (lopdev reviewer role), instance Verdict: clean. No BLOCKER or MAJOR. No pair needs a change, so there are no I2 corrected strings. All 14 rows can be bound with the command file below. Retest of round-1 findings
Mechanical checks (16 files at head): 0 problems
Second independent read: ur/vi
|
QA report — round 2Verdict: FAIL on 4 rows, all MINOR. No BLOCKER and no MAJOR.
Scope: Rig. Round 1's rig, copied into this session's scratchpad and extended. Everything runs on loopback:
Two QA-only wrappers run the head code unchanged, apart from what is listed here:
Re-run: New fixtures on the owner:
Matrix
Retained (9a6ef56), not re-run: rows 1, 2, 3, 4a, 4b, 7, 7b, 9a, 9b, 9c, 10, 13a, 13c, 13d, 14, 15a, 15b, 16 and 8-auth. The delta doesn't touch the transcript attachment route, the listings, LRU ordering, capability skew or the strace no-dial proof. The other rows that share a route with the delta were re-run above. Diff's own tests. FindingsQ2-1 — MINOR: R1-5 is not applied when the Library opens a remote DIGEST asset that is already cached, which is the usual case after precache.
Q2-2 — MINOR: R1-3 on the same branch. An eviction between
Q2-3 — NIT (copy), non-gating: the 504 after two evictions says the owner is offline, but the owner is online.
Q1-2 — MINOR, still open as deferred in the remediation reply: the pad-path image thumbnail is still 404 with the bytes cached and the owner stopped. I agree with the deferral reason (the owner's index rows carry no content digest for path files), so it doesn't gate. Out of scope, environment: 4 Untested gaps:
|
QA round 2 on #3872 (Q2-1, Q2-2): when a peer's digest asset was already in cache/remote-files, resolve_content returned the cache path and the Library content route served it inline as the owner's declared type, with no allowlist, no CSP sandbox, and a file opened only once the 200 headers were out. A precached evil.svg was safe on its first open and not on any later one; an eviction before the first body chunk dropped the response. ContentRef now carries `remote` for cache hits. The route hands those to remote_session_files.library_content, the uncached remote branch, whose lookup finds the cache first (no dial), opens before answering, re-resolves once after an eviction, and applies _servable + CSP sandbox. Thumbnails and waveforms still read the cached bytes directly. Q2-3: a second eviction inside _opened now raises the pending retryable (attachment_fetch_pending + Retry-After), not "owner not answering": the owner was not what failed.
Measured eager totals (raw UTF-8 wire bytes, non-deferred namespaces): origin/main 548f3bd hi 65,187 B (the largest; 349 B under the old 64 KiB cap); with #3872's copy hi is 69,002 B. 80 KiB keeps the guard a growth guard with 12,918 B of room over hi, not a packing target. The proof test's three-file fixture now gives the last file the remainder, so its sum is ceiling - 1 for any ceiling (80 KiB does not divide by 3).
Appends the 14 audit rows (7 locales x wire.errors.desktop, wire.settings) from the round-2 i18n audit on #3872 with scripts/i18n/ledger.py, each against the history digest measured at 14d8601, and adds their evidence records (9 rfp-187b766ab67d, 5 rfp-a3fade2429f4). Every catalogue was checked against its audited sha256 before binding. Regenerating restores mobile/web/src/i18n/bundled.gen.ts to base: an en-only generate run had committed it with an empty BUNDLED_CHUNKS map. shipped.json stays byte-identical to base (8 locales).
Remediation — round 2Head
Eager totals. These are raw UTF-8 bytes of the non-deferred
80 KiB is 81,920 B. That leaves 12,918 B over hi, which sits at 84% of the cap. i18n bind ( Targeted test evidenceHead |
Agent review — round 3Reviewer: reviewer (lopdev), Verdict: not clean. There are 2 MAJORs, each with a repro. Everything else the brief asked me to check holds. Verified
FindingsR3-1 — MAJOR: the mobile relay serves a cached peer file inline with the owner's declared type. XML types get no sandbox, so a paired peer can run script on the relay origin.
R3-2 — MAJOR (delta regression): a cached peer asset no longer opens when its owner is offline. It returns a 404 "not found", and a cache miss on the peer row now triggers a mesh listing read.
R3-3 — MINOR (neighbour): the desktop thumbnail route serves a cached peer image with the owner's type, without
R3-4 — NIT. TUI caller (the coder's question)
Not verified
|
QA report — round 3Verdict: FAIL, on one new MAJOR finding (Q3-1). The fix in this delta causes it, and I reproduced it against a base control.
Scope: Rig. The round-2 harness, copied into this session's scratchpad. The only changes are paths, and the new
I created the rig data directory Same-code control for Q3-1.
Every other module is head's. The wrapper's start record proves which source files were loaded. Matrix
Retained (14d8601), not re-run: R1-2, 8, 8-active, R1-5 path, 6/Q1-1, 5/Q1-1, 8u/N1, I1-2, I1-3, R1-4, R1-3 pre, R1-3g, R1-3h, R1-3i, R1-1 ctrl, R1-1 and Q1-2 (deferred). Rows retained from round 1 (9a6ef56) are unchanged. The delta doesn't touch the transcript attachment route, the listings, the precache gate, the path-row resolution, download landing or the malicious-row validation. FindingsQ3-1 — MAJOR: since e3ac0f2, opening a cached remote Library asset fails with a false "not found" when the owner is offline. It worked at 14d8601.
Q3-2 — lead, outside this delta's surface, non-gating this round: the transcript attachment route behaves the same way.
Observation, not a finding. In row 12, relay A, the owner, peaked at +18.6 MiB (round 2: +0.2). It had been restarted shortly before the baseline sample, and the delta doesn't touch it. The figure is still under 10 % of the file size. Untested gaps:
|
…, sandbox them on the phone The digest routes (Library open, transcript attachment, waveform, document, child) read the owner's listing row before this device's copies, so a precached file 404'd as 'not found' exactly when its owner was asleep, and a row-cache miss dialled the relay for bytes already here. They now look in the store, replica and remote-files cache first, keyed by a durable existence proof (the Library index's recorded origin device), and read the row only on a miss; a miss with the owner silent is the retryable 504 attachment_owner_unreachable. The phone's Library routes served a cached peer file inline with the owner's declared type (XML unsandboxed on the relay origin); they and the desktop thumbnail now apply the remote routes' servable_type allowlist plus a sandbox. Fixes #3872 R3-1..R3-4, Q3-1, Q3-2.
Remediation — round 3Head: Dispositions
Regression tests:
|
Agent review — round 4Reviewer: reviewer (lopdev),
Verdict: clean. No BLOCKER or MAJOR remains. R3-1, R3-2/Q3-1, Q3-2 and R3-3 are closed. The round is terminal: the two MINORs and one NIT below are for the thread and need no new commit. Verified
FindingsR4-1 — MINOR: remote SVG tiles now render as broken images, on desktop and on the phone. (This is a side effect of the R3-3 fix I suggested in round 3.)
R4-2 — MINOR: for a session id the index never recorded, the 504 names every silent device.
R4-3 — NIT: rare double-eviction race on the transcript routes.
Not verified
|
QA report — round 4Verdict: PASS. No BLOCKER or MAJOR. Every R3 and Q3 row passes on the real path at
Scope: Rig. The round-3 harness, copied into this session's scratchpad. The helpers from rounds 1–3 are unchanged. New files:
Everything runs on loopback:
Each process had its own isolated Matrix
¹ The raw harness results preserve W1, d and W2 as FAIL because the Library-waveform request was in the same set. The other 12 requests pass in all three windows. The scoped verdict separates that 404 as Q4-1: out of scope — pre-existing at The 13 GETs in each R4-off window (results from W1):
R3-1 results on the mobile daemon:
Retained (5d7e1a5), not re-run: 12 and 12c (RSS; the path-asset branch of FindingsQ4-1 — MINOR, lead, non-gating; outside this delta, present in the PR since
Observation (NIT, by design). A never-recorded session id with a cached digest returns 504 "qa-owner-a is not answering", not 404, while any peer is silent: with no indexed owner, the silence isn't narrowed. No bytes are served, so the existence check holds. Only the sentence names a device that never held the id. Observation (unchanged behaviour). The child route serves a cached digest for any well-formed child id of an indexed peer session. Before this delta the cache hit didn't validate the child either: Untested gaps:
Primary evidence (QA session
Rig hygiene.
|
Copy the generated desktop fixture at local-operator-ui 5033f642352e93c30b699ff62e5bae0e1939da6c and verify its manifest before re-pinning. Every field other than desktop.sourceHash is unchanged, including all 700 cases and the latch: no condensation behavior port is needed. Core main still pins the preceding desktop generation, so merging main alone cannot repair the required cross-repo drift check.
Resolve the two generated condensation fixture/pin conflicts to the byte-verified local-operator-ui 5033f6423 generation. Preserve upstream changes and remote-files implementation; convergence proof/review required for the upstream-overlap files.
CI remediation — condensation parityHead: The Mobile web drift guard was red after desktop regenerated its fixture. Core main Authoritative instructions: Parsed old/new fixtures with only Condensation CI remediation
Convergence
No guard weakening, executable repair, catalogue edits, full suite or dependency installation. Manager authorized merging main after GitHub became CONFLICTING; it is now MERGEABLE. Temporary dependency symlink removed. No merge-gate dispatch. |
Rebase proof — bec264a..f7337ad
Re-run: Merge/convergence detailsMerged All conflicts (complete list):
Semantic review scope for ONE convergence round,
Additional byte compatibility: comparing old-base..old-head and new-base..new-head +/- line multisets plus file headers with only the TWO fixture paths excluded is exactly equal. The proof's six changed lines are the fixture identity removals relative to the advanced base; non-condensation PR changes are preserved. The retained latest fixture is byte-identical across the merge and to UI Convergence
|
CI remediation — directory-removal guardActual model: Fixed the invariant, not just the allowlist. Before adding exceptions, fresh regressions proved two authoritative-store traversal paths (
Actual execution (serial, isolated HOME/config; final run also isolated TMPDIR; env -i strips CMUX_/LOP_; own worktree venv verified; run_bounded --timeout 600 --memory-mb 3000): The focused regression matrix covers both symlink parents, a swap between check/open, a swap after fd acquisition, regular-file/digest-only selection, preservation of sessions and authoritative attachments, and pre-existing download-temp file/link/directory collision safety. The index/fetch neighbors exercise real paired relays and HTTP routes, including download verification/no-overwrite, mismatch, and LRU eviction. Changed paths: Residual trust assumptions: the configured root is trusted (its own spelling may resolve through an operator-configured symlink, as in the existing sync primitive); this does not defend against privileged bind mounts or a same-user attacker replacing regular contents inside the private expendable cache. The unchanged download cleanup assumes a same-user attacker does not replace this call's unpredictable private temp name or move the destination directory during the call; remote input cannot select that name/directory. Independent narrow source review and real-path QA are manager-owned and not claimed here. Local primary logs: |
Agent review — cache confinement deltaReviewer: reviewer (lopdev), Verdict: clean — no BLOCKER or MAJOR. No new findings. Existing MINOR/NIT findings remain deferred. All unaffected feature paths retained from the prior clean reviews. Security/source verification
Only Fresh execution on the pushed headTwo serial
Total: 19 passed, covering 8 independent reviewer probe cells and 11 shipped regression/neighbour cells. I did not duplicate the 614-test guard suite or the implementer's 654-test run; the before-fix failing controls are sourced to the implementer's evidence, not claimed as rerun here. Independent external two-relay+viewer smoke remains QA's stream. Cleanup receipts: |
QA report — cache confinement deltaVerdict: PASS. No new BLOCKER or MAJOR. Independent real two-relay + viewer-HTTP eviction probes reproduce the three defects with the previous cache implementation and pass with the fixed implementation. Both directory-swap timing probes executed successfully, not BLOCKED. Scope: Real path / negative-control provenanceReused my own existing synthetic R4 relay pair/config/data. Two real Every attack probe is driven by a real authenticated viewer HTTP GET for the same 200,002-byte remote digest. The driver removes only that synthetic rig's evictable copy before each request, forcing a real relay fetch. The normal path is:
QA-only All symlink targets are new synthetic authoritative-store decoys under this session's scratchpad, never the operator's real data. They contain a valid digest Actual matrixAttachment request used for all nine eviction arms:
The zero-budget arms intentionally retain the just-fetched digest through the production Commands / executionOne heavy-run precheck was clear, then one serial bounded real-path run: env -i PATH=/usr/local/bin:/usr/bin:/bin LANG=C.UTF-8 PYTHONUNBUFFERED=1 \
HOME=/home/ec2-user/qa-rfp-core-r4/home-setup \
LOCAL_OPERATOR_CONFIG_DIR=/home/ec2-user/qa-rfp-core-r4/setup-config \
PYTHONPATH=<own-pad>/core-cache-confinement-qa:<worktree> \
<worktree>/.venv/bin/python <worktree>/scripts/run_bounded.py \
--timeout 300 --memory-mb 2000 -- \
<worktree>/.venv/bin/python <own-pad>/core-cache-confinement-qa/smoke.py \
/home/ec2-user/qa-rfp-core-r4Actual: Compatibility / retained evidenceIndependently verified the exact changed-path list from git objects:
All other product paths, including download implementation, serving/type/sandbox/auth routes, relay registration, and catalogues, remain unchanged from Primary evidence / cleanup / gapsDirectory:
Verified both rig cache directories are restored as ordinary directories and no held/renamed paths remain. Removed only the exact planted non-digest/link/dummy-digest leaves created by these probes; retained scratch decoys/evidence. No repo receipts/untracked files were created. The old rig root's final deletion denial was not retried. Unverified by design: Windows/non-POSIX behaviour, privileged bind mounts, arbitrary same-user tampering inside trusted regular cache contents, new rendered UI/WAN/native/RSS evidence. Download check covers unchanged same-byte success/on-disk integrity; no new temp-collision matrix was run in QA. Neither directory-swap probe is blocked: both ran against actual HTTP-triggered eviction. |
Copy exact generated fixture bytes from local-operator-ui 72043c4ec (regenerated in b0edf02e5, #1877) and verify manifest identities. All 700 cases, latch, modelSources and every field except desktop.sourceHash remain unchanged. The hashed source delta only adds locale formatting/bidi helpers; no condensation model port is required. Strict drift guard fails on prior bytes and passes after; 1361 condensation tests pass. Remote-files security source remains byte-identical to 928c9fc.
CI remediation — condensation parity refreshHead: Source and actual-source diagnosis
Changed paths (complete list)
3 insertions / 3 deletions. Fresh targeted execution Condensation replay used isolated HOME/config and stripped inherited CMUX_/LOP_; existing dependencies reused after byte-identical lockfile check, no installation. Temporary dependency link removed. All bounded runs reported cleanup proven empty. Evidence logs/source diff: delegated session Recurring reproducibility blocker — separately owned follow-up recommendation, not implemented here Recommend a separate owner change the required reproducible parity check to fetch an actual declared immutable desktop source revision, then enforce manifest consistency: fetched fixture SHA-256, embedded sourceHash and copied pin must agree, and source identity must be validated against that same revision (using the generator's declared hashed inputs). Keep freshness against current desktop main as an explicit separately-owned update detector/port workflow, not silently removed or confused with reproducible compatibility. Add negative controls for tampered fixture/manifest, incorrect revision/source identity and failed fetch; retain hard failures. No workflow/source-policy change, extra PR/worktree, or guard weakening is included here. PR was MERGEABLE before and after refresh; no main merge. No full suite, re-arm, cancellation or gate dispatch. Parent owns the next gate; stop refresh cycles here pending separately owned source-level work. |
Agent review — condensation refresh CI deltaReviewer: reviewer (lopdev), Verdict: clean — no BLOCKER or MAJOR introduced by this delta. Terminal within this allowed scope. Existing MINOR/NIT findings remain deferred. Independently verified provenance and retentionI read the implementer's primary evidence, then verified against actual Git objects in the named repositories:
Fresh exact guard executionCreated a scratch-only sibling fixture/manifest snapshot from those exact UI Git objects. Node checked clear of other heavy runs; isolated scratch HOME/config with Run Recurring CI reproducibility issue — outside this delta, separately ownedDirectly inspected unchanged Recommend a separate owner implement a declared immutable UI source revision for the required reproducible compatibility check, bind fixture/manifest/sourceHash validation to that same revision, retain hard-failure negative controls, and keep latest-main freshness as an explicit separately owned update detector. Do not weaken or modify the workflow in this scoped PR delta. This review finishes the allowed refresh scope; manager owns the next exact-head gate and escalation if moving main invalidates it again. |
CI remediation — wheel sizeCommit: Exact sites / rationale. Primary failed CI: run 43319909082 / job 131567736599, job log read directly through the jobs logs API. Original Actual local build and paired repack: own worktree
Measured saving: 93,113 bytes; final headroom: 98,255 bytes. Archive and runtime proof: both wheels pass Targeted checks: isolated serial Independent QA artifact: |
Agent review — wheel packaging deltaReviewer: reviewer (lopdev), Verdict: clean — no BLOCKER or MAJOR. No new findings; existing MINOR/NIT findings remain deferred. Product/security reviews retained by unchanged source bytes. Code and policyOnly two executable call arguments change: The new regression uses structured data that independently compresses smaller at level 9 than level 6, verifies the slimmer's exact compressed size against level 9, and asserts all uncompressed member bytes remain equal. I executed it against the parent script loaded from its exact Git object in a scratch-only negative control: the new regression rejects the old script, as required. Existing whole-file tests also cover RECORD validity, metadata/mode retention and fail-closed behavior.
Independent artifact comparisonOpened the existing same-input before/after wheels; no rebuild or install performed:
These are the actual paired local artifacts, not a claim of byte-identical reproduction of the hosted failing build. The implementer's evidence discloses the local baseline being 5,580 bytes smaller than hosted and the frontend/backend used. Fresh hosted CI remains authoritative for the hosted cap; independent installed-wheel/relay smoke belongs to QA. Fresh targeted execution and cleanupNode checked free of other heavy runs. Existing worktree
Run receipts |
QA report — installed wheel deltaVerdict: PASS. No BLOCKER or MAJOR. Independently installed the supplied local wheel offline with no dependencies installed, verified complete lossless archive equivalence and RECORD/CRC, then exercised real installed-source CLI, two relays and viewer HTTP routes. No feature matrix/native/capture rerun. Scope: Artifact and installation provenanceArtifact: Installed into my own scratch target using the existing worktree Python 3.12 dependency environment, without modifying it: <worktree>/.venv/bin/python -m pip install \
--no-index --no-deps --no-compile \
--target <own-pad>/core-wheel-qa/installed \
<worktree>/dist/ci-size-after/local_operator-0.102.1-py3-none-any.whlActual installer: Actual import root for the probe, CLI, both real relays, and viewer daemon: Each daemon entry shim hard-asserts this installed path before calling the normal CLI; each process log records the installed path and exact CLI argv. Matrix — actual archive, CLI, resources, HTTP
Bounded execution and retained scopeNode heavy-run check clear before serial dispatch. Installation bounded at 180 s/2,000 MiB; archive/CLI verification and installed HTTP smoke each bounded at 300 s/3,000 MiB. Actual archive verification 2.7 s, HTTP smoke 9.4 s, both rc 0 with Independently verified git changed-path list for Evidence and limitsOwn directory:
Unverified: exact hosted CI wheel/result (this is the supplied local artifact, not the failed hosted build), pristine dependency resolution/other Python/platforms, mobile browser bundle rendering, WAN/native/UI/RSS. Static resources were read as packaged resources, not rendered. Those limits do not invalidate the observed installed-wheel real relay/HTTP path. All own daemon processes are stopped. Existing rig root remains under its earlier final deletion denial; no delete/move retry. |
CI remediation — test formattingFixed in Verified failure: fast CI run 43339327881 / lint job 131645635486, source log Exact sites: only collapse the signature of Pinned tools / no install: read Actual checks after formatting:
Semantic/source proof: saved both pre-edit parses, then compared the complete
This is the requested static formatting-only delta. Existing product/security/artifact evidence remains unchanged; no artifact-QA rerun or full suite. No gate arm, workflow cancel, merge or release. Manager owns static delta review and required hosted CI. Delegated session |
Agent review — formatting CI deltaReviewer: reviewer (lopdev), Verdict: clean — no BLOCKER or MAJOR. Static review only, as briefed. Existing product/security/packaging reviews and independent QA retained; no extra QA round or test execution needed for these AST-identical formatting changes. Existing MINOR/NIT findings remain deferred. I opened the implementer's primary evidence, read the entire narrow diff, and independently verified:
Thus all shipping product/cache/download/security/catalogue/packaging-script bytes and version are unchanged from reviewed Execution disclosure: I ran only static diff, full-file AST and Git tree/blob identity checks. No tests, formatter, full suite, capture, native rig, server, install, workflow action or repository edit executed in this review. The reported pinned Black/isort/flake8 passes and 3 test passes belong to the implementer's linked evidence, not fresh reviewer execution. Required hosted lint/CI remains the manager's gate. This review is terminal for the exact formatting delta. |
CI remediation — test type narrowingFixed in Verified the four diagnostics in required type-check job 131708826639 / run 43364213305, source log
Read the actual implementations: Exact scoped type command, before and after: own .venv/bin/python scripts/run_bounded.py --contract=command-status --timeout 600 --memory-mb 4000 -- \
.venv/bin/python -m pyright --pythonpath <worktree>/.venv/bin/python \
tests/unit/network/test_remote_files_index.py \
tests/unit/server/test_desktop_sessions.py \
local_operator/server/utils/desktop_sessions.pyIsolated scratch HOME, inherited CMUX/LOP removed. Before: 4 errors / rc=1. Final after: 0 errors, 0 warnings, 0 informations / rc=0. The included product module defines the actual attachment union and Exact affected runtime selectors: isolated HOME/config, serial,
Final result 24 passed, 5 existing deprecation warnings in 5.90s. The unrelated Exact-file lint: Black 26.1.0 Product byte proof: before/after SHA-256 binding each tracked filename and its content digest under No upstream-main chase, merge, gate rearm/cancellation, artifact rebuild or full matrix. Source prerequisite #3892 remains manager-owned; manager will perform the single convergence/gate after it lands. Delegated session |
Agent review — test type CI deltaReviewer: reviewer (lopdev), Verdict: clean — no BLOCKER or MAJOR. No weakened assertions or new findings. All prior shipping product/security/artifact evidence retained; existing MINOR/NIT findings remain deferred. Read the exact two-file delta and implementer's primary evidence. Independently verified:
Execution disclosure: reviewer performed static diff/fixture inspection, full-file assertion-AST retention and Git tree/blob checks only. No additional pytest/pyright/lint run, QA/native rig, capture, dependency install, repository write or full-suite execution. The reported scoped pyright 4→0, 24 test passes and exhaustion negative control are the implementer's linked evidence, not newly executed reviewer results. This delta review is terminal. It does not arm a gate or authorize an upstream merge; manager retains the stated hold for prerequisite #3892 and the single subsequently authorized convergence/gate. |
Convergence after reproducible parity prerequisiteOne authorized ordinary merge pushed: Conflict and manager allocation. Ordinary merge auto-merged both Python overlaps and conflicted only in Manager explicitly authorized selecting already-published UI revision
Removing only the two adopted provenance fields yields JSON exactly equal to the reviewed branch metadata; every pre-existing JSON field and all fixture bytes are retained. Source/overlap proof. Upstream moved 64 files; branch patch spans 98. Exact overlap:
All 95 non-overlap feature files are byte-identical to Actual validation. Isolated HOME, inherited CMUX/LOP stripped; bounded commands (no installs):
Hosted before discriminator: independently read completed failed run 43364213244 / build job 131684668141. At Disposition: CONVERGENCE REVIEW NEEDED, not mechanical re-pin. Independent static review scope is Evidence retained in delegated session Rebase proof — 360315b..7fb7cfc
Re-run: |
Agent review — convergence after #3892Reviewer: reviewer (lopdev), Verdict: clean — no BLOCKER or MAJOR. Existing MINOR/NIT ledger frozen. This is convergence review, not a mechanical re-pin. Opened the primary proof, inspected the three overlap diffs, and independently derived:
Retention limit: prior feature, security, 24-test typing and packaging evidence is retained for the proven unchanged feature implementations. This does not claim the entire shipping tree/new wheel is byte-identical: ordinary upstream changes and its version are merged. The final exact-head hosted gate covers that integrated tree. This review introduces no PR-authored version bump or new artifact claim. Execution disclosure: static source/AST/patch/Git-object hashing checks only. No pytest/QA/native/capture/full-matrix/rebuild/installation or server process launched. The actual strict-checker PASS, 15 negative controls and pyright/lint results are sourced to the implementer's linked proof, not claimed as rerun here. No repository writes or new receipt files. Review terminal; manager owns the one subsequent exact-head gate. |
Core half of remote files parity (design:
docs/design/remote-files-parity.md, §7 "Core PR", C1–C6). UI half: damianvtran/local-operator-ui#1880.Operator request (verbatim)
Commits
1af12f60e perf(network): memoise and stream the referenced-digest scan(C1):referenced_attachments_inscans in 4 MiB windows (64-byte overlap) instead ofread_bytes(), memoised on the journal's identity, so one 32 MiB pull scans the owner transcript once instead of once per chunk.cb746abbf feat(desktop): fetch peer attachments on miss(C2): attachment / waveform / supplement document / child-route bytes of a peer session resolve: local store → replica (G1) → bounded remote cache → ONE single-flight pull from the owner (S1net_file_get,land="cache"). Streamed from an fd withRange/206 (server/utils/byte_range.py); 504 retryable (attachment_owner_unreachable,attachment_fetch_pending+Retry-After), 409 permanent (attachment_peer_outdated,attachment_on_peer). Newremote_files/{cache,fetch}.py, LRU eviction under flock, config keydesktop.remote_files_cache_bytes.fc4d0aa96 refactor(session-files): move family discovery out of the desktop module(C3): pure move intosession_files/family.py(re-exported under the same names) +family_for/family_blocking, so the relay never importsserver.utils.desktop_sessions/fastapi.0614e8aa5 feat(network): net_session_files peer index(C4): owner-answered Files and user-attachment index (view capability,mesh-file-index-v1), display paths only, per-rowremote_read/remote_unreadable, page trimmed to one link record; viewer…/files,…/user-attachmentsanswer peers withremote,origin_device; new peer-onlyGET …/files/{id}/contentandPOST …/files/{id}/download;features.remote_files = 1; DESKTOP_API "Remote files".2655dbaf7 feat(desktop): precache active remote sessions(C5).ab6f4a92b feat(library): index remote sessions' files(C6): schema v3,identity.remote_key,library/remote.pyfeeder,LibraryRow.remote.9a6ef5661 docs(design): record where the core remote-files PR differs from the plan(doc only).14d8601fd fix(remote-files): round-1 remediation for #3872(#issuecomment-6831178771).e3ac0f2ae fix(library): serve cached remote assets through the remote content path(QA r2 Q2-1/Q2-2/Q2-3).0cd05c51e test(i18n): raise the eager wire ceiling for the remote-files keys(80 KiB; measured basis in the test).5d7e1a5e4 chore(i18n): bind the remote-files audit rows(14 rows, evidence,bundled.gen.tsback to base).3a842e70b fix(remote-files): serve a peer's cached bytes with its owner offline, sandbox them on the phone(round-3 R3-1..R3-4, Q3-1/Q3-2, #issuecomment-6833147857). Digest routes (Library open, transcript attachment/waveform/document/child) check this device's store → replica → cache FIRST (local_peer_blob), gated by the Library index's recorded origin device as the existence proof; the owner's row only on a miss; miss + silent owner → 504attachment_owner_unreachable. Mobile/contentand/thumbnailplus the desktop thumbnail applyservable_type+ sandbox to remote refs.Wire shapes match the UI PR's assumptions: content 504 retryable / 409 (
session_files_remote_unreadable,attachment_on_peer) / 404 permanent; download envelope{path,name,bytes,sha256,created,origin_device};origin_device = {id,name}; every remote row carries bothremote_readandremote_unreadable;LibraryRow.remote = {device_id, device_name}withref.path: null.Acceptance → test
All on the REAL two-relay loopback (
meshfixture fromtest_mesh_file_pull.py/_pair), B's desktop routes on an ASGI app over B's root.test_remote_files_scan.py::test_a_32_mib_pull_scans_the_owner_transcript_once,…straddling_a_window_boundary…,…memo_follows_an_append_and_is_boundedtest_remote_files_fetch.py::test_a_…test_b_…test_c_…test_d_…fetch_pending+ Retry-After, retry joins the same transfer →test_e_…test_f_…test_g_…test_h_…test_i_…test_j_…test_k_…test_l_…session_files/test_family_module.py(same objects re-exported; relay import path loads neither fastapi nor the desktop module)sensitiverule →test_remote_files_index.py(11 cells)test_remote_files_precache.py(6 cells)stat_pathspy), I5-2 (desktop_mesh._callraises; list/detail/thumbnail of every remote asset 200/404), thumbnail after open survives eviction, local+remote digest = one asset with two occurrences, removal after two answered catalogue reads (silent device keeps rows) →test_remote_files_library.py(6 cells)Budgets and their basis
desktop.remote_files_cache_bytes(LIVE), default 1 GiB = ~55× a typical active set (16 sessions × 6 digests × p90 187.5 KiB ≈ 18 MiB), 3.8× the largest measured pad (269 MiB) — design §4.0= staging only, precache off.mtimemoved; per file ≤32 MiB (the UI'sMAX_DOWNLOAD_BYTES); 1 precache pull per owner inside the owner's semaphore of 2.REMOTE_FETCH_WAIT_S= 20 s for a pull before answering 504 pending (the pull continues);net_session_filesowner deadline 30 s (1.5 s scan budget + discovery + stats,buildingcovers slower), viewer client 45 s; Library remote feed every ≤300 s while in a network.RED on base / negative controls
assert 65 == 1).409 == 200 attachment_on_peer; (i)404 == 200 child_not_found; (k)serving buffered 235.6 MiB.desktop_session_files.py:KeyError: 'remote'(base answeredunsupported, no rows)._fitdisabled → the oversize page closes the link → 504.OWNER_CONCURRENCY=1: "the on-screen request waited 1.54s behind precache".library.remoteabsent (ImportError); storing the display path aspath→ I5-1 fails ('report.md' is None).i18n
New keys (en + context; fr/es/zh-CN/ru/vi/ur/hi translated in this PR,
scripts/i18n/check.py --catalogues-only: 0 problems):wire.errors.desktop.{attachment_owner_unreachable, attachment_fetch_pending, attachment_peer_outdated, session_files_owner_unreachable, session_files_remote_unreadable, session_files_sensitive_name, session_files_download_exhausted, session_files_download_mismatch},wire.settings.desktop_remote_files_cache_bytes.{label,help}.shipped.jsonis at base bytes. All 14 ledger rows are bound in5d7e1a5e4, from the independent round-2 audit (#issuecomment-6831544890):generate.py --check161 artifacts byte-identical,ledger.py validatevalid and append-only,check.py --audit-base origin/mainok.Targeted test evidence
One authorized convergence
7fb7cfc2d122cbb21096081b67a401636914bbcf: merges exact main/#38924d113fc6c23b3fbb2afa426c0c6747ee0c399467into reviewed360315b47. Sole conflict resolved through manager-approved immutable provenance revision72043c4ecbfddd734abec28694b05a163786bf0c, authenticating the existing b0edf02e5/#1877 fixture; no fixture/hash refresh. Actual repo strict checker fetched and authenticated fixture, manifest and 36 model sources: PASS; checker controls 15 passed. Scoped overlap/neighbors pyright 0 errors/warnings; overlap-only Black/isort/flake8 PASS. 95 non-overlap feature files byte-identical, all 97 non-metadata feature +/- patches identical, 19 range-diff pairs=. Only metadata patch differs (4 lines); overlaps are parity-sources.json, relay.py, desktop_sessions.py. Full### Rebase proofand before/after hosted discriminator are in### Convergence after reproducible parity prerequisite. CONVERGENCE REVIEW NEEDED by independent reviewer; manager owns one gate rearm. No gate arm/cancel, full matrix, artifact/native QA rerun or PR merge. Actual modelopenai/gpt-6.1-sol.Required-CI test-type repair
360315b4776f61f5ff3e7fd330c70e8f7ffb8614(actualopenai/gpt-6.1-sol): fixes the index polling helper's possibly-unbound page and three local attachment union unpacks through explicit runtime invariants; no ignores/casts or product changes. Scoped CI-pinned pyright 1.1.414 on the two test modules plus actualserver/utils/desktop_sessions.pytypes: 4 errors before → 0 errors/warnings after. Isolated bounded real-loopback index matrix plus exact local attachment/refusal selectors: 24 passed, 5 deprecation warnings in 5.90s; unrelated MP3 fixture cases not selected. Direct polling success/exhaustion controls PASS; all original assertion ASTs retained. Exact-file Black 26.1.0, isort 5.13.2, flake8 7.4.1, whitespace PASS. Shipping source hash remains512e12dcc19b7ab1a900073c108fc320be022d9a9065239ef0229bd2b207caaa. See### CI remediation — test type narrowingfor T1–T4 and exact commands/proof. No gate rearm/cancellation, upstream convergence or full matrix; prerequisite #3892/convergence remain manager-owned.Late lint-only repair
f49dcc68bc18daae64429429e72de065471ab1e8(actual modelopenai/gpt-6.1-sol): CI-pinned Black 26.1.0 reformats exactly two expressions intest_remote_files_hardening.pyandtest_slim_wheel.py. Reproduced 2-file FAIL before, exact-file Black/isort 5.13.2/flake8 7.4.1 and whitespace checks PASS after; isolated bounded minimal compression/symlink selections 3 passed in 0.77s. Complete location-independent ASTs exactly equal in both tests. All tracked product/security implementation/scripts/catalogues plus pyproject bytes retain SHA-256512e12dcc19b7ab1a900073c108fc320be022d9a9065239ef0229bd2b207caaa; shipping subtree diff empty. No installs, artifact-QA rerun, full suite or workflow manipulation. Exact commands/proof in### CI remediation — test formatting; static delta review manager-owned.Wheel-size required-CI repair
f10d7c17956ead2ae06b1d6bb356b04fee771556(modelopenai/gpt-6.1-sol): lossless DEFLATE level 9 in the existing repacker only; no product/assets/version/budget changes. Failed hosted wheel 18,350,518 bytes (438 over). Actual localuv build --wheel --no-build-isolation --offlinewith cached setuptools 84.2.1 / wheel 0.48.0 (no installs), paired same-input original/new repacks 18,344,938 → 18,251,825 bytes, saving 93,113; final headroom 98,255. All 3,165 ordered member names, timestamps/permissions, uncompressed SHA-256s and RECORD bytes identical; CRC + RECORD validation PASS. Isolated bounded targetedtest_slim_wheel.py+mobile/test_web_package_data.py: 25 passed; old compression fails new regression. Wheel-loaded base imports, runtime-resource reads and CLI--versionPASS; flake8/compileall/diff whitespace PASS. Repack wall 31.51s → 33.95s. Exact commands, baseline discrepancy disclosure and independent-QA wheel path/hash in### CI remediation — wheel size. No full-suite/workflow manipulation/merge/release.Directory-removal guard CI repair
928c9fc280ab109de37d9b6551a3bf13f7f3fd63(actual modelopenai/gpt-6.1-sol; source-delta review/QA manager-owned):test_no_session_deletion.pyRED before: 1 failed, 613 passed; focused confinement controls RED: 3 failed, 3 passed. Fixed proven cache-parent/root symlink traversal and non-digest selection using held no-follow directory fd + fd-relative unlink;landunchanged and O_EXCL collision controls PASS. Final isolated serialrun_bounded --timeout 600 --memory-mb 3000 -- .venv/bin/python -m pytest -q -n 0 -p no:cacheprovider tests/unit/session/test_no_session_deletion.py tests/unit/network/test_remote_files_hardening.py tests/unit/network/test_remote_files_index.py tests/unit/network/test_remote_files_fetch.py::test_l_eviction_keeps_the_cache_within_its_budget: 654 passed, 5 warnings in 62.04s; cleanup proven-empty. Flake8 (three changed files) + diff whitespace PASS. Exact evidence/proof in the### CI remediation — directory-removal guardcomment. No catalogue or other product changes.Condensation CI remediation
bec264a29(fixture identity only): desktop generated bytes fromdamianvtran/local-operator-ui@5033f642352e93c30b699ff62e5bae0e1939da6c; SHA-256523e8a5959fa198795ee2e963139560906aa5f425ff9be77c167a369303a35f1, model sourceHasheb9d2c6d83bb277ee50d1018009a86d5218b5d5d9263bacfa25de1e339be5a82. Strict JSON comparison proves all 700 cases, latch, modelSources and every field exceptdesktop.sourceHashunchanged. Commands below ran fromlocal_operator/mobile/web, each under.venv/bin/python scripts/run_bounded.py --timeout 600(repository-relative wrapper paths resolved absolutely):node scripts/check-condense-parity-latest.mjs --latest --require-sibling --sibling <committed-desktop-fixture-snapshot>: existing branch bytes FAIL (rc=1) → copied fixture PASS (rc=0).node scripts/check-condense-parity-latest.mjs: PASS, 2 fixtures match their pins.node node_modules/vitest/vitest.mjs run src/lib/condense.test.ts --maxWorkers=1 --minWorkers=1(--memory-mb 3000, isolated HOME/config, inherited CMUX_/LOP_ stripped): 1 file / 1,361 tests passed. Reused existing dependencies with byte-identicalpnpm-lock.yaml; no install.git diff --check: PASS. No executable code, catalogues or condensation decisions changed.Convergence
f7337ad27944df4eef4a3026831fb1df9b386ab1merges core main97eed22c9a81191ebee3a7490d3001d77a14e5ad(old headbec264a29, old merge base89678ffc7aa536076cc3c16d9d204829a4e83a97). Two conflicts only: generated fixture and pins, resolved to the already-proven desktop bytes. Final head repeats strict drift guard/self-consistency PASS, condensation 1,361 passed, andtest_settings_io.py -k "remote_files_cache_bytes or network.credentials.github.repositories"4 passed. Network neighbors (test_remote_files_index.py,test_mesh_file_pull.py,test_create_idempotency.py): 48 passed / 1 failed. Failure attest_create_idempotency.py:114sees an additionalnet_definitionsframe; the single test fails serially too and with exact upstream relay source (SHA256838ae42704fcc6bfdc1b13fda18ca3c125deddefa91d76a9b45380086488ec95) plus byte-identical upstream test, establishing it is not introduced by remote-file relay additions. This source-control run uses the current package for other modules; upstream relay refuses the branch-only remote-files slice as expected. No out-of-scope change made. Full logs retained in the delegated session'sscratchpad://condensation-parity/.rebase_proof.pyis CONVERGENCE REVIEW NEEDED, not mechanical: 14/14 range-diff pairs=, but 6 fixture/pin +/- lines differ and upstream overlaps five files. All non-condensation PR +/- lines and file headers are byte-identical; manager owns one convergence delta review and CI/merge gate.Head
3a842e70b(round-3 remediation, #issuecomment-6833147857). IsolatedHOME/LOCAL_OPERATOR_CONFIG_DIR, CMUX stripped, each runpython3 scripts/run_bounded.py --timeout 600 --memory-mb 3000 -- .venv/bin/python -m pytest -q -p no:cacheprovider …:git diff 5d7e1a5e4..HEAD -- local_operator/i18nempty;generate.py --check161 artifacts byte-identical.Head
5d7e1a5e4: remote-files files → 55 passed; library route/service + i18n catalogue tests → 60 passed; RED on 14d8601:test_remote_files_hardening.py -k q2→ 3 failed.Earlier head
14d8601fd: the same remote-files files + test_mesh_file_pull.py → 81 passed; tests/unit/test_settings_io.py, server/test_desktop_remote_open.py, session_files/test_{family_module,route}.py, i18n/test_language_manifest.py → 805 passed (round 2 did not touch these surfaces). RED on 9a6ef56: test_remote_files_hardening.py → 14 failed.Release: patch — Fetch and cache remote-session assets for desktop previews and verified downloads, with indexed file listings and bounded disk use.