Skip to content

fix: process large recordings reliably without daily allowances - #2231

Merged
richiemcilroy merged 6 commits into
mainfrom
codex/fix-large-recording-processing
Sep 6, 2026
Merged

fix: process large recordings reliably without daily allowances#2231
richiemcilroy merged 6 commits into
mainfrom
codex/fix-large-recording-processing

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Sep 6, 2026

Copy link
Copy Markdown
Member

Large retained recordings could be held until midnight by a shared 512 GiB reservation cap. A 13.2 GB Drive output also failed after its single upload request stopped at 8 GiB, while another recording aborted because the memory guard counted reclaimable file cache as application memory.

Remove the daily admission limit while retaining immutable-attempt reservations, per-recording retry bounds, direct storage transfer and worker capacity controls. Explicit busy responses now honor server backoff with a stable spread between recordings, persist the waiting status, and retain the same attempt and reservation. Recovery selects interrupted active work before the old incomplete-source backlog and excludes exhausted or retired jobs before applying batch limits. Drive outputs upload in 32 MiB chunks, reconcile confirmed offsets after interruptions, and bound retries and retransmitted bytes. Memory pressure excludes clean inactive file cache while retaining dirty pages and process memory in the guard.

Validation: 88 web tests; 37 upload and memory tests; 42 worker and real-media route tests. A real local HTTP transfer of a 9 GiB sparse file completed across 289 chunks with 158 MiB peak RSS growth. Both Linux production-image architectures run the upload, memory and large-file regression gates in CI. No database schema change. Full source/output decode, content identity and preservation checks remain enforced.

Greptile Summary

This PR improves reliability for large retained recordings and media outputs.

  • Removes the shared daily processing allowance while preserving immutable-attempt and per-recording reservation limits.
  • Persists media-server capacity waits and reuses the same processing attempt while honoring bounded Retry-After delays.
  • Adds chunked, resumable Google Drive uploads with offset reconciliation and retry/retransmission limits.
  • Excludes clean inactive file cache from memory pressure while retaining process memory, dirty pages, and writeback accounting.
  • Prioritizes active interrupted recordings over source-blocked recovery backlog and expands production-image regression coverage.

Confidence Score: 5/5

The PR appears safe to merge, with the prior capacity-wait finding resolved and no new actionable defects identified.

Recognized busy responses now persist a waiting state, honor bounded server-directed backoff, and retain the same attempt. The newly added recovery admission changes preserve eligibility for deferred work, while Drive upload and memory-accounting paths include conservative validation and focused regression coverage.

Important Files Changed

Filename Overview
apps/media-server/src/lib/drive-resumable-upload.ts Implements bounded 32 MiB Drive resumable uploads with validated server offsets, status probes, cancellation, and retransmission limits.
apps/media-server/src/lib/container-memory.ts Computes container working-set pressure by conservatively subtracting only clean inactive file cache.
apps/media-server/src/lib/media-video.ts Routes Drive resumable destinations through the new chunked uploader while preserving upload receipt verification.
apps/web/workflows/finalize-desktop-recording.ts Persists recognized capacity waits, honors server-directed delay, and retains the same processing attempt.
apps/web/lib/desktop-recording-jobs.ts Adds fenced capacity-wait persistence and prioritizes active interrupted jobs during recovery admission.
apps/web/lib/media-processing-budget.ts Removes the aggregate daily allowance while retaining attempt-idempotent and per-recording reservation bounds.
apps/web/lib/media-server-backpressure.ts Parses numeric and date-form Retry-After values into bounded, jittered capacity delays.
.github/workflows/docker-build-media-server.yml Expands architecture-specific image validation to cover resumable uploads, storage uploads, memory accounting, and large transfers.

Reviews (3): Last reviewed commit: "fix: prioritize interrupted recordings o..." | Re-trigger Greptile

Context used (3)

Comment thread apps/web/workflows/finalize-desktop-recording.ts Outdated
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@richiemcilroy
richiemcilroy merged commit 34f9a76 into main Sep 6, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant