fix: process large recordings reliably without daily allowances - #2231
Merged
Conversation
Member
Author
|
hey @greptileai, please re-review the PR |
Member
Author
|
hey @greptileai, please re-review the PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large retained recordings could be held until midnight by a shared 512 GiB reservation cap. A 13.2 GB Drive output also failed after its single upload request stopped at 8 GiB, while another recording aborted because the memory guard counted reclaimable file cache as application memory.
Remove the daily admission limit while retaining immutable-attempt reservations, per-recording retry bounds, direct storage transfer and worker capacity controls. Explicit busy responses now honor server backoff with a stable spread between recordings, persist the waiting status, and retain the same attempt and reservation. Recovery selects interrupted active work before the old incomplete-source backlog and excludes exhausted or retired jobs before applying batch limits. Drive outputs upload in 32 MiB chunks, reconcile confirmed offsets after interruptions, and bound retries and retransmitted bytes. Memory pressure excludes clean inactive file cache while retaining dirty pages and process memory in the guard.
Validation: 88 web tests; 37 upload and memory tests; 42 worker and real-media route tests. A real local HTTP transfer of a 9 GiB sparse file completed across 289 chunks with 158 MiB peak RSS growth. Both Linux production-image architectures run the upload, memory and large-file regression gates in CI. No database schema change. Full source/output decode, content identity and preservation checks remain enforced.
Greptile Summary
This PR improves reliability for large retained recordings and media outputs.
Retry-Afterdelays.Confidence Score: 5/5
The PR appears safe to merge, with the prior capacity-wait finding resolved and no new actionable defects identified.
Recognized busy responses now persist a waiting state, honor bounded server-directed backoff, and retain the same attempt. The newly added recovery admission changes preserve eligibility for deferred work, while Drive upload and memory-accounting paths include conservative validation and focused regression coverage.
Important Files Changed
Reviews (3): Last reviewed commit: "fix: prioritize interrupted recordings o..." | Re-trigger Greptile
Context used (3)