Repository navigation
fix: race-free yt-dlp update (staging + atomic symlink swap) - #3
Merged
Merged
Conversation
pip --target wrote into the live PYTHONPATH directory non-atomically and
the 403 update check ran while jobs were still downloading, so a job could
import a half-written library (ModuleNotFoundError: yt_dlp.extractor.youtube).
- install into <config>/ytdlp-libs/.staging, park it as yt-dlp-<version>
and flip <config>/ytdlp-lib with a temp symlink + os.replace; previous
install parked as ytdlp-lib.prev (never deleted), versions rotate to newest two
- flag ytdlp_updating: the queue worker waits instead of starting a job,
discovery skips (also force=True) with 'skipped: updating'
- runner classifies ModuleNotFoundError while listing as infra ('was being
updated, retry'), not as a broken playlist
- 403 path stays wait_for_idle=False (the gate covers it)
- tests: queue gate, discovery skip, failed pip keeps old install, successful
swap flips the symlink, swap units (park/.prev/rotation)
SPEC 6.7/14, AGENTS, PLAN decision log, README updated. 172 tests green.
Long tracebacks in jobs.error_summary wrapped across rows and broke the job history layout. Clamp to 40ch with an ellipsis (title shows the full message, the job log has it anyway). SPEC 14 updated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
pip install --target /config/ytdlp-libwrote the live PYTHONPATH directory non-atomically, and the 403 update check (update_ytdlp(wait_for_idle=False)) could start while jobs were still downloading (the wait only covered the current job, discovery not at all). A job could import a half-written library:ModuleNotFoundError: No module named 'yt_dlp.extractor.youtube'while listing a playlist.Changes
<config>/ytdlp-libs/.staging, the version is read from the staged copy, then the install is parked asytdlp-libs/yt-dlp-<version>and<config>/ytdlp-libis flipped via a temporary symlink +os.replace. Any concurrent run sees a complete library (old or new); a failed pip run keeps the previous install.ytdlp-lib.prev(never deleted, replaced on the next swap), versioned directories rotate to the two newest.ytdlp_updating– the queue worker waits instead of starting a job, discovery skips (including the manual button) withskipped: updating. A running job finishes; the next one starts against the new library.runnerclassifiesModuleNotFoundErrorwhile listing as an infrastructure error ("yt-dlp installation was being updated, retry") instead of a broken playlist.wait_for_idle=False(the gate covers it), the daily update keeps its idle wait and theasyncio.Lock.jobs.error_summaryis clamped to one line in the job history and playlist tables (ellipsis, full message on hover / in the job log) – long tracebacks no longer break the table layout.Tests
8 new tests: queue gate, discovery skip, failed pip keeps the old install, successful swap flips the symlink, swap units (park /
.prev/ rotation),lib_env/ path helpers.ruff check .clean,pytest -q172 passed,node --check app/static/app.jsok.Docs: SPEC §6.7 + §14, AGENTS, PLAN decision log, README.