Repository navigation
perf(prd-262): API — W2–W5, pull-first self-host, boot, and the 19 Oct cuts (1,920 → 789 MB) - #1226
Merged
Merged
Conversation
… first run that pulls Grounded on main @ 73f18f1 and registry/CI measurements of 16 Oct 2026. Reviewed by 18 adversarial agents (213 confirmed or partly-confirmed findings applied). §3 and §9 Q1-Q13 taken at the recommendations on 16 Oct; Q14-Q21 open. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
The frontend (3000), Adminer and Gotenberg published on all interfaces: their ports: lines had no host IP, while every other service used ${BIND_ADDRESS:-127.0.0.1}. In the local edition the web app's chat proxy forwards an anonymous session the backend serves as the owner, so anyone on the network could chat with the backend as the owner. The three lines now carry the ${BIND_ADDRESS:-127.0.0.1}: prefix; Gotenberg's line goes with the service in US-018.
test_prd234_s1a_compose_bind.py checked five named variables, which is how these three slipped through. It now reads every ports: entry of every service in every tracked compose file (short and long syntax, compose's !reset/!override tags), requires the BIND_ADDRESS host IP, and proves itself non-vacuous: the canonical file is found, its core services publish, and parametrised cases show the checker refuses the old frontend line, bare ports, 0.0.0.0, a hard-coded 127.0.0.1, ${BIND_ADDRESS} with no default and a long-syntax entry without host_ip. The five-variable test is replaced, not kept.
The compose header, self-hosting.md's Network exposure section and the threat model now say every published port, the web app's included, is on loopback by default.
Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…7.0.0.1 Once the backend answers /health/ready, smoke-fresh-clone.sh runs scripts/ci/smoke-loopback-ports.sh before tearing down. The check starts the frontend from the checkout when it is not already running (--no-deps, so the ready backend is left alone), requires 'docker compose port frontend 3000' to print 127.0.0.1:3000 (0.0.0.0:3000 before US-005), then reads every running service's published ports from 'docker compose ps --format json' and fails on any address other than 127.0.0.1. It writes the port table to the step summary. The smoke's success exit carries the check's result; teardown still runs either way. The helper is its own file so the W0 smoke rewrite (US-002), which also starts the frontend, can call it in one line. The workflow's push and pull_request path filters add the helper. test_prd234_s1a_compose_bind.py guards the wiring: the call and exit in the smoke script, the expected address and the port command in the helper, and the path filters. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
docker compose prints interpolation warnings on stderr. Capturing 2>&1 would put a warning line into the address and fail the 127.0.0.1:3000 comparison for the wrong reason; stderr now goes to the log. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
scripts/ci/image_report.py reads a pushed image back from its registry and reports what it weighs and what it costs an existing user to update: - measure: picks the build's arch out of the index BuildKit pushes (skipping the unknown/unknown attestation manifest), sums the layer sizes, pairs each layer with the Dockerfile instruction that made it (non-empty history entries) and a kind (system / dependencies / code), lists each layer's paths (every prefix to depth 5, the 25 heaviest at depth 6, so the pip layers and site-packages show), and diffs the layer digests against the published edge. The delta is split by kind, so a requirements change is reported apart from a code-only change; an image with no edge reports its size only. JSON to an artifact, Markdown to the step summary. - budget: checks the size against .github/image-budgets.json. The file is in warn mode (report only); US-004 sets the five-run baselines and flips it to fail. Budgets start at edge 0d9f0ac's sizes rounded up to 0.1 MB, each with a reason. - report: per arch, the layers the API and the worker share (today only the 29.8 MB Debian layer; layers under 1 KiB are left out of the count) and the app images' first pull, using edge for any image the run did not build. image_registry.py reads registries with the standard library only: the arm64 runner image has no crane or skopeo, and imagetools cannot fetch layer blobs. Bearer tokens are anonymous unless REGISTRY_USER/REGISTRY_TOKEN are set, and are never forwarded on a blob redirect. No new dependency. Tests feed manifests and configs recorded from the published edge images (tests/fixtures/prd262_edge_images.json): edge plus a 10 MB public/ layer is a +10 MB code-only update and the committed budgets warn on it; a pip-layer change is filed as dependencies; the platform pick skips the attestation manifest; the API and worker share one real layer; path listing, zstd refusal, and the three CLI commands against a fake registry. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…anged images, measure every build images.yml: - PR builds, and branch builds started by workflow_dispatch, push by digest to a registry:2 service in the job (localhost:5000, buildx driver-opts network=host) instead of building without pushing, so they can be measured (and, from US-013, scanned) without writing to GHCR. Main and v* builds push to GHCR as before; their service image is empty, so no registry starts. - A changes job (scripts/ci/image_matrix.py) picks the build matrix: a PR builds only the images whose inputs it touches, on both arches; a change to the workflow, the budgets, .grype.yaml or the report scripts builds them all; a push or a manual run builds every image. The image definitions moved into its IMAGES (with a publish flag, for the scan-only images US-006 and US-054 add). - pull_request paths add frontend/next.config.js, csp-sources.js, .npmrc and public/**, .github/image-budgets.json and .grype.yaml (D1), and the report scripts. A test holds the paths equal to the images' inputs. - Every build runs image_report.py measure and budget and uploads its layer report; a report job adds the API/worker shared layers and the first pull and exposes them as job outputs (US-058 reads them). - §9-Q13: cache-to is written only by publishing runs (PR and branch builds read the GHA cache but never write it), with mode=min for the API and max for the frontend and worker. test_compose_published_images.py reads the published image names from the changes job's IMAGES now, and also checks the merge job publishes the same set. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
Every route but /health and /metrics now sits on one APIRouter whose dependency requires X-Internal-Token equal to AGENT_OPT_WORKER_TOKEN (hmac.compare_digest, worker_auth.py). It is a router-level dependency, not a path comparison, because starlette 0.41.3 lets a forged Host header rewrite request.url.path (GHSA-86qp-5c8j-p5mr), and it never skips the check for an empty token: worker_config.internal_token() stops the import, and so the start-up, when the token is unset or blank. FastAPI(docs_url=None, redoc_url=None, openapi_url=None). GET /test, which ran three FutureAGI evaluations for anyone, is deleted. Inputs are capped at what the platform sends: metrics are known template names, at most one per template (11; /assess and /score start a thread and an evaluation per entry, and an unknown name still cost a turing_flash call); scoring_template is a known template; teacher_model is allow-listed (gpt-4o-mini); dataset holds at most 10 items; and a BoundedSemaphore (AGENT_OPT_MAX_CONCURRENT_JOBS, default 2) bounds concurrent /optimize jobs, answering 429 beyond it. The job thread frees its slot however the job ends. The service gets its own suite (tests/, pytest.ini, requirements-test.txt with pytest==9.0.3 as in media-render, installed only in the image's test stage). It runs inside the image with TestClient: 401 for every protected route with no, empty or wrong token and with forged Host headers; start-up fails with the token unset, empty or blank; /docs, /redoc and /openapi.json are 404; more than 11 or unknown metrics, a non-allow-listed teacher_model, more than 10 dataset items and an unknown scoring_template are 422; the job bound answers 429 and frees its slots. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…e PyPI package fi requirements.txt becomes requirements.in, and requirements.lock is generated from it with uv pip compile --generate-hashes (Python 3.11, x86_64 Linux; the command is in requirements.in and the lock header). The Dockerfile installs the lock with pip --require-hashes --no-deps, so every Railway build installs exactly the 94 pinned, hashed packages instead of whatever the floating ranges resolved to at the last cache miss (24 of 99 pins matched between March and October). overrides.txt removes fi from the resolution with a never-true marker. agent-opt 0.0.1 requires fi>=0.0.2, an unrelated 2019 sdist that runs setup.py at build and would install any future upload of that name. The fi.evals and fi.opt modules the worker imports come from ai-evaluation and agent-opt themselves; the CI lane imports both inside the image. ai-evaluation's floor moves to 1.1.0 because 0.1.x still required fi. Dependency changes, all inside this one service and none new to it: - uvicorn[standard]==0.34.0 becomes uvicorn==0.34.0: uvloop, httptools, watchfiles and websockets (17.8 MB installed) go; this service uses none of them. - fastapi 0.115.6 -> 0.144.0 and starlette 0.41.3 -> 1.8.0 (a starlette>=1.0.1 floor in requirements.in), past GHSA-86qp-5c8j-p5mr. The resolver allows it: fastapi 0.144.0 requires starlette>=0.46.0 with no upper bound. - litellm stays at 1.80.0: ai-evaluation 1.1.0 requires openai<2 and litellm 1.80.5+ requires openai>=2.8.0. Its Criticals go to the US-013 allowlist with that reason. Dockerfile: useradd runs first and the code is copied with COPY --chown=1000:1000, so the chown -R layer goes (taken over from US-016). Stages are runtime, test (pytest on top, for CI) and production, which stays last because Railway builds no target. No BuildKit cache mount: Railway fails one without a hard-coded service id, and that id is unknown (PRD-262 section 10). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…endored metrics copy
services/shared/automatos_metrics.py labelled requests_total and request_duration with the raw request path. Every /optimize/{job_id} poll, every scanner path that reached the public domain and, under starlette 0.41.3, every forged Host header (request.url.path) added a counter series per status and a histogram family, kept until the next restart.
The FastAPI middleware now reads the matched route from the ASGI scope after call_next (scope['route'].path, set by FastAPI's APIRoute.matches in 0.115 and 0.144 alike) and the aiohttp middleware reads request.match_info.route.resource.canonical, which aiohttp resolves before any middleware runs. A path no route matches (404, and aiohttp's 405) is labelled 'unmatched'. A method outside GET/HEAD/POST/PUT/PATCH/DELETE/OPTIONS is labelled 'OTHER', since the client chooses it too. The /metrics skip reads scope['path'], not request.url.
The fix is made in services/shared and copied byte for byte to agent-opt-worker and workspace-worker. test_prd262_vendored_metrics.py (required orchestrator suite) fails when any services/*/automatos_metrics.py differs from services/shared, and drives both middlewares with job ids, scanner paths, a forged Host and PROPFIND, asserting one label per route template plus 'unmatched'. The worker's in-image suite checks the same on the real app. US-045's later /metrics content-type fix goes to the same copies.
Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…ion while it waits futureagi_service sends X-Internal-Token on every worker call (POST and the optimisation poll) from the new setting AGENT_OPT_WORKER_TOKEN (config.py, listed in reports/config-surface.json by hand, as the generator would sort it; the generator needs the app importable). The Railway manifest and infrastructure/.env.example name it for both services. The per-turn live-eval hook (consumers/chatbot/service.py _post_response) now calls futureagi_service.schedule_live_eval, which reads a cached 'any prompt has scoring on' flag. While the flag is fresh and off, nothing is scheduled and the database is untouched; before, every chat turn opened a SessionLocal and ran a SELECT on the event loop. A stale flag (older than SCORING_FLAG_TTL_SECONDS, 60 s) is re-read in a thread by the scheduled task; the admin toggle drops this process's copy at once. eval_live_traffic awaits the worker with no session checked out and writes the live runs in a new short session in a thread, so no pooled connection sits idle in a transaction for up to 120 s. Scheduled tasks are held in a set until they finish. run_assessment reads and marks the run running in one short session that commits and closes (a thread), awaits the worker (assess, safety, or optimize_prompt, which polls for up to 1,500 s) with no session checked out, and writes the result in a new session. Before, one pooled connection (pool 10 + overflow 20) and one threadpool token were held for up to 25 minutes. Failures are logged with logger.exception and recorded on the run in their own session. test_prd262_futureagi_worker_client.py: the header on both calls (and none when the token is empty); a fresh-off flag schedules nothing and touches no database; a stale flag is re-read off the event-loop thread; no session is checked out while live scoring awaits the worker or while optimize_prompt starts, sleeps and polls; results land in a new session; a worker failure and a missing version fail the run. Data scope is unchanged: live scoring and _collect_optimization_dataset send what they sent before (the PRD's open data question). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…suite inside the image Nothing built, linted or tested services/agent-opt-worker before; only Railway built it. test.yml's changes job gains an agent_opt lane (^services/agent-opt-worker/ and test.yml; every lane runs on main and by hand). The agent-opt-worker job: - fails on a '^fi==' line in requirements.lock and on any pin without a hash, and on uvicorn extras; - builds the image with no target, as Railway does; - imports fi.evals and fi.opt.optimizers inside it, and fails if pip shows the PyPI package fi; - runs the image with no token and requires a non-zero exit naming AGENT_OPT_WORKER_TOKEN; - builds --target test and runs the service's own TestClient suite inside it; - runs the container with a token and checks /health and /metrics answer, every protected route 401s without the token (and with a wrong one or a forged Host), and /docs, /openapi.json and /test are 404. test_prd262_agent_opt_worker_image.py (required orchestrator suite) pins the files a later edit could quietly loosen: requirements.in plus a fully hashed lock, fi kept out by overrides.txt, plain uvicorn, starlette >= 1.0.1, production last, no cache mount, --require-hashes --no-deps, useradd before COPY --chown with no chown -R, docs off, only /health on the app, and the lane itself. The images.yml build-and-scan (local registry, US-013 policy, size budget, litellm allowlist) is US-013's, after W0 merges into this lane. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
agent-opt-worker-service.md gains an Access section (the token on every route but /health and /metrics, no API docs, the input caps, why /metrics stays open), AGENT_OPT_WORKER_TOKEN and AGENT_OPT_MAX_CONCURRENT_JOBS in its variable table, and the locked install. live-traffic-scoring.md lists the token for both services. The prompt-optimization pages point at requirements.in instead of the deleted requirements.txt. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…fails to start Once optimize() took a job slot, an exception before the worker thread started (the stale-job sweep, the job-store insert, Thread()) leaked the slot. The route is a plain def, so concurrent requests run in FastAPI's threadpool, and the sweep iterating the store while another request inserted could raise. After MAX_CONCURRENT_JOBS leaks every /optimize answered 429 until restart. Everything after acquire now sits in one try that releases on any exception, and the job store is guarded by a lock (sweep over a snapshot, pop instead of del). Found by the automated commit security review. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
scripts/ci/smoke-fresh-clone.sh is now the measurement behind PRD-262 §8's boot bars: - Every boot is timed from the container's own start (docker inspect State.StartedAt) to the response that first carried 200, polled every second (was 5 s and the script's own sleeps). /health and /health/ready are timed separately. - After the fresh boot it starts the frontend, asserts :3000 answers 200 (following / -> /chat) and records what `docker compose port frontend 3000` reports (0.0.0.0:3000 today; US-005 makes it the 127.0.0.1 assertion). - Two warm phases run `docker compose restart backend` with the database kept and are timed the same way; the report gives their median. - The step summary is written on every exit, pass or fail (it used to print logs only on failure): the boots, Started -> Healthy per service and per start from Docker's start/health_status events (streamed to a file, since the daemon replays only a short buffer), the /health/bootstrap stage table of each boot, and the alembic runs per boot counted the §8 way (Context impl PostgresqlImpl lines after 'init_fresh_db: done', or all of them when init_fresh_db did not run; review addenda S2/S9). The raw Context impl count is shown beside it. - The images are built first and the build is timed for the record only (§7: no CI build-time bar). The records, the report JSON and the stack's logs are uploaded as the smoke-fresh-clone-boot artifact, which US-004 takes its five-run medians from. No time is a bar yet: the lane fails only when a boot or the frontend never answers 200. scripts/ci/smoke_boot.py polls and records one boot; scripts/ci/smoke_report.py writes the summary and the JSON. Standard library only, run by the runner's python3: no new dependency. smoke-fresh-clone.yml: - paths add services/workspace-worker/**, docker-compose.dev.yml, Makefile, frontend/Dockerfile, orchestrator/requirements*.txt, frontend/package-lock.json, orchestrator/main.py, core/seeds/**, core/database/load_seed_data.py, alembic/**, envs/*.defaults (replacing envs/api.defaults), and from the review addendum orchestrator/requirements.in and requirements*.lock (US-022). Also orchestrator/docker-healthcheck.sh and alembic.ini (they decide Started -> Healthy and the alembic log lines) and the two new tools. - a nightly schedule (02:41 UTC), since most main pushes skip the filter; - actions/checkout and upload-artifact pinned by SHA (v7.0.1, as images.yml); persist-credentials false; permissions contents: read; - runs-on ubuntu-24.04, so baselines compare like with like; - only pull requests cancel an older run; main, nightly and manual runs are measurements and finish. test_prd262_smoke_fresh_clone.py: the poller times from the container start on a 1 s tick (fake clock), gives up at the timeout and records liveness and readiness apart; the report's alembic count, Started -> Healthy, warm median, bootstrap table and frontend address from recorded records/events/log, and a summary on a run that failed before any boot; the script's frontend, warm-restart and report-on-exit wiring; the workflow's paths, schedule, SHA pins and artifact. test_prd209_boot_lanes_hard.py's guards still hold (readiness probe, exported DEFAULT_WORKSPACE_ID, no continue-on-error). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…e with only the node binary
The base stage (development and builder) moves from node:20-alpine, which reached end of life on 30 April 2026, to node:24-alpine3.24, pinned as tag@digest (24.22.0, index digest looked up on Docker Hub on 18 Oct). It drops apk python3, make, g++ and curl: no package in the lock compiles on Linux (sharp, esbuild, unrs-resolver and @next/swc ship prebuilt binaries; Prisma and Clerk only download or print).
The production stage is alpine:3.24 (tag@digest) plus libstdc++, with /usr/local/bin/node copied from the base stage and nothing else from it: no npm, yarn, corepack, C headers or curl. The musl binary is built for Alpine 3.24, so the base tag's alpine3.24 suffix and the runtime's alpine:3.24 name the same release; the comment says to bump them together. Streaming the node:24-alpine3.24 amd64 layer shows the binary needs only libstdc++.so.6 and libgcc_s.so.1 beyond musl.
The HEALTHCHECK keeps its options and uses busybox wget on /favicon.ico at 127.0.0.1:${PORT:-3000}: the middleware matcher skips paths with an extension, so a probe never renders a page or calls Clerk, and the server listens on IPv4 only (HOSTNAME=0.0.0.0).
FROM ... as becomes FROM ... AS on all four stages (BuildKit's FromAsCasing warning). production stays the last stage. The docs' frontend stage table and line references follow.
Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…e official image, not NodeSource workspace-worker (Node 20, end of life) and media-render (Node 22) installed Node from deb.nodesource.com. That brought two extra apt updates, Debian's python3.13, gnupg and 49-59 MB of C headers, and on 15 Oct a 7.5-minute download stall in CI. Both now take Node 24 (24.22.0) from a named stage, FROM node:24-trixie-slim@sha256:... AS node, so the tag@digest pin sits on a FROM line, the only place an updater reads (a bare COPY --from=<image> is never read). They copy /usr/local/bin/node, npm and corepack, and link npm, npx and corepack into /usr/local/bin as the official image does. /usr/local/include is not copied: node-gyp downloads the headers on its first native build. Libraries: the glibc node binary needs only libstdc++6 and libgcc-s1 beyond glibc (DT_NEEDED read from the amd64 and arm64 node:24-trixie-slim layers). The NodeSource deb used to declare them; the worker gets them from build-essential, and media-render now names libstdc++6 on its apt line. Worker: pnpm is pinned exactly to 12.14.0, the current 12.x on npm on 18 Oct (PRD-262 §9-Q9), and npm cache clean --force runs in the same RUN, so the 31 MB /root/.npm cache is not in the layer. Global installs move from /usr to /usr/local; the agent sandbox PATH already lists /usr/local/bin first. Adding the node stage makes the worker Dockerfile multi-stage, so its own stage is named production and stays last (FR-8); Railway, images.yml, test.yml and compose all build it with no target, so what ships is unchanged. This also gives US-011 the stage name it needs. media-render: npm stays for the global Hyperframes install and versions.py's npm root -g (unchanged until US-053). Tests that pinned the old Node move with it: test_hyperframes_is_pinned_exactly_on_node_22 becomes ..._on_node_24 and asserts the named stage instead of the NodeSource script; the in-image test_hyperframes_is_pinned_exactly_and_runs_on_node_22 becomes ..._node_24 and expects v24. The PRD-251 W1 acceptance script's Node check accepts the official image (22 or later). The OSS Scanner image, which mirrors the frontend CI lane, moves to node:24-bookworm through a named stage too. Docs follow, including line references the new stage shifted. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… built images frontend-ci (test.yml) and browser-e2e set up Node 24 instead of 20. browser-e2e is newer than the PRD's count of one node-version line; it moves too, so no CI lane runs an end-of-life Node. prod-image-parity (test.yml), which already builds the frontend and worker with Railway's no-args defaults, gains two steps: - frontend: node -v prints v24., and command -v npm, curl, yarn and corepack each fail, one name per check; - worker: node -v prints v24.; npm, npx and corepack answer; pnpm --version equals the Dockerfile's exact pin (read from the Dockerfile, not repeated); dpkg -s python3.13, nodejs and gnupg each fail, one package per check; /root/.npm/_cacache and /usr/local/include/node are absent. media-render.yml checks node -v and the same three dpkg packages right after its build. That workflow runs on dispatch only, so the ship agent dispatches it on the wave branch. New orchestrator/tests/test_prd262_node24_images.py (pure file reads, run by the required orchestrator job): - frontend base on node:24-alpine3.24 with tag@digest and no apk build tools; runtime is alpine:<same release>@digest, the last stage, copying only /usr/local/bin/node from base, apk-adding only libstdc++, naming no npm, yarn, corepack, curl or headers; HEALTHCHECK is wget on /favicon.ico; - worker and media-render: a node stage on node:24-trixie-slim@digest, production last, exactly node, npm and corepack copied from it, no NodeSource, headers, gnupg or python3.13; - the worker's pnpm pinned exactly to a 12.x and the npm cache cleaned in the same RUN; - every Dockerfile that runs Node (including .oss-scanner) copies only from named stages, and every node image is node:24-*@sha256; - every actions/setup-node step in .github/workflows runs 24; - the parity and media-render jobs keep the checks above. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…checks what ran scripts/ci/smoke-fresh-clone.sh gains SMOKE_IMAGES, so smoke-images.yml runs the same boot steps as the fresh-clone lane on images instead of a source build: - pull: docker-compose.images.yml with AUTOMATOS_IMAGE_TAG (today's published path). The Docker image cache is emptied first, then `docker compose pull` is timed and the bytes the runner's default-route interface received are counted. - prebuilt: the PR variant. The workflow built the three amd64 images under the GHCR names, so compose neither pulls nor builds. Both start the stack with `up --no-build`; unset, the lane builds as before. scripts/ci/smoke_images.py does what is not the boot itself: - target: sha-<7> after a main publish, pr-<7> on a PR, a manual run's tag (validated; a sha-<7> tag names its commit's prefix). - workers: the first CI boot of the production stage. From `docker top -eo pid,ppid,args -ww`, the backend must run uvicorn without --reload and with the production CMD's 4 workers (the spawn_main children of the uvicorn process). - images: each app container runs the ghcr.io/automatosai/automatos-* image compose names, carrying the expected org.opencontainers.image.revision. When pulled, its RepoDigests must hold the index digest the tag resolves to on GHCR (a local build has none). No image may carry a local-* tag or compose's <project>-<service> name. It also sums the app images' unique compressed layers. - report (smoke_images_report.py): pull seconds and bytes, the first run from the pull's start (or compose up's) to /health/ready 200 and :3000 200, the provenance table and the process check, on every exit, plus smoke-images-report.json beside US-002's smoke-report.json for US-004. image_registry.resolve(ref) reads the digest a tag points at (HEAD and Docker-Content-Digest, else the hash of the served manifest). smoke_report.py takes a --title, so each lane names its summary. Docker Hub 429 recording (§9-Q17) is not added: the question is open. Standard library only; no new dependency. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…s only after it passes New .github/workflows/smoke-images.yml: - smoke-images: runs after images.yml publishes on main (workflow_run: a successful push or manual run from this repository's main) and on workflow_dispatch with a tag. It runs on ubuntu-24.04 and ubuntu-24.04-arm, checked out at the commit the images were built from. It pulls sha-<7> into an empty Docker cache and boots it through scripts/ci/smoke-fresh-clone.sh (SMOKE_IMAGES=pull), uploading the records as smoke-images-<arch>. - promote-edge: once both arches pass for a main publish, scripts/ci/promote_edge.py plans every image, then runs `docker buildx imagetools create -t …:edge …:sha-<7>`. With one source and no annotations, edge gets the same signed digest, and the tool reads it back to check. It moves edge only when this commit is ahead of, or the same as, the commit edge was built from (its revision label against GitHub's compare API). So a re-run of an old smoke never moves edge back, and one unreadable image moves none. A manual run never promotes. The job is serialised (concurrency promote-edge) and is the only one with packages: write. - smoke-images-pr: on PRs that touch compose, the Makefile, envs/ or an image input (images.yml's PR paths, plus the backend's entrypoint and healthcheck and this lane's files). It builds the three amd64 images in the job with images.yml's context, target, build args and metadata labels, reading its GHA cache and never writing it (§9-Q13). It loads them as ghcr.io/automatosai/automatos-*:pr-<7> and boots them with SMOKE_IMAGES=prebuilt. It measures the boot only; the pull numbers come from main runs. images.yml's merge job pushes sha-<7> (and semver tags) but no longer edge. Report only until US-004: no time or size fails a job. A boot that never answers 200, a built image, a missing or wrong revision label, or --reload or the wrong worker count fails it. test_prd262_smoke_images.py holds the workflow to images.yml (PR paths are a superset; the PR builds repeat IMAGES' context, target, args, cache scope and labels; the tags are docker-compose.images.yml's names; promote-edge moves exactly the merge job's images). It also holds the script's modes, and the expected worker count to orchestrator/Dockerfile's production CMD. test_prd262_promote_edge.py covers ahead, identical, behind, diverged, no label, no edge yet, a partial failure and a move that lands elsewhere. CONTRIBUTING lists the lane; QUICKSTART says edge is the latest main build whose published images booted in CI. No new dependency. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
Seam review of W0. The image tools caught registry read failures with (URLError, LookupError, ValueError, OSError), copied in image_report.py, smoke_images.py and promote_edge.py. A blob or manifest body that breaks off mid-read raises http.client.IncompleteRead, an HTTPException that is neither a URLError nor an OSError, so it escaped every handler: - image_report.py _listing (the per-layer path listing, report only) let it fail the build job; on main that skips the merge job, so nothing publishes and edge never moves; - read_edge did the same instead of reporting 'no update delta'; - smoke_images.py app_layer_bytes (report only) crashed the images check; - promote_edge.py exited with a traceback instead of its 'edge stays where it is' error. image_registry.py now defines READ_ERRORS once, with http.client.HTTPException added, and the three tools use it instead of their own copies. Each consumer gets a test that feeds it an IncompleteRead. No new dependency. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… with every route walk reading FastAPI's route tree starlette 0.41.3 carried GHSA-86qp-5c8j-p5mr (on CISA's KEV list). The web stack moves to fastapi 0.146.0 (the current release), starlette 1.8.0 and uvicorn[standard] 0.50.1, the newest uvicorn whose [standard] extra still accepts websockets 12.0 (0.50.2 needs >= 13). Its --workers manager restarts a worker that dies (since 0.30). All four are pinned exactly. uv pip compile resolves the file for linux amd64 and arm64, with and without the [leiden] extra; the only other change in the resolution is annotated-doc 0.0.5, a dependency of fastapi. F092 is why this is not a version edit. Since FastAPI 0.137, include_router keeps each included router as one node of app.routes, with no path and no endpoint, instead of copying its routes in. A plain walk over app.routes then sees about a dozen routes, which is F092's symptom. Every walk now goes through fastapi.routing.iter_route_contexts, FastAPI's public API for this, which gives each served route with its full path and inherited dependencies: - scripts/dump_routes.py, so reports/route-manifest.json is unchanged by the bump; - scripts/scaffold_platform_tool.py's route lookup; - tests/authz_sweep_probe.py. Without the change, the sweep would silently have covered only the top-level routes. - The socials tests that look up routes defined in the routers api/socials.py includes. They do not change what they assert. slowapi is pinned to 0.1.10, which the old range already resolved to. The next commit subclasses its middleware. New test_prd262_web_stack.py: - checks the pins, and that the suite runs on the pinned versions; - runs tests/web_stack_probe.py in a fresh process with the database unreachable. The probe asks the app's routing about every HTTP path of every router in router_manifest.py, and every HTTP route in the committed manifest. It uses a method no route takes, so 405 means the path is served and 404 means it is lost. No middleware, lifespan or endpoint runs. - requires every committed manifest route to still be collected by dump_routes. The entrypoint's watchdog comment no longer says the --workers supervisor outlives a dead worker. That was true only before 0.30. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…s of included routers FastAPI 0.137+ changed two more readers of the route list besides the route walks. 1. slowapi 0.1.10's middleware looks a request's endpoint up by walking app.routes for a full match that has an endpoint. An included router is now one node of that list, with no endpoint. So the lookup finds nothing for a route of an included router, which is nearly every route of this API, and slowapi exempts the request. Under the policy plane (F040), main.py's 60/minute default limit would therefore have applied to almost nothing. - RouteTreeSlowAPIMiddleware (core/security/rate_limit_middleware.py) keeps slowapi's dispatch line for line. It looks the endpoint up with core.route_tree.matching_endpoint, which walks iter_route_contexts and takes the last full match, as slowapi did. - slowapi is pinned to 0.1.10 for that reason. - test_prd174_flag_gating now requires the subclass, registered under the flag. 2. A routed request's scope['route'] is now the route as its router declared it, without the prefixes it was included under. main.py's API stats keyed requests by scope['route'].path, so routes under an include prefix would have lost it: /api/alerts would have become /alerts, and the routes api/socials.py includes would have lost /api/socials. core.route_tree.route_template gives the full template again. - It finds the route's served contexts once per app (cached). - When a route is included under two prefixes, the one that matches the request's path wins. - It uses only FastAPI's public route API. test_prd262_route_tree.py checks: - the template for a route of a nested included router, for a route included twice, for a plain route and for a 404; - the endpoint lookup; - that the subclass returns 429 on the second request to a route of an included router; - that slowapi's own middleware still exempts that route. When that last check fails, slowapi reads the tree itself and the subclass can go. main.py's line count is unchanged. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…corn to replace it scripts/ci/smoke_worker_restart.py is the kill-a-worker step for smoke-images' pull-request variant (US-003). That variant boots the production stage through compose, with no /app bind mount, and the CMD runs uvicorn --workers 4. The step runs after the variant's worker-count check. 1. It reads the backend's processes from /proc inside the container, using the container's own python, because the slim image has no ps. From them it finds the uvicorn process and its --workers children, which run multiprocessing's spawn_main. It never takes the sh -c 'alembic ... && uvicorn ...' process for uvicorn. 2. It kills the newest worker with SIGKILL. 3. Within 30 s it requires all of these: - one new worker under the same uvicorn process, with the worker count back where it was; - 'Started server process [<new pid>]' in the container's log; - --health-url answering 200 after that; - an unchanged restart count and start time, so the container itself never restarted. uvicorn 0.24 never replaced a dead worker. The step refuses a single uvicorn process, because a single process has no manager to restart it. Once US-029 takes the worker count from WEB_CONCURRENCY, the job sets WEB_CONCURRENCY=2. The step writes a JSON report and a step-summary table. It uses the standard library only, like the other smoke tools. smoke-images.yml and its image-mode boot are W0's (US-003) and are not on this lane's base, so the step is wired into W0's smoke-fresh-clone.sh when W1 is rebased onto W0. That script marks the spot: 'PRD-262 US-008's kill-a-worker step goes after this check.' test_prd262_worker_restart.py covers: - finding the processes; - the replacement rule (new, logged, count restored, killed worker gone); - the three failure cases and the one-worker refusal; - the report and the exit code; - the /proc listing, run on the CI runner. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…ile over 800 lines check_changed_code_shape.py warns when a file over 800 lines grows. test_prd251_api.py had grown by one comment line in the previous commit, so the comment moves onto the loop it describes. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
FastAPI 0.132 added strict_content_type, on by default. A body with no Content-Type header is not parsed as JSON, so a typed body parameter answers 422. Under 0.115 such a body was parsed. A browser sends a Blob body with no Content-Type without a CORS preflight, and the local edition has no login, so the old behaviour let any web page the owner opened call a JSON endpoint. The API keeps FastAPI's default: - the frontend's apiClient sends application/json; - a grep of the Python callers found none that post JSON without the header; - the public webhooks (api/webhooks.py, the recipe webhook) read the raw request, which this setting does not touch. The test pins the behaviour on the pinned FastAPI and requires that main.py never sets strict_content_type. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…e way only CodeQL (py/redos, high) flagged RUN_PREFIX in scripts/ci/image_layers.py: in (?:\S+=\S* )* a token such as '!==' splits several ways, so a history line starting 'RUN |0 ' followed by many '!== ' and no shell backtracks exponentially. A build-arg name now stops at its first '=' ([^\s=]+), so each NAME=value token has exactly one match and the scan is linear. Values may still contain '='. Tests pin the stripping (including a value with '=') and feed the CodeQL input, which pytest-timeout would kill if it ever backtracked again. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…tive builds by C library, no Prisma or unused dependencies sharp resolves to 0.35.5, inside next@15.5.27's optional range '^0.34.3 || ^0.35.4' (#1023 moved next but left sharp at 0.34.5, which carries three High findings). sharp 0.35.5 itself needs semver ^7.8.5, @img/colour ^1.1.0 and, through its wasm build, @emnapi/runtime ^1.11.3, so those three move too (7.7.4 to 7.8.5, 1.0.0 to 1.1.0, 1.8.1 to 1.11.3). No other version in the lock changes, and no dependency is added. The lock is written with npm 11.19.1, the npm Node 24.22.0 ships and so the one the Docker builder and frontend-ci now install with (US-007). npm 10 never persisted the libc field, so the @img, @rollup and @unrs native entries had os and cpu only, and npm installed both the glibc and the musl build everywhere: on Alpine, Next traced the glibc libvips (16.65 MB, unloadable there) into the standalone image. Every native entry whose published manifest declares libc now records it. The @rollup and @unrs entries (dev only) were re-resolved at their locked versions to pick the field up; only license and libc changed. The two @unrs arm 32-bit bindings publish no libc, so they stay as they were. A dry run of npm 11.19.1 against the new lock for linux x64 musl adds only the linuxmusl sharp and libvips builds (and the musl swc, rollup and unrs builds); against the old lock it added both libcs. Removed, because nothing imports, requires or configures them (git grep for each as a quoted module name, including import() and require(); tailwind.config.ts loads only tailwindcss-animate; .scrollbar-hide is hand-written CSS in globals.css; react-grid-layout appears only in a comment and CSS): - prisma and @prisma/client, with frontend/lib/db.ts (its only user, imported by nothing), frontend/prisma/ and the package.json prisma block, which seeded from scripts/seed.ts, a file that does not exist. That ends the Prisma engine download (about 14.8 MB) on every cold install; - @anthropic-ai/sdk, @floating-ui/react, @headlessui/react, @hookform/resolvers, @tiptap/react, @tiptap/starter-kit, dotenv, lodash, react-grid-layout, @types/react-grid-layout and tailwind-scrollbar-hide. npm pruned exactly 85 packages from the lock, the number the PRD's reachability walk gives. lodash stays in the lock as a dependency of recharts. The deepwiki line references into package.json follow the new line numbers; the Classic Shell reference pointed at @headlessui/react and now points at @radix-ui/react-dialog, which the Sheet uses. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…nside the image prod-image-parity (test.yml), which already builds the production frontend image, gains a step that lists node_modules/@img in the image, fails on any glibc build (sharp-linux-*, sharp-libvips-linux-*) and, when sharp is in the standalone trace, fails unless its version is the lock's (read with jq, not repeated). The WebAssembly fallback, @img/sharp-wasm32, prints a warning if it ever appears; npm 11 leaves it out on Linux. New orchestrator/tests/test_prd262_frontend_lockfile.py (pure file reads, run by the required orchestrator job): - sharp is 0.35.5 or later, and every @img/sharp-* build is at sharp's version; - the x64 and arm64 sharp and libvips builds record musl or glibc, and so does every other Linux build whose name targets one C library (the two @unrs arm 32-bit bindings publish no libc and are listed with that reason). The failure message says to write the lock with Node 24's npm, because npm 10 drops the field; - Prisma is gone: no package.json prisma block, no frontend/prisma/, no frontend/lib/db.ts, no prisma or @prisma/* in the lock; - none of the 13 removed dependencies is declared again (the test says to drop a name from its list once code imports it); - the lock's root entry equals package.json's dependency lists, and every lock entry is reachable from them through Node's lookup, so a lock that still carries a removed package's closure fails. Against the old lock and the new package.json this walk finds the 85 packages npm pruned; against the new lock it finds none; - the parity job keeps the image check above. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…the unread watch file goes Railway's config file has no target key: its build keys are builder, dockerfilePath, buildCommand, watchPatterns and the Nixpacks/Railpack ones. So railway.json's "dockerfileTarget": "production" was never read, and test_dockerfile_prod_parity.py asserted a value that proved nothing. Production ships only because it is the last stage of each Dockerfile; the next commit adds the test that checks that order. - railway.json: dockerfileTarget removed. - test_dockerfile_prod_parity.py: the assertion of it removed; the builder and manifest checks stay. - orchestrator/.railway-watch.json deleted. It is not a Railway file (Railway's watch paths are build.watchPatterns) and nothing in the repo reads it. - Comments that said Railway builds 'target production' (orchestrator/Dockerfile, test.yml's prod-image-parity note, the manifest's builder note) now say it builds the last stage. Comment lines are replaced one for one, so no line number moves. - Docs: production-deployment.md no longer says railway.json targets the production stage, and the railway.json line references in it and in system-health-monitoring.md follow the removed line. docker-containerization.md names the new rule and its test. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… Dockerfile Railway builds Railway builds the last stage of each Dockerfile, because its config has no target key. A stage added after production (a debug or test stage) would ship to production with every check green. test_prd262_production_last_stage.py parses the FROM lines and checks the order: - orchestrator/Dockerfile, frontend/Dockerfile and services/media-render/Dockerfile, which the PRD names, end with the stage named production. So do the workspace worker and agent-opt-worker, which Railway also builds and which became multi-stage in US-006 and US-007. - That list must equal the in-repo services of infrastructure/railway-manifest.json (root_dir plus railway.json's dockerfilePath), so a new Railway service has to join it. - Any other tracked Dockerfile with a production stage keeps it last (FR-8). Today that adds the root Dockerfile.backend until US-018 deletes it; .oss-scanner/Dockerfile has no production stage and is not built by Railway. - railway.json carries no dockerfileTarget, and orchestrator/.railway-watch.json stays gone. - The checker fails on fixtures with a named stage after production, an unnamed stage after it, and no production stage. It refuses a FROM line it cannot read rather than skip it. It reads --platform flags, lower-case 'as', continuation lines and comments the way BuildKit does. The fixtures are inline strings, not files, so no fixture Dockerfile is picked up by an image scanner or a dependency updater. No new dependency. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… watchfiles out of production Breakdown rows #11 (~46.9 MB) and #47 (watchfiles, ~1.3 MB). #11: graphifyy hard-requires all 25 tree-sitter grammars; only graphify/extract.py imports them, and the app never imports graphify.extract (it parses code with the 13 grammars requirements.in names). Taking the 12 extra grammars out of the lock itself would break every CI install: test.yml, nl2sql-eval-scheduled.yml, .oss-scanner and start.sh run pip install --require-hashes without --no-deps, and pip refuses a requirement of graphifyy the lock does not pin. So the lock keeps them and orchestrator/scripts/split_lock.py (the breakdown's first option) leaves its new LEFT_OUT set out of both halves pybuild installs with --no-deps; it refuses a LEFT_OUT name the lock no longer pins. #47 watchfiles: only uvicorn --reload uses it. Dropping it from the lock would leave the development stage on StatReload, which rglobs and stats every .py under the mounted /app four times a second, a real cost on Docker Desktop bind mounts. Instead split_lock.py's DEVELOPMENT_ONLY goes to a third part, pybuild installs it into /install/development (stripped and compiled like the halves), and only the development stage copies it, after the RUN it shares with production. make dev keeps WatchFilesReload; the production image has no watchfiles. Proof: images.yml (both arches) and test.yml's prod-image-parity run image_dists_absent.py inside the built image with LEFT_OUT and DEVELOPMENT_ONLY. test_prd262_left_out_packages holds LEFT_OUT equal to the grammars the lock pins for graphifyy alone, scans orchestrator/ for any import of graphify.extract (graphify's lazy attribute included), a left-out grammar or watchfiles, and follows every import of the installed graphify from the modules the app imports. Guards that read the old lines were updated: the install loop is now 'for part in stable rest development', the compile loop covers /install/development, split_lock.main takes four paths, and the development stage's COPY list includes the part. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… data in the bytecode layer Breakdown row #47, API part (~2.65 MB). The production stage's stdlib compileall now excludes idlelib, tkinter, turtledemo, lib2to3 and pydoc_data as well as site-packages. tkinter cannot import in python:slim (no Tk libraries), nothing in the API runs IDLE, turtledemo or 2to3, and pydoc's help() compiles pydoc_data in memory if anyone ever calls it. Only .pyc files go; every module stays importable. test_prd262_test_tools_leave_the_image reads the new line and checks the -x pattern against sample paths the way compileall searches it: the five directories and site-packages are skipped, json, asyncio and pydoc itself are still compiled. images.yml's stdlib bytecode layer check is unchanged (stdlib .pyc only, nothing under site-packages). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…RW fonts Breakdown rows #13 (~41.1 MB) and #35 (~12.5 MB), both in the base stage's apt RUN. #13: tesseract-ocr depends on libicu76 for its 15 training tools; the breakdown's scan of every ELF file found only four of them linking ICU, and neither /usr/bin/tesseract (which pytesseract runs) nor libtesseract.so.5 does. The RUN deletes the files dpkg -L libicu76 lists as libicu*.so.* (an empty match fails the RUN) and the 15 tools, the way it already deletes osd.traineddata and git's server binaries, so dpkg keeps its records and apt sees no broken Depends. Then ldconfig, and the build itself fails if ldd /usr/bin/tesseract finds anything missing or tesseract --list-langs fails. Debian builds tesseract the same way on amd64 and arm64. #35: WeasyPrint draws through Pango >= 1.44, which can't use Type1, and fontconfig's metric aliases give Helvetica, Times, Courier, Palatino and Bookman the package's OpenType faces. The RUN deletes /usr/share/fonts/type1/urw-base35 and /usr/share/fonts/X11/Type1 and rebuilds fontconfig's cache with fc-cache -f (fontconfig comes with libpango). Proof: scripts/ci/image_system_tools.py gains an ICU check (libraries and tools gone, ldd of tesseract complete, and ldd of every ELF file in the image that names ICU finds it) and a fonts check (no Type1 under /usr/share/fonts, fc-list lists every OpenType URW face, fc-match puts all five families on one). It still OCRs its fixture through pytesseract. test.yml's prod-image-parity already runs it; images.yml now runs it on every API build, on both arches. test_prd262_system_packages holds the Dockerfile's tool list equal to the script's and checks the order: deletions, ldconfig, ldd, list-langs; fonts, then fc-cache. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… leaves the image Breakdown row #20 (19 Oct, ~22.9 MB on arm64: grpcio, portalocker, plus h2/hpack/hyperframe and qdrant-client itself). The call sites are live, not legacy: the PRD-108 mission field (modules/context/adapters/vector_field.py), PRD-187 durable memory (modules/memory/durable_store.py), the PRD-197 snapshot job (services/qdrant_snapshots.py) and the orphan sweep script all talk to Qdrant whenever QDRANT_URL is set (SaaS). qdrant-client served them over REST and brought grpcio for a gRPC transport the code never used. core/qdrant is a thin async client on the httpx the image already ships: the 15 methods the code calls (collection_exists, get_collections, create/delete_collection, create_payload_index, upsert, delete, set_payload, retrieve, scroll, count, query_points, create/list/delete_snapshot), with qdrant-client's argument names, defaults (wait=true on writes, with_payload=True, limit 10) and REST bodies, the api-key header, and the same model names and fields (Filter.must/should, FieldCondition.key/match, MatchValue.value, MatchAny.any, PointStruct, VectorParams, ...), so the callers change only their imports. Errors keep qdrant-client's shape (UnexpectedResponse with status_code; ResponseHandlingException with source), so core/qdrant_writes.py's F103 timeout retry reads them unchanged. Answers are parsed into frozen dataclasses and a shape Qdrant never sends is refused at the boundary. This replaces a library client for our own datastore; it adds no third-party integration. Lock: qdrant-client leaves requirements.in; requirements.lock is recompiled with its header's flags (only removals: qdrant-client, grpcio, portalocker, pywin32, h2, hpack, hyperframe). grpcio leaves split_lock.py's stable set. qdrant-client moves to requirements-dev.txt (pinned at the 1.19.1 the lock had) because tests/test_prd108_proof.py and test_prd108_scenarios.py search with its in-memory engine in CI; requirements-dev.lock is recompiled against requirements.lock. test.yml's 'no test tools in the production image' step already checks every requirements-dev.txt name is absent from the built image. The orphan sweep's 86-line main was split into its two passes (same behaviour; failures now logged with logger.exception) so the touched code meets the shape limits. Tests: test_prd262_qdrant_rest_client drives every method through httpx.MockTransport (method, path, wait, body, api-key, parsed answer), the 408/400/ReadTimeout/ConnectError mapping through is_timeout, an F103 retry over the client, payload encoding, malformed answers, every method the callers call, and the lock and dev-lock contents. test_prd262_orphan_field_sweep covers the refactored script. Existing memory tests now patch QdrantRestClient and import core.qdrant's errors. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…ion through _matfuncs_sqrtm, as scipy does scipy 1.17's linalg/_matfuncs_sqrtm_triu.so takes SqrtmError from _matfuncs_sqrtm at init, and _matfuncs_sqrtm is the module that imports it (for logm). Imported first on its own it always fails with a circular ImportError, in the pre-cut 973 MB image as in the cut one, so scripts/ci/image_site_packages.py failed on every API image since the lock pinned scipy 1.17.1 (US-022). The probe now imports, before an extension, its parent and the module LOADED_THROUGH names for it; the extension must still load. Found by the first local build of the api lane (breakdown row #25's .symtab check runs in the same script). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
Brings the PRD-262 API work (W2-W5 and the 19 Oct API image cuts) up to origin/main dd82d53: W1+W6 as one squash (#1166), W8 (#1182), PRD-263 harness PRs, PRD-261 W-A2, Auto defaults to Haiku, the vendored fonts and dependabot bumps #1199/#1201/#1202. Method: W1 reaches both sides as different commits (74ed2ff here, inside the #1166 squash on main), so each of the 51 conflicted files was re-merged 3-way with W1's last commit 74ed2ff as the base (git merge-file --diff3: ours = this branch, theirs = origin/main). W1 then cancels out and only real differences remain. Every non-conflicted file git merged was checked against the same W1-based merge: identical in all 708. Clean through the W1 base, main's side taken (this branch has not touched them since W1): action-pins, browser-e2e, check-shopify-isolation, checkout-line-endings, codeql, dco, gitleaks, import-linter, malware-scan, media-render, scan-allowlist workflows; docs/workspace-execution/_index.md; test_prd262_node24_images.py; test_prd262_vendored_metrics.py; services/media-render/{Dockerfile,README.md,tests/test_image.py}; services/workspace-worker/Dockerfile. The worker and media lanes are main's W6/W8 exactly; this branch adds nothing there. Clean through the W1 base, this branch's side taken (main's copy is W1's): docs/getting-started/self-hosting.md, frontend/Dockerfile, orchestrator/Dockerfile, test_prd234_s1a_compose_bind.py, test_prd262_web_stack.py, test_prd262_weekly_rebuild.py, test_prd262_worker_restart.py, scripts/ci/smoke_images.py, scripts/ci/smoke_worker_restart.py. Dockerfile.backend (US-018) and orchestrator/requirements.txt (US-022) stay deleted: main's copies are W1's. Clean through the W1 base, both sides combined: QUICKSTART.md, docker-compose.yml (main's MinIO digest and media-render urllib health check), docs/deployment-infrastructure/docker-containerization.md, nl2sql-eval-scheduled.yml, test_prd251w1_media_render_image.py. Resolved by hand, keeping both sides: - .github/dependabot.yml: one lock-files comment naming the worker's locks (US-044) and the API's locks (US-022). - .github/image-budgets.json: `about` describes on_disk_max_mb and the `updates` bars; media-render's budgets and the api code-only bar both kept. - images.yml: both IMAGES comments; main's On-disk size step, then the API's renamed "Size and update budgets" step; the tree-<12> tag step ends the merge job and main's media-assets job follows it. - smoke-fresh-clone.yml, smoke-images.yml: path filters are the union; the smoke-images header lists US-035, US-038 and US-045's bars. - test.yml prod-image-parity: the API's backend steps, then main's W6 worker step ("the headless shell only, every library resolved, no Mesa"). - .oss-scanner/Dockerfile: the API's hashed lock install, and the worker's requirements.in (main deleted its requirements.txt). - CONTRIBUTING.md: the API's smoke rows, with main's images row. - test_prd143_full_surface_positive.py: neither import (the API dropped the camelot shim's importlib, main dropped the harness half's json). - test_prd262_dependabot.py: both sets of lock reasons. - test_prd262_image_matrix.py: both docstring bullets. - scripts/ci/image_matrix.py: entry() carries the API's SOURCE_DATE_EPOCH and rewrite_timestamp and main's cache_from/cache_to (cache_mode goes, as on main). - scripts/ci/image_layers.py: docstring lists on_disk_verdict, update_verdict and layer_verdict. - scripts/ci/image_report.py: five commands; budget judges the size budget, the on-disk bar and the update bar, with each side's titles and labels. - scripts/ci/smoke-fresh-clone.sh: main's worker health check (US-045) runs after check_images in the API's flow; variables, header and exit lines merged. Alembic: main's head auto_haiku_default_20261019 and this branch's prd262_boot_columns both chained onto prd263_harness_runs. prd262_boot_columns now chains onto auto_haiku_default_20261019 (down_revision and docstring), and the pins follow: test_prd209_alembic_single_head.py (history comment, EXPECTED_HEAD = prd262_boot_columns), test_prd236_w1_routes.py (both chain asserts and the guard) and test_prd262_boot_columns.py (PARENT). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…edia-render step order names the renamed budget step Two failures the merge of origin/main brought together: - main's Auto-defaults-to-Haiku change (683e61e) has seed_auto_agent import core.llm.auto_model_sync and core.llm.new_agent_model, which US-031's seed fingerprint did not hash, so a change to either would not rerun the seeds. test_prd262_seed_fingerprint.py's import walk failed on both; they join INPUT_FILES. - images.yml's budget step is "Size and update budgets" since US-016 (it judges the update bar too). W8's test_prd262_media_render_images.py checked the step order by the old name "Size budget"; it reads the new one. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
test_the_base_stage_fetches_the_encodings_into_the_cache_dir looked for `COPY --from=pybuild /install /usr/local`, a line the base stage lost in US-022, when pybuild began installing into two layers, /install/stable and /install/rest, and base copies both. The test failed on every run since. It now finds every `COPY --from=pybuild /install...` line in the base stage, requires the stable and the rest layer both to be there, and keeps the order it guarded: every install layer is copied before ENV TIKTOKEN_CACHE_DIR, which comes before the RUN that fetches the encodings. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
test_script_times_boots_from_the_container_start_not_its_own_sleeps forbade every `sleep` in scripts/ci/smoke-fresh-clone.sh. US-034 (25c1db5) added one, in check_ready_after_core: `sleep "$left"`, where $left comes from `smoke_ready.py --window-left`, the seconds until the last backend boot's first 60 s (counted from the container's StartedAt) have passed. It waits for those log lines to exist before the APScheduler check reads them; it times no boot, which smoke_boot.py still does from the container start. The test failed on every run since. The test now allows exactly that sleep line, requires it to sit in check_ready_after_core with its length read from --window-left, and still fails on any other sleep in the script. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
test_a_renamed_large_file_counts_as_newly_tracked_at_its_new_path ran
`git mv moves-large.bin moved/large.bin` in a throwaway repository with no moved/
directory. git mv does not create the destination's parent, so the call failed
("renaming 'moves-large.bin' failed: No such file or directory") on every
platform and the rename case was never checked.
The test now makes moved/ before the git mv, so it checks what it names: a large
file renamed to a new path counts as newly tracked there.
Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
#1201 bumped actions/checkout, actions/setup-python and actions/upload-artifact across main's workflows. Seven references this branch added were written before that bump and still carried the old SHAs, so one workflow file mixed two versions of the same action: - test.yml reference-goldens: actions/checkout 11d5960a (v4.4.0) and actions/setup-python a26af69b (v5.6.0); - actions/upload-artifact 043fb46d (v7.0.1) in test.yml (reference-goldens), bytecode-experiment.yml, smoke-fresh-clone.yml (twice) and smoke-images.yml. Each now uses main's pin for the same action: checkout 3d3c42e5 # v7.0.1, setup-python 5fda3b95 # v7.0.0, upload-artifact cf430e03 # v7.0.2. No other `uses:` under .github/ differs from main's pin for its action. scripts/ci/check_action_pins.py passes (127 references pinned). Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
Brings main from dd82d53 up to 111b430: PRD-261 W-A4P (#1215), W-A4 (#1217) and W-C2 (#1218); PRD-264 EUAI-05/06a (#1209), EUAI-11a/11b (#1207) and EUAI-02a/03/04 (#1210); the PRD-262 media-render cuts (#1219, e7b7d88). No conflicts. Four files changed on both sides and merged by hunk, each in a different place: orchestrator/config.py (main adds AI_DISCLOSURE_IDLE_HOURS, VOICE_AI_GREETING and VOICE_AI_REMINDER_MINUTES; this branch drops GOTENBERG_URL), orchestrator/reports/config-surface.json (the same names), docs/frontend-architecture/play-theme.md and orchestrator/tests/test_prd251w1_media_render_image.py (main's node-stripped and Chrome .so assertions; this branch's lock-file scan). Main added no Alembic revision, so prd262_boot_columns stays the one head, on auto_haiku_default_20261019. services/media-render/**, media-render.yml, test_prd262_node24_images.py and the media-render cut tests are main's versions unchanged. #1168 (frontend) is not merged on main yet, so it is not in this merge. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
Brings main from 111b430 to 469ca59: the PRD-262 W7 frontend diet and its 19 Oct cuts (#1168, squash), which merged after the previous merge commit. Three conflicts, resolved by hand keeping both sides: - .github/workflows/images.yml, the build job after "Scan policy": both sides added steps at the same spot. The API's in-image steps (stdlib bytecode layer, site-packages, botocore, scipy, system packages, left-out packages, pip layers) come first, then the frontend's "serves with no network" step; each keeps its own `if: matrix.name == ...`. - .github/workflows/test.yml: - PROD_IMAGE_PATHS is the union: the frontend's .npmrc and scripts/check-(standalone-image|layer-reuse).sh with the API's requirements*.(txt|in|lock), split_lock.py, scripts/ci image checks, boot modules, code-graph tests and fixtures, and the Leiden adapter. Its comment names both sides' inputs. - The prod-image-parity header names the API's Leiden clustering (US-027) and the frontend's dummy Clerk key, local build and booted frontends (US-046). - In prod-image-parity, main's four new frontend steps (local build with no network, npm ci layer reuse, layer reuse, build context) go in after the builder's sharp step, where main put them, and the API's backend steps follow them. Main kept the "Backend — graph extras present with no args" step there; this branch replaced that step with the Leiden step in US-027, so it stays replaced. - docs/getting-started/self-hosting.md, first boot: the API's pull-first text, with the frontend's install named `npm ci` as main now has it. Main added no Alembic revision; prd262_boot_columns stays the one head. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…is fingerprinted A range scan of this branch (gitleaks 8.30.1, `--no-merges --first-parent`, as the "Scan history for secrets" lane runs on a pull request) found one generic-api-key match: test_prd262_pandas_leaves.py (US-025, 49448b5) set config.PANDASAI_API_KEY to a made-up placeholder string, only so the PandasAI service counts as enabled. It is not a credential, but the lane would fail on it. The test now sets the short fake key the repo's other fixtures use, so no new blob matches (including the squash commit a merge to main would scan), and .gitleaksignore fingerprints the historical blob, described without quoting it. test_prd262_pandas_leaves.py: 20 passed. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.
Contributor
|
Important Review skippedToo many files! This PR contains 350 files, which is 250 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files. ⚙️ Run configuration
⛔ Files ignored due to path filters: 6
📒 Files selected for processing: 350
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| def test_make_devs_stack_names_no_published_image(): | ||
| services = _compose_config("docker-compose.yml", "docker-compose.dev.yml") | ||
| assert {name: spec.get("image") for name, spec in services.items() | ||
| if str(spec.get("image", "")).startswith("ghcr.io/")} == {} |
| def test_this_repositorys_own_compose_images_are_its_published_and_dev_names(): | ||
| """docker-compose.yml names the three published images, docker-compose.dev.yml their dev builds.""" | ||
| own = {where: image for where, image in _compose_images().items() if FIRST_PARTY.match(image)} | ||
| published = {where for where, image in own.items() if image.startswith("ghcr.io/")} |
…in site-packages images.yml's stdlib bytecode layer check failed on #1226: the compileall layer held usr/local/lib/python3.11/site-packages/_distutils_hack/__pycache__/__init__.cpython-311.pyc. python:3.11-slim ships setuptools with distutils-precedence.pth; at interpreter start that .pth imports _distutils_hack, and the root compileall process wrote its bytecode into site-packages, inside the stdlib layer. (The pip uninstall RUN US-014 removed used to be the first interpreter start and took that write.) python -S skips site, so no .pth runs; compileall needs only the stdlib. Proved in a container of the built API image: after removing that .pyc and the json stdlib .pyc, python -S -m compileall wrote 0 files under site-packages and put the 5 json .pyc back; the same command without -S recreated _distutils_hack/__pycache__. The two tests that read the RUN line follow it. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…e US-031 moved the seeds test_prd230_packages_seed::test_boot_lane_runs_the_packages_seed_create_only and test_prd251w1_socials_package::test_boot_seeds_the_socials_marketplace_after_the_builtin_skills_and_before_the_packages read main.py, but since PRD-262 US-031 the boot leader's seeds run as core/seeds/leader_stage.py's SECTIONS (built-in skills, then the Socials package, then the marketplace packages, create-only), which main.py runs. Both failed on #1226's CI. They now read leader_stage.py: the same imports and calls, the same order (by the SECTIONS entries), and that main.py imports run_leader_seeds. Nothing they guarded is dropped. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
… mount point first The egress-boot lanes (refused, blackholed) and the §9-Q7 bytecode experiment start the backend alone (--no-deps) on a fresh stack. Docker refused every backend container there: error mounting ".../workspaces" to rootfs at "/workspaces/00000000-0000-0000-0000-0000000000c1": ... mkdirat .../merged/workspaces/00000000-...-c1: read-only file system docker-compose.yml's backend mounts workspaces_data at /workspaces read-only (F220, on main) and nests the deliverables folder in it at /workspaces/<DEFAULT_WORKSPACE_ID>. runc must create that mount point inside the volume and cannot on a read-only mount. Reproduced with throwaway volumes and redis:7-alpine: both mounts read-only on a fresh volume exit 125 with the same error; after a container with the worker's shape (volume writable, the same nested bind) has run once, the read-only pair starts. Self-hosters do not hit it. make up / docker compose up start the worker first (backend depends_on workspace-worker: service_started; the fresh-clone lane's log shows the worker Started before the backend Starting), and the worker's writable mount creates the mount point. The core stack (docker-compose.core.yml) mounts the volume writable on the backend itself. Only a backend started with --no-deps on a never-used volume meets it, and --no-deps skips every depends_on, so a one-shot init service in compose would not reach these lanes either. So the lanes create it themselves: before any backend container, a one-off backend under docker-compose.core.yml (`compose -f docker-compose.core.yml run --rm -T --no-deps --entrypoint true backend`) creates the mount point with a writable mount and exits. The boots keep docker-compose.yml's read-only mounts, and the volume stays compose's, so `down -v` still removes it. Tests: the egress smoke's order (datastores, mount point, probes, boots) on the fake docker in both ways, the base and core mounts the step relies on, and the bytecode script's step before its first boot. The five new or changed tests fail on the scripts as they were. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
…need no embedding provider
The fresh-clone and smoke-images lanes failed the US-021 step:
POST /api/missions/upload answered 500: {"detail":"File upload failed"}
with, in the backend's log, "Uploaded document to S3: s3://automatos-ai/
workspaces/<c1>/documents/1_prd262-us021-smoke.md", then "No embedding provider
configured/reachable; refusing to return random vectors." and "Error uploading
document prd262-us021-smoke.md".
The mission upload runs DocumentManager.upload_document: _record_upload stores
the source in object storage first, then _process_document chunks and embeds
it. Since ef92387 (PRD-185 S3, 4 Jul, on main) the disabled embedding
provider raises EmbeddingUnavailableError instead of returning random
vectors; the document is marked failed, the error is re-raised and the route
answers 500. CI has no embedding provider and must call no paid API, so that
route can never pass there, and US-021 is about object storage through the
pruned botocore, not retrieval.
The smoke now goes through the chat's attachments, which store a file and
nothing else (modules/attachments imports nothing of modules.rag or core.llm):
1. POST /api/attachments: AttachmentStore.put creates the bucket on first use
and puts the bytes (ensure_bucket + put_object); the answer's size_bytes
must be the size sent;
2. GET /api/attachments/<id>: the backend lists and HEADs the object and
answers the same size;
3. docker exec in the backend's container runs AttachmentStore.sign_url, the
app's presign for an attachment against S3_PUBLIC_ENDPOINT_URL (no route
answers that link to a client), printed after a marker so no log line is
read as the link;
4. unchanged: the link must be SigV4 on the origin compose publishes MinIO on,
and the host's GET must return the bytes uploaded.
smoke-fresh-clone.sh passes --container "$BACKEND_CONTAINER" and --workspace
"$DEFAULT_WORKSPACE_ID". Product behaviour is unchanged.
Checked against a throwaway MinIO (cgr.dev/chainguard/minio, port 19000) from
the API venv: the real attachments router (permission dependencies
overridden) answered POST 200 and GET 200 with size_bytes 275, and
AttachmentStore.put/get plus SIGN_SNIPPET run as `python -c` in orchestrator/
gave a SigV4 link on the published origin whose GET returned the 276 bytes
put. That venv has the full botocore; the pruned one is what CI exercises.
Tests: the stand-in round trip and each way it breaks (refused upload, wrong
size, object not found, other origin, SigV2, other bytes, nothing
listening), the exec command and each way the sign fails, the snippet against
what the store defines, the routes in the manifest, the no-RAG imports, and
the smoke's wiring to the backend's container and the local workspace. The
suite does not collect against the old tool.
Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
Owner
Author
|
CI state at ef68909, before merge. Every required check passes. All of these are green: test, images (api amd64 and arm64, with every in-image check), smoke-fresh-clone, bytecode-experiment, chart, browser-e2e and large-files. Two non-required checks are red, both understood:
Fixed on this PR since it opened:
|
AutomatosAI
pushed a commit
that referenced
this pull request
Oct 20, 2026
Brings in #1219 media-render cuts, #1168 W7 frontend, #1222, #1226 (PRD-262 API W2-W5 and cuts), the PRD-261 theme waves and #1232. Three conflicts: - .github/workflows/images.yml: main's in-image steps after "Scan policy" (the API's stdlib bytecode layer, site-packages, botocore, scipy, system packages, left-out packages and pip layers, then the frontend's "serves with no network") and this branch's two worker steps (agent toolchain, site-packages and psycopg2) all kept, main's first, each under its own `if: matrix.name == ...`. The worker's site-packages step still follows its toolchain step, the one its "Same conditions as the step above" names. - orchestrator/tests/test_prd262_dependabot.py: both sides' lock entries kept, this branch's services/workspace-worker/requirements-build.lock and main's orchestrator/requirements*.lock and scripts/ci/reference/*.lock. - orchestrator/tests/test_prd262_node24_images.py: the code lines were the same on both sides (NODE_COPIES, NODE_LAYER, _node_stages). The module docstring and the NODE_LAYER comment take this branch's wording, which names both lanes' tests (test_prd262_worker_diet.py for the worker's #21, test_prd262_media_render_node_strip.py for media-render's #22); main's frontend #24 assertions and its NODE_COPIES comment are kept. test.yml, charts/automatos/templates/worker.yaml and orchestrator/core/workspace_client.py merged without conflict. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com>
This was referenced Oct 20, 2026
AutomatosAI
added a commit
that referenced
this pull request
Oct 24, 2026
… when uid 1000 cannot (#1314) * fix(prd-262): the worker writes your deliverables folder as its owner when uid 1000 cannot smoke-images' core-stack step (#1226, run of 20 Oct 03:04) got 500 from GET /api/workspaces/<c1>/files once the worker started on the core stack. The worker logged PermissionError: [Errno 13] Permission denied: '/workspaces/00000000-0000-0000-0000-0000000000c1/repos' from open_workspace -> ensure_workspace_exists. Root cause: not the core stack's start order. /workspaces/<c1> is the bind-mounted deliverables folder, created by the smoke (and by make up's mkdir) as the runner's user, uid 1001, mode 755. The worker runs as uid 1000 and cannot create repos/ there. The entrypoint's ownership block fires only when the volume root is not uid 1000, and a fresh workspaces_data volume is uid 1000 whichever service mounts it first (both images' /workspaces), so nothing makes the folder writable, by design: US-040/US-045 forbid re-owning the host folder. Reproduced with docker run --rm on the worker image (edge, the same entrypoint as this branch) and a 1001:1001 folder nested in a fresh volume: PermissionError on repos/ in BOTH orders, worker first (the default make up) and backend first (CORE=1). The default stack never showed it because no CI step listed the local workspace through the worker on Linux; Docker Desktop on macOS lets uid 1000 write a host bind (checked: mkdir as uid 1000 succeeds, files land as the Mac user), so nights never hit it. A self-hoster hits it on any Linux host whose user is not uid 1000 (a second user, a CI runner), with make up or make up CORE=1. Fix, in the worker's entrypoint, which already decides who the worker runs as and fixes ownership for it: - compose passes LOCAL_WORKSPACE_ROOT, the deliverables bind mount's target; - when uid 1000 cannot write it (asked with setpriv as the worker, so Docker Desktop's binds, which show root:root but are writable, change nothing), the worker takes the folder owner's uid and gid (groupmod -o, usermod -o, which also re-owns /home/worker). What agents write there is the user's, as on macOS; - a root-owned folder keeps uid 1000 and logs what to do: the worker never runs as root; - the volume walk compares with the worker's uid and prunes LOCAL_WORKSPACE_ROOT as it prunes */projects, so it never re-owns host files (before, a root-owned volume re-owned the whole deliverables folder). Railway and the chart set no LOCAL_WORKSPACE_ROOT: their walk is unchanged. Checked in containers (edge worker image + this entrypoint): host folder 1001:1001, worker first and backend first: the worker runs as 1001, ensure_workspace_exists succeeds for the local and another workspace, a restart and a recreate keep it, the probe stays 1001:1001 and a uid 1000 file in the folder stays 1000; folder 1000: no change; root-owned: uid 1000 and the log line; no LOCAL_WORKSPACE_ROOT with a root-owned volume: re-owned to 1000 as before; owners 1001:1000, 1001:100, 33:33, 65534:65534 all switch. test.yml's parity step, run the same way, passes all checks, the new uid 1001 case included. Tests: test_prd262_worker_boot.py drives the entrypoint with stand-ins (writable folder, another user's, root's; the walk's new prune) and checks compose's LOCAL_WORKSPACE_ROOT against the bind mount's target; 6 of its tests fail on the old entrypoint and compose. prod-image-parity gains the uid 1001 case in the real image. smoke-core-stack.sh also asserts that the reports/ folder the worker makes belongs to the deliverables folder's owner. self-hosting.md's Linux ownership note and the QUICKSTART line say what happens now. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com> (cherry picked from commit d0faad9) * test(prd-262): read an image's registry from its parsed host, not a prefix CodeQL flagged two test checks as py/incomplete-url-substring-sanitization (alerts 307 and 308 on refs/pull/1226/merge): image.startswith("ghcr.io/") in test_compose_published_images.py:214 and test_prd262_dependabot.py:252. They are assertions, not sanitizers, but the check now compares the registry exactly: urlparse(f"//{image}").netloc == "ghcr.io", the host before the reference's first slash. Not weakened: on eleven references (the published ghcr.io names with a tag, a ${AUTOMATOS_IMAGE_TAG:-edge} or a digest; the :dev names; redis, pgvector, Chainguard MinIO; ghcr.io.evil/x and evil.com/ghcr.io/x; the empty string) the new check and the old prefix agree. test_prd262_dependabot.py passes locally; the docker-compose-config test in test_compose_published_images.py skips without the docker CLI and runs in CI. Signed-off-by: Gerard Kavanagh <gerard161@gmail.com> (cherry picked from commit 18b9105) --------- Signed-off-by: Gerard Kavanagh <gerard161@gmail.com> Co-authored-by: Gerard Kavanagh <gerard161@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
PRD-262 (Container diet), the API side. W0 (#1145), W1 and W6 (#1166), W7 (#1168), W8 (#1182) and the media cuts (#1219) are already on main. This PR is what's left for the API:
Branch
feat/prd-262-api. The waves and the cuts were built onfeat/prd-262-api-integrationand come in through its merge commits. The branch has been merged with main four times (see Merges with main).Measured on a local arm64 build, installed size: the API image was 1,920 MB before PRD-262, 973 MB after W0–W5 (972.6 MB measured) and 789.2 MB with the cuts (
docker image inspect .Size). That is 266.1 MB gzip (docker save | gzip -1). The cuts saved 183.3 MB. The breakdown estimated about 186 MB, and #17 and #25 overlap by a few MB.W2: API clean-up, no behaviour change (US-014 to US-022)
requirements-dev.txtis installed only by CI.chown -Rlayer.orchestrator/.dockerignore).Dockerfile.backendandorchestrator/requirements.txtare gone.requirements.incompiles torequirements.lockwith hashes, and there is arequirements-dev.lock.scripts/split_lock.pyinstalls the lock as two layers, the stable set and the rest.W3: API code swaps (US-023 to US-028)
[leiden]extra and graspologic's heavy closure are gone.W4: API boot (US-029 to US-034)
ALTER TABLE … ADD COLUMN IF NOT EXISTSstatementsmain.pyran on every boot become the revisionprd262_boot_columns.TIKTOKEN_CACHE_DIR.W5: pull-first default stack (US-035 to US-042)
make dev.make uppulls the published images.make buildbuilds from the checkout.make upwrites the three secrets into the user's.envand refuses placeholders.start_interval: 1s, on a stated floor of Docker Engine 25.make cleankeeps a week of build cache.The 19 Oct cuts
Row numbers come from the 19 Oct ranked size breakdown. The MB are the breakdown's arm64 estimates.
split_lock.py'sLEFT_OUT. The lock still pins them, because CI'spip install --require-hashesneeds every graphifyy requirement pinned.split_lock.pyrefuses aLEFT_OUTname the lock no longer pins/usr/bin/tesseractdoesn't link ICU. The build fails ifldd /usr/bin/tesseractfinds anything missing ortesseract --list-langsfailscore/math/probability_theory.pynow takes the t quantile fromscipy.special.stdtrit, which is exactly whatstats.t.ppfreturnscore/qdrant, an httpx REST client covering the 15 methods the code calls, with qdrant-client's names, defaults, models and error classes. grpcio, portalocker, h2, hpack and hyperframe leave the lock. qdrant-client stays inrequirements-dev.txtfor the PRD-108 tests' in-memory enginestrip --strip-unneededreplaces--strip-debugon site-packages' compiled libraries..dynsymstaysmake devkeeps it, production has none (1.3 MB)There is also one fix, 74d0899.
scripts/ci/image_site_packages.pynow loads scipy's_matfuncs_sqrtm_triuthrough_matfuncs_sqrtm, as scipy does. Imported on its own, that extension always fails with a circular ImportError, and it did so in the pre-cut 973 MB image too. Without this fix, the site-packages step fails on every API image since the scipy 1.17.1 pin.Proven on the built image
These checks ran on a local arm64 build on 19 Oct (image e12bfdd3fd1f), from the local integration branch at ac83f96b7. Since that build, this branch has merged main. The following are byte-identical to what was built:
orchestrator/Dockerfile,requirements.in,requirements.lock,scripts/split_lock.py,orchestrator/.dockerignore,core/qdrant/and the in-image check scriptsimage_site_packages.py,image_scipy.pyandimage_system_tools.py. Other orchestrator code has main's changes since (PRD-261, PRD-263 and PRD-264, and the Haiku default).Health and imports.
/healthand/health/readyboth answered 200.FR-6.
import maintook 4.3 s. Afterwards none of camelot, cv2, matplotlib, pandas, nltk, google.generativeai or googleapiclient was loaded. Nor was any pruned scipy subpackage, graphify.extract, qdrant_client, grpc, watchfiles or a left-out grammar.Tesseract OCR (Testing the PR with Automatos workflows #13).
image_system_tools.pyreported ok for each of its checks: packages, tesseract, OCR fixture, git, ICU and fonts.ldd tesseractfinds nothing missing.Fonts (Pushing updates to load into railway #35).
fc-listlists 10 URW families from 35 OTF files, and no Type1 file is left.fc-matchresolves Helvetica, Times, Courier, Palatino and Bookman to the OTF faces.scipy (Credential management #17).
image_scipy.pyreported ok for its 7 checks: subpackages, datasketch, graphify dedup, the code graph's pagerank, the Knowledge Graph (Leiden), Student's t, and that no pruned subpackage was ever loaded.Qdrant over httpx (Merge pull request #18 from AutomatosAI/credential-management #20). All 15 client methods ran against the local Qdrant, on a scratch collection called
prd262_probe:Afterwards the collection list equalled the one before. A count on the deleted collection raises
UnexpectedResponse404. The existing collections were only read. The orphan sweep in--dry-runreported 0 orphans.site-packages (PRD-30 modular architecture complete - 11 modules, 3 layers, 263 files #25, Claude/tool router registry z85 pz #37).
image_site_packages.py, after 74d0899, checked 173 files: 172 extensions imported and 1 library opened with ctypes. No file has debug sections or.symtab, notests/directory or C or Cython source is left, andimport mainfinds no stale bytecode.Other in-image checks.
image_botocore.py,image_app_bytecode.py(951 .pyc) andimage_dists_absent.py(the left-out grammars, watchfiles, qdrant-client and its transport, the FR-6 distributions, the dev tools) all exited 0.Warm recreate. The backend was healthy 10.6 s after
docker compose up --force-recreatestarted, 6.2 s after the container started. Before this wave it took 19 s.Alembic
prd262_boot_columnschains ontoauto_haiku_default_20261019, main's head. Main has added no revision since, so there is exactly one head.test_prd209_alembic_single_head.py(EXPECTED_HEADand its dated history comment),test_prd236_w1_routes.py(the chain asserts) andtest_prd262_boot_columns.py(PARENT).ADD COLUMN IF NOT EXISTS, so it does nothing on a database the oldmain.pybooted on. The downgrade leaves both columns.prd262_boot_columns(from a local build where the revision chained ontoprd263_harness_runs) needs:alembic stamp prd263_harness_runs && alembic upgrade heads. That runs main's auto-Haiku data migration, then the idempotent boot-columns migration. Production never hadprd262_boot_columns, so it needs nothing.Fixes made while preparing this PR
These three tests failed on CI before this PR, and none of the failures came from a merge. Each failed locally before its fix and passes after it.
test_prd262_offline_boot.py::test_the_base_stage_fetches_the_encodings_into_the_cache_dir. It looked forCOPY --from=pybuild /install /usr/local, which US-022 split into/install/stableand/install/rest. It now requires both layers to be copied into base beforeENV TIKTOKEN_CACHE_DIR, which must come before the fetch RUN.test_prd262_smoke_fresh_clone.py::test_script_times_boots_from_the_container_start_not_its_own_sleeps. US-034 (25c1db5) added one sleep,sleep "$left"incheck_ready_after_core. That sleep waits until the last boot's first 60 s, counted from the container's StartedAt, are in its log. It times no boot. The test now allows exactly that line, requires its length to come fromsmoke_ready.py --window-left, and still forbids any other sleep.test_prd262_large_files.py::test_a_renamed_large_file_counts_as_newly_tracked_at_its_new_path.git mvinto a directory that doesn't exist fails on every platform. The test now createsmoved/first.The action pins and the secret scan:
3d3c42e5(v7.0.1), setup-python5fda3b95(v7.0.0) and upload-artifactcf430e03(v7.0.2). The changed lines are intest.yml(3),bytecode-experiment.yml,smoke-fresh-clone.yml(2) andsmoke-images.yml. No otheruses:under.github/differs from main's pin for its action.scripts/ci/check_action_pins.pypasses (127 references).PANDASAI_API_KEYintest_prd262_pandas_leaves.py(US-025), not a credential. The test now uses a short fake key, and.gitleaksignorefingerprints the old blob. The scan now finds no leaks.Merges with main
git merge-file --diff3, using W1's last commit 74ed2ff as the base. W1 then cancels out and only the real differences remain. Every file git merged without a conflict matched the same W1-based merge, all 708 of them. The commit message lists each resolution. Two independent verifiers checked this merge and found it clean.core.llm.auto_model_syncandcore.llm.new_agent_model), and the seed fingerprint now hashes them;images.yml: the API's in-image steps, then the frontend's "serves with no network" step. Each keeps its ownif: matrix.name == ….test.yml:PROD_IMAGE_PATHSis the union of both sides, and its comment names both.self-hosting.md: the API's pull-first text, naming the frontend installnpm ci.After each merge, a script compared line counts 3-way (ours + theirs − base) for every file either side changed. No line main added is missing unless a conflict resolution above changed it, and no line this branch added is missing. A whole-file check, which also covers binary files, found every file one side left alone equal to the other side's version.
Left for after WebSummit (Gerard's decision, 19 Oct)
None of these is in this PR:
Merging with the other open PRD-262 PR
A trial merge (
git merge-tree) of #1220 (worker,feat/prd-262-worker-cutsd121874) with this branch shows three conflicts:images.yml: keep both sides' in-image steps.test_prd262_dependabot.py: keep both sides' lock entries.test_prd262_node24_images.py: this conflict is between perf(prd-262): worker cuts — 1,074 → 910 MB, toolchain kept #1220 and main's later W7, W8 and media changes, not this branch.Test plan
--no-merges;images.ymlforapion amd64 and arm64, including the in-image steps: stdlib bytecode layer, site-packages, botocore, scipy, system packages, left-out packages and pip layers. Also the size report againstedge.test.ymlprod-image-parity: the Railway build's backend steps (Leiden, build context, no test tools, removed packages, left-out packages, site-packages, botocore, scipy, boot bytecode, ownership, system packages, the Helvetica render, and the clone and code-graph tests in the test image), with the frontend and worker steps beside them.smoke-fresh-cloneandsmoke-images:make uppull-first to a working UI;/healthand/health/readyanswer 200; one document is stored and fetched back through the pruned botocore.alembic stamp prd263_harness_runs && alembic upgrade headson the local DB; confirmalembic current=prd262_boot_columns.How it was verified
AUTH_EDITION=local) and hosted. The local edition gets the pull-first stack. Railway builds the production stage with no build args, which prod-image-parity builds and checks.Local checks on this branch's HEAD (no Docker; nothing ran against the local stack):
uvx ruff@0.16.9 check --select F821,F401,F811,F841on the 115 changed.pyfiles found 13 findings. All 13 are in files main changed (api/sites.py,api/webhooks.py,api/widgets/chat.py,channels/,consumers/chatbot/service.py, three tests), and main has the same 13. Neither the merges nor the fixes add one..pyfile compiles.bash -npasses on the changed shell scripts.yaml.safe_loadloads the 6 changed workflows.jqreads the 34 changed JSON files. There are no conflict markers.scripts/ci/check_changed_code_shape.py --base origin/main: 0 errors.requirements.lockandrequirements-dev.lock, ran on everytest_prd262_*.py, the two head-pin tests and every orchestrator test file the merges brought in (127 files). Result: 1 failed, 2,826 passed, 41 skipped, 28 errors. None of these comes from the change; each needs something this machine doesn't have:test_f149_…andtest_f232_…, which need Postgres (CI has it), and they error the same way on main;test_prd262_system_packages.py::test_the_fixture_names_helvetica_and_fills_two_pages_here, which needs WeasyPrint's pango, not installed here; it also fails at the commit before the fixes, and the built image runs the same render;/proc, the docker CLI and node.prd262_boot_columns, whosedown_revisionisauto_haiku_default_20261019.AI assistance
Checklist (AGENTS.md)
git commit -s). Every non-merge commit is; the one old merge commit 8ed2299 is not, and the DCO lane checks with--no-merges.make upgenerates the local secrets into the user's.envat run time; none is committed.orchestrator/config.py; no hardcoded values.TIKTOKEN_CACHE_DIRis read by tiktoken itself (FR-12).V2copies). camelot, pandas, langchain's splitters, qdrant-client,Dockerfile.backendandrequirements.txtare gone, not kept alongside.prd262_boot_columns), chained onto the current head,IF NOT EXISTSso it is safe on acreate_all-built database, and the head pins moved.orchestrator/reports/route-manifest.json. There are no new routes.graspologic-nativeis now listed directly (US-027). It was installed before through graspologic.tree-sitter-<language>packages replace tree-sitter-language-pack (US-028): c, c-sharp, cpp, go, java, kotlin, php, ruby, rust and swift. python, javascript and typescript were already listed.openpyxlandpytzare now listed directly. Before, they came only through camelot and pandas.