Skip to content

Say on the Routines page when nothing is there to fire them - #460

Open
zopeVaibhav wants to merge 1 commit into
CopilotKit:mainfrom
zopeVaibhav:feat/say-when-routines-have-no-worker
Open

Say on the Routines page when nothing is there to fire them#460
zopeVaibhav wants to merge 1 commit into
CopilotKit:mainfrom
zopeVaibhav:feat/say-when-routines-have-no-worker

Conversation

@zopeVaibhav

@zopeVaibhav zopeVaibhav commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What this changes

A routine needs a second process to fire it, and a deployment that never started one looked exactly
like a deployment that had. Every sweep now records that it happened, and the Routines page reads
that record: somebody with standing routines and nothing sweeping is told so, instead of being shown
a page that looks correct.

  • routine_sweeps, one row keyed "routines", holding swept_at and the owner that last swept
    (server/drizzle/0029_routine_sweeps.sql, schema at server/src/db/schema/coworker.ts:133-137).
  • routineStore.recordSweep / lastSweptAt (server/src/routines/store.ts:797-814), an upsert on
    the primary key so the table stays exactly one row however many passes run.
  • offerDueRoutines records the pass before it reads what is due
    (server/src/routines/sweep.ts:142), so the Kubernetes CronJob and the laptop loop both report
    through the one funnel they already share.
  • GET /api/routines gains sweep { lastSweptAt, working } (server/src/routines/routes.ts:39-47).
  • The page draws a warning when routines stand and nothing is sweeping, naming when the last pass
    was or saying none ever happened (app/src/components/routines/routines-list.tsx:151).
  • docs/routines.md and docs/configuration.md asserted the old behaviour and now describe this one.

Why the window is fifteen minutes, and fixed

working means a sweep inside SWEEP_SILENCE_MS, which is MINIMUM_INTERVAL_MS — the floor a
routine's own schedule already has (server/src/routines/schedule.ts:4), so a longer silence is one
no routine could have wanted.

It is a constant rather than something derived from the deployment's own cadence, and the chart is
why that holds: routines.schedule defaults to */5 * * * * (charts/openbot/values.yaml:268),
chosen, in that file's own words, to sit "inside the 15-minute floor the tools enforce, so a firing
waits at most one tick". A deployment on the defaults sweeps three times inside the window. Setting a
schedule looser than fifteen minutes would make this page read as quiet between runs — but such a
schedule is already outside what the floor is built for, and the docs now say so.

Deriving the window instead (from observed sweep intervals, or from whether any routine is actually
overdue) would remove that edge, at the cost of history to read and of detection latency that follows
each routine's own cadence — a weekly routine would take a week to notice a dead worker. Fifteen
minutes notices in fifteen minutes, whatever is standing.

Where it runs

  • New state that outlives a request? Yes: one row in routine_sweeps, in Postgres. Not a
    module-level Map.
  • What happens on the second replica? Both write the same row, and the last writer wins. That
    is the intended semantic — the question the page asks is "did anything sweep recently", not
    "did this process sweep". owner records which process wrote last, so a stuck deployment still
    traces back to a claimant.
  • Anything serialised? The write is insert ... on conflict (id) do update, a single
    conditional statement on the primary key, not a check-then-write. Concurrent sweepers cannot
    produce two rows or a lost table.
  • Anything fanned out to a browser? No. The page reads the value on its existing routines
    query; no socket, no push.
  • New listener, port, or schedule? None. The heartbeat rides the sweep that already exists; a
    hundred copies of it write one row.

Boundary and audit

  • Every acting call still goes through the gateway: nothing here acts. recordSweep is
    bookkeeping on the dispatch path and takes no decision.
  • New refusals and new failures each write a row: the heartbeat is wrapped and logs
    routine-sweep-heartbeat-failed rather than writing an audit row, deliberately — it is not an
    acting call, and a broken heartbeat must not be able to stop a sweep that would otherwise have
    worked.
  • Nothing new is trusted from the client that the server can resolve itself: working is
    computed server-side from the stored timestamp; the client is handed a boolean, not a rule.

This widens one response rather than a permission: GET /api/routines now tells a signed-in caller
when the deployment last swept. It is visible only to somebody already entitled to the routines on
that page, and it carries a timestamp and a lease name, not a hostname or a secret.

One skew to know about

GET /api/routines changed shape from { routines } to { routines, sweep }, and the app defaults
a missing sweep to { lastSweptAt: null, working: false }. A new app served by an old server would
therefore show the warning when nothing is wrong. The app and the server ship from this repository
together, so there is no supported configuration where that happens — but it is the failure mode to
expect if anyone ever serves them from different builds, and the default was chosen that way on
purpose: a false warning is recoverable, a silently swallowed one is the bug this PR exists to fix.

Changelog

  • A line under Unreleased: "The Routines page says when nothing is there to run them".

Proof

Reproduced against the bug before fixing it. On main, GET /api/routines answers with keys
[ "routines" ] and there is no field a page could read to tell a dead sweeper from a live one; the
same assertion on this branch answers [ "routines", "sweep" ]. The new tests fail on main for the
same reason and pass here.

Driven in a browser against a live deployment, all four states:

  • worker sweeping, row four minutes old — no warning, routines listed normally
  • worker stopped and the row backdated 22 minutes — "Nothing is running these. The routines worker
    last checked 22 minutes ago. Until it is running again, none of these will fire."
  • row deleted, nothing has ever swept — "No routines worker has ever checked in, so none of these
    will fire. A deployment needs one running to carry them out."
  • a worker started again — the warning clears on the next tick, and the row comes back stamped with
    that process's lease name

Gates: typecheck clean across app, server and worker; lint 606 files, no warnings; format 602 files,
no diff. drizzle-kit generate reports no schema changes, so the snapshot and the schema agree.
Migration 0029_routine_sweeps applied to a live database and confirmed by information_schema:
id NOT NULL, swept_at NOT NULL, owner nullable.

Rebased onto 1c7bd92 and re-measured there. The control is written against the surface main
already has, so it fails on the behaviour rather than on a missing export:

expect(Object.keys(body)).toContain("sweep")

Expected to contain: "sweep"
Received: [ "routines" ]

Suite, run in one session against a same-session control, with the two agent-handoff-*.integration
files excluded because they fail most of their tests on bare main and make any comparison
meaningless:

pass skip fail tests files
origin/main @ 1c7bd92 2609 25 8 2642 218
this branch 2620 25 8 2653 220

The failing test names are byte-identical on both sides, as is the @langchain/core/messages
resolution error, so none of them belong to this change. Six of those eight are in
server/tests/routine-sweep.integration.test.ts, which this PR touches — they fail the same six ways
on bare main, checked on its own in this session. The +11 tests are exactly this PR's: 5 in
app/tests/routines-no-worker.test.ts, 4 in server/tests/routine-sweep-liveness.test.ts, and 2
added to server/tests/routine-sweep.integration.test.ts.

One thing worth flagging for whoever merges: this branch adds server/drizzle/0029_routine_sweeps.sql
and PR #466 adds server/drizzle/0029_last_signed_in_at.sql. main tops out at 0028, so each is
valid alone; whichever lands second needs renumbering, journal and snapshot included. Say the word and
I will renumber this one.

Closes #459

@zopeVaibhav
zopeVaibhav force-pushed the feat/say-when-routines-have-no-worker branch 2 times, most recently from 9895a99 to da7fa41 Compare September 9, 2026 17:33
@zopeVaibhav
zopeVaibhav force-pushed the feat/say-when-routines-have-no-worker branch from da7fa41 to 577a3dd Compare September 10, 2026 16:59
@zopeVaibhav
zopeVaibhav requested a review from mxmzb as a code owner September 10, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A deployment with no routines worker is indistinguishable from one running them

1 participant