Skip to content

scheduler: don't resurrect a task deleted mid-run (#276) - #277

Open
CakeCrusher wants to merge 2 commits into
exoharness:mainfrom
CakeCrusher:fix/scheduler-no-resurrect-deleted-task
Open

CakeCrusher wants to merge 2 commits into
exoharness:mainfrom
CakeCrusher:fix/scheduler-no-resurrect-deleted-task

Conversation

@CakeCrusher

@CakeCrusher CakeCrusher commented Sep 8, 2026

Copy link
Copy Markdown

Fixes #276.

A fire runs a model turn, so a task can be deleted while a run holds a copy of its record. run_task then wrote that stale copy back, and because put_task created the file when it was missing, the delete was undone and the task kept firing. delete_scheduled_task and cancel_scheduled_task are model-facing tools, so a delete naturally arrives during a run — that is how this was hit.

create_task is the only creator, so put_task becomes replace-only: a no-op once the task is gone. The existence check and the write happen under an exclusive lock on the tasks directory, taken by delete_task as well, so a delete cannot land between them. The lock is held around the file operations only, never across a run, and the OS releases it if a holder dies (std::fs::File::lock, stable since 1.89 — no new dependency).

Deletion is therefore terminal: an in-flight fire still finishes, and the task is simply never scheduled again.

The whole change is in scheduler_store.rs. No call sites changerun_task is untouched.

Includes a test that reproduces the original failure: a run holds a record, a delete lands mid-run, and the stale write-back must not recreate the task. It fails if put_task is allowed to create.

Related, if useful

The doc comment on claim_due_tasks notes its read-lease-write is unconditional pending "the conditional puts in upstream PR #113". Replace-only put_task also covers that call site.

Worth flagging: #113 as it stands keeps the unconditional put_task at the end of run_task, so this bug survives it — and #113 removes claim_due_tasks entirely. Happy to rebase onto #113 or #232 in whatever order suits.

cargo test -p executor (115 passed), cargo clippy --all-targets -D warnings, cargo fmt, and cargo check --workspace --all-targets all clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QhK4EoQJsbjEaqGR3VJ7g5

Comment thread crates/executor/src/scheduler_runtime.rs Outdated
A fire runs a model turn, so a task can be deleted while a run holds a
copy of its record. run_task then wrote that stale copy back, and because
put_task created the file if it was missing, the delete was undone and
the task kept firing. delete_scheduled_task and cancel_scheduled_task are
model-facing tools, so a delete naturally arrives during a run.

create_task is the only creator, so put_task becomes replace-only: it is
a no-op once the task is gone. The existence check and the write happen
under an exclusive lock on the tasks directory, taken by delete_task as
well, so a delete cannot land between them. The lock is held around the
file operations only, never across a run, and the OS releases it if a
holder dies.

An in-flight fire still finishes; the task is simply never scheduled
again. No call sites change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QhK4EoQJsbjEaqGR3VJ7g5
@CakeCrusher
CakeCrusher force-pushed the fix/scheduler-no-resurrect-deleted-task branch from 269e976 to 30ad950 Compare September 8, 2026 18:52
@CakeCrusher
CakeCrusher marked this pull request as ready for review September 8, 2026 18:52
Comment thread crates/executor/src/scheduler_store.rs Outdated
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Write {
Create,
Replace,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nit is that Replace to me sounds like "if it exists, overwrite it". I spent like 10 minutes trying to understand why it wasn't doing that before reading your comment. I'd prefer names like CreateIfNotExists (although I think Create is actually Replace?) and ReplaceIfExists

@CakeCrusher CakeCrusher Sep 10, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, cleaned up the ambiguioty, also renamed to CreateOrReplace instead which is what that path actually does.

@CakeCrusher
CakeCrusher force-pushed the fix/scheduler-no-resurrect-deleted-task branch 3 times, most recently from 1a40106 to f818265 Compare September 10, 2026 23:26
`Create` did not enforce non-existence and `Replace` did not overwrite an
absent record, so both names described contracts the code does not have.
Say which case each one handles when the file is missing: the write
`create_task` uses is an upsert, and the write `put_task` uses is a no-op
once the task is gone.

Naming only; behavior is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QhK4EoQJsbjEaqGR3VJ7g5
@CakeCrusher
CakeCrusher force-pushed the fix/scheduler-no-resurrect-deleted-task branch from f818265 to 3b338e9 Compare September 10, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduler resurrects a task deleted while it was running

2 participants