Skip to content

feat(timing): persist runner-measured duration - #319

Merged
slowdini merged 1 commit into
devfrom
issue-310-runner-duration
Sep 4, 2026
Merged

slowdini merged 1 commit into
devfrom
issue-310-runner-duration

Conversation

@slowdini

@slowdini slowdini commented Sep 4, 2026

Copy link
Copy Markdown
Owner

What and why

Runner-dispatched Codex runs expose token usage but no native duration, which left all-Codex campaign timing summaries with zero duration samples. The universal runner already owns the subprocess lifecycle, so it can provide one cross-harness measurement without relying on transcript-specific timestamps.

This change measures and persists elapsed eval-agent subprocess time for every supported harness and carries that value into canonical timing.json artifacts.

Fixes #310.

Measurement contract

  • Measure monotonic wall time only while the eval-agent harness subprocess is running.
  • Sum the initial round and every native resume round.
  • Exclude responder consultations, judge calls, queue time, and runner bookkeeping.
  • Retain elapsed time for timed-out conversations; do not write a successful completion artifact after a failed round.

The measurement applies to Codex, Claude Code, Cline, OpenCode, and runner-ready BYOH descriptors without adding a harness-specific capability.

Artifact and compatibility behavior

  • Add duration_ms to runner-owned conversation.json completion artifacts.
  • Carry runner duration into timing.json while keeping canonical timing out of run.json.
  • Replace the ambiguous shared source writer contract with independent token_source and duration_source fields, including the runner duration source.
  • Continue reading historical records that contain only source; reserialization emits explicit per-metric provenance.
  • Preserve native transcript duration as the fallback for historical and externally assembled runs.
  • Leave historical Codex duration missing when no reliable timestamp exists; no runner time is fabricated or backfilled.
  • Preserve an existing timing.json unless the operator requests overwrite behavior.

Before, a Codex timing artifact could not supply a duration sample:

{
  "total_tokens": 1234,
  "duration_ms": null,
  "source": "transcript"
}

Runner-driven ingestion writes independent provenance:

{
  "total_tokens": 1234,
  "duration_ms": 42891,
  "token_source": "transcript",
  "duration_source": "runner"
}

Aggregation checks source consistency independently for tokens and duration, so mixed provenance remains visible without conflating the two metrics.

Review notes

  • The timing wire-compatibility logic lives in a focused core::timing module and remains re-exported through the existing core::types surface.
  • Deterministic fixture delays cover every built-in harness plus BYOH, multi-turn accumulation, responder exclusion, timeout and failure paths, legacy deserialization, incomplete transcripts, overwrite behavior, and mixed-source aggregation.
  • CLI help, shipped conversation and BYOH guides, harness notes, descriptor guidance, and JSON schemas document the boundary and fallback behavior.

Verification

  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test
  • cargo run --quiet -- dispatch --help
  • cargo run --quiet -- record-runs --help
  • cargo run --quiet -- docs conversations
  • cargo run --quiet -- docs byoh

Measure eval-agent subprocess time at the universal runner boundary so every harness records a comparable duration. Preserve independent token and duration provenance while keeping historical timing records readable.
@slowdini
slowdini merged commit f9bc6be into dev Sep 4, 2026
7 checks passed
@slowdini
slowdini deleted the issue-310-runner-duration branch September 4, 2026 04:09
@slowdini slowdini mentioned this pull request Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Persist runner-measured elapsed duration for every harness

1 participant