Skip to content

feat: report judge context overflow as could-not-evaluate, not score-0 - #399

Open
pdebjyot wants to merge 1 commit into
strands-agents:mainfrom
pdebjyot:fix/342-overflow-status
Open

pdebjyot wants to merge 1 commit into
strands-agents:mainfrom
pdebjyot:fix/342-overflow-status

Conversation

@pdebjyot

Copy link
Copy Markdown
Contributor

What & why

Closes the first half of #342. A judge-based evaluator whose prompt overflows
the model's context window currently records score: 0 / test_pass: False
indistinguishable from a genuine zero-quality result. That silently drags
down overall_score and the pass rate, so a run where the judge never actually
read the trace looks like a run where the agent failed.

This makes context overflow a distinct could-not-evaluate outcome that is
excluded from aggregates instead of being counted as a quality failure.

What's in the PR

  • types/evaluation.pyEvaluationStatus enum {evaluated, could_not_evaluate}.
  • experiment.py — a shared _error_result helper tags only
    context-overflow errors (via the existing _is_context_exceeded predicate)
    as could-not-evaluate. Every other exception stays a real
    score: 0 / test_pass: False failure, so a genuine evaluator bug (e.g. a
    KeyError in a custom evaluator) still surfaces rather than being hidden.
    Statuses are threaded through aggregation; could-not-evaluate cases are
    excluded from overall_score and from the diagnosis on-failure trigger.
  • types/evaluation_report.py — new statuses list + mean_score()
    classmethod (excludes could-not-evaluate); flatten() threads it; _display
    renders N/A for an unscored case instead of 0.00. Reports written before
    this field existed deserialize unchanged.
  • display_console.py — could-not-evaluate cases are dropped from both the
    numerator and denominator of the printed Pass Rate.

Behavior

case before after
judge overflows context score: 0, counted as a fail could_not_evaluate, excluded from score + pass rate, shown N/A
genuine evaluator error score: 0, counted as a fail unchanged — still a fail
everything fits unchanged unchanged

Raw per-case scores / test_passes lists are left intact; only the aggregates
and the display change.

Tests / checks

  • Report-level: mean_score exclusion, flatten threading, round-trip
    serialization, legacy reports without the field.
  • Display-level: pass-rate excludes could-not-evaluate; all-unscored is guarded
    against divide-by-zero.
  • Experiment-level: overflow → could-not-evaluate & excluded from score; a
    non-overflow error stays a failure.
  • Full suite 1700 passed; ruff check and mypy -p src clean.

@pdebjyot
pdebjyot requested a review from a team as a code owner September 14, 2026 18:10
@pdebjyot
pdebjyot requested a review from mehtarac September 14, 2026 18:10
@github-actions github-actions Bot added enhancement New feature or request area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics area-core Core eval framework: Case, Experiment, task handler, evaluation data stores labels Sep 14, 2026
… a failure

A judge-based evaluator whose prompt overflows the model context window
recorded score:0/test_pass:False, indistinguishable from a genuine
zero-quality result and silently dragging down overall_score and pass rate
(first half of strands-agents#342).

Reuse the existing NOT_APPLICABLE mechanism instead of a parallel status:
- experiment._run_evaluator: a context-overflow error (gated on the existing
  _is_context_exceeded predicate) returns a single NOT_APPLICABLE EvaluationOutput
  (test_pass=True) so is_applicable/calculate_overall_score drop it from the mean.
  Every other exception stays a real score:0 failure with empty detailed_results.
- display_console: exclude not-applicable cases from the printed Pass Rate,
  mirroring EvaluationReport.is_applicable, so an unscored case can't deflate it.
@pdebjyot
pdebjyot force-pushed the fix/342-overflow-status branch from efab725 to c954fe3 Compare September 15, 2026 03:14
pdebjyot added a commit to pdebjyot/evals that referenced this pull request Sep 17, 2026
- Type disclosure kwarg as DisclosureMode (Literal) across all 18 evaluators
  instead of bare str, so misuse is caught at type-check time.
- Resolve TraceIndex once per case in tool-level and skill evaluators rather
  than re-probing inside the per-tool / per-skill loop.
- Skill probe uses the full inline prompt (body + response), not just the
  trajectory, so the auto threshold reflects real prompt size.
- Skip the inline-render probe for always/never modes (probe is irrelevant).
- Fix README + docstrings: genuine overflow under "never" surfaces as a judge
  context-length error (the could-not-evaluate path is strands-agents#399, not this branch).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant