Conversation
5 tasks
pdebjyot
requested a deployment
to
manual-approval
September 14, 2026 18:21 — with
GitHub Actions
Waiting
pdebjyot
requested a deployment
to
manual-approval
September 14, 2026 18:21 — with
GitHub Actions
Waiting
… a failure A judge-based evaluator whose prompt overflows the model context window recorded score:0/test_pass:False, indistinguishable from a genuine zero-quality result and silently dragging down overall_score and pass rate (first half of strands-agents#342). Reuse the existing NOT_APPLICABLE mechanism instead of a parallel status: - experiment._run_evaluator: a context-overflow error (gated on the existing _is_context_exceeded predicate) returns a single NOT_APPLICABLE EvaluationOutput (test_pass=True) so is_applicable/calculate_overall_score drop it from the mean. Every other exception stays a real score:0 failure with empty detailed_results. - display_console: exclude not-applicable cases from the printed Pass Rate, mirroring EvaluationReport.is_applicable, so an unscored case can't deflate it.
pdebjyot
force-pushed
the
fix/342-overflow-status
branch
from
September 15, 2026 03:14
efab725 to
c954fe3
Compare
pdebjyot
requested a deployment
to
manual-approval
September 15, 2026 03:14 — with
GitHub Actions
Waiting
pdebjyot
requested a deployment
to
manual-approval
September 15, 2026 03:14 — with
GitHub Actions
Waiting
pdebjyot
added a commit
to pdebjyot/evals
that referenced
this pull request
Sep 17, 2026
- Type disclosure kwarg as DisclosureMode (Literal) across all 18 evaluators instead of bare str, so misuse is caught at type-check time. - Resolve TraceIndex once per case in tool-level and skill evaluators rather than re-probing inside the per-tool / per-skill loop. - Skill probe uses the full inline prompt (body + response), not just the trajectory, so the auto threshold reflects real prompt size. - Skip the inline-render probe for always/never modes (probe is irrelevant). - Fix README + docstrings: genuine overflow under "never" surfaces as a judge context-length error (the could-not-evaluate path is strands-agents#399, not this branch).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Closes the first half of #342. A judge-based evaluator whose prompt overflows
the model's context window currently records
score: 0 / test_pass: False—indistinguishable from a genuine zero-quality result. That silently drags
down
overall_scoreand the pass rate, so a run where the judge never actuallyread the trace looks like a run where the agent failed.
This makes context overflow a distinct could-not-evaluate outcome that is
excluded from aggregates instead of being counted as a quality failure.
What's in the PR
types/evaluation.py—EvaluationStatusenum{evaluated, could_not_evaluate}.experiment.py— a shared_error_resulthelper tags onlycontext-overflow errors (via the existing
_is_context_exceededpredicate)as could-not-evaluate. Every other exception stays a real
score: 0 / test_pass: Falsefailure, so a genuine evaluator bug (e.g. aKeyErrorin a custom evaluator) still surfaces rather than being hidden.Statuses are threaded through aggregation; could-not-evaluate cases are
excluded from
overall_scoreand from the diagnosis on-failure trigger.types/evaluation_report.py— newstatuseslist +mean_score()classmethod (excludes could-not-evaluate);
flatten()threads it;_displayrenders
N/Afor an unscored case instead of0.00. Reports written beforethis field existed deserialize unchanged.
display_console.py— could-not-evaluate cases are dropped from both thenumerator and denominator of the printed Pass Rate.
Behavior
score: 0, counted as a failcould_not_evaluate, excluded from score + pass rate, shownN/Ascore: 0, counted as a failRaw per-case
scores/test_passeslists are left intact; only the aggregatesand the display change.
Tests / checks
mean_scoreexclusion,flattenthreading, round-tripserialization, legacy reports without the field.
against divide-by-zero.
non-overflow error stays a failure.
ruff checkandmypy -p srcclean.