[FEATURE] Could-Not-Evaluate verdict for evaluators that cannot score a case

[max-rattray-aws]

[FEATURE] Add status field to EvaluationOutput for non-gradable results

Problem Statement

When an evaluator cannot produce a meaningful score for a given case, it has no way to say so. The only output shape is EvaluationOutput(score=..., test_pass=..., reason=...), which forces every result into the pass/fail binary. An evaluator that legitimately cannot grade a case (missing input data, inapplicable rubric, harness error) must pick a score anyway, and that score pollutes aggregates.

Take an error-handling evaluator. If the agent's trajectory contains no tool errors, there is nothing to evaluate. Today the evaluator has three bad options:

1. Return score=1.0, test_pass=True and inflate the pass rate with a hollow pass.
2. Return score=0.0, test_pass=False and penalize a correct agent for something that never happened.
3. Encode "could not evaluate" into the reason string and hope downstream consumers parse it out.

Option 3 is what I currently do. It works, but it's fragile and invisible to the framework. The _default_aggregator on Evaluator computes a straight average:

avg_score = sum(o.score for o in outputs) / len(outputs)

There is no way to exclude a result from this sum without the aggregator parsing reason strings. That coupling should not exist.

The same problem shows up at the Experiment level. When building the EvaluationReport, scores are averaged directly:

overall_score=sum(scores) / len(scores) if scores else 0,

A non-gradable result at score=0.0 drags the average down. A non-gradable result at score=1.0 inflates it. Either way the aggregate lies.

Proposed Solution

Add an optional status field to EvaluationOutput that defaults to "graded":

class EvaluationOutput(BaseModel):
    score: float
    test_pass: bool
    reason: str | None = None
    label: str | None = None
    status: str = "graded"  # "graded" | "could_not_evaluate" | "informational"

Three values:

• "graded" (default) means the score and test_pass are real verdicts. Backward compatible with every existing evaluator and consumer.
• "could_not_evaluate" means the evaluator tried to grade but could not. Preconditions not met, harness failure, missing data. Aggregators should exclude this from pass-rate and score math.
• "informational" means the evaluator does not pass/fail by design. It surfaces content for human review but should never count toward a numeric aggregate.

The field is optional and defaults to "graded", so existing code keeps working with zero changes.

The _default_aggregator in evaluator.py would filter before computing:

@staticmethod
def _default_aggregator(outputs: list[EvaluationOutput]) -> tuple[float, bool, str]:
    graded = [o for o in outputs if o.status == "graded"]
    if not graded:
        return (0.0, False, "No gradable evaluation outputs produced")
    avg_score = sum(o.score for o in graded) / len(graded)
    all_pass = all(o.test_pass for o in graded)
    combined_reason = " | ".join(o.reason for o in graded if o.reason)
    return avg_score, all_pass, combined_reason

The Experiment report assembly in experiment.py would also need to exclude non-graded rows when computing overall_score. Today it does sum(scores) / len(scores). With this feature, it would filter on the status carried through from the aggregated outputs. Concretely, the _run_evaluator method already calls evaluator.aggregator(outputs), and since non-graded outputs would be excluded at the aggregator level, the per-evaluator score would already be correct. The remaining gap is when multiple evaluators are flattened via EvaluationReport.flatten(), which also averages all scores. Adding status to EvaluationOutput (and propagating it into the report's cases dict) would let flatten() skip non-graded rows too.

An evaluator producing a non-gradable result would look like:

def evaluate(self, evaluation_case: EvaluationData) -> list[EvaluationOutput]:
    if not self._has_errors_in_trajectory(evaluation_case):
        return [EvaluationOutput(
            score=0.0,
            test_pass=False,
            reason="No tool errors occurred in trajectory.",
            status="could_not_evaluate",
        )]
    # ... actual grading logic

Use Case

1. An error-handling evaluator that only applies when tool errors exist. No errors means nothing to score. Today it must fake a pass or a fail. With status="could_not_evaluate" the result gets excluded from aggregates and surfaced separately in reports.

2. A faithfulness evaluator where the agent declined to answer (empty output). There is no claim to ground-check. Instead of scoring 1.0 or 0.0, it returns CNE and the aggregate reflects only cases where faithfulness was actually testable.

3. A "referenced tickets" evaluator that lists which tickets the agent mentioned but does not pass/fail. It produces status="informational" results that show up in a report for human review without counting toward the suite's pass rate.

4. Context-window overflow during judging. Issue [BUG] Judge-based evaluators silently score 0 when a trace exceeds the judge's context window #342 describes judge-based evaluators scoring 0 when a trace exceeds the context window. The error-isolation catch block could set status="could_not_evaluate" instead of recording score=0, test_pass=False, giving downstream consumers a clean signal that the 0 is a harness failure, not a quality failure.

Alternatives Considered

A convention on score value (e.g. score=-1.0 means skip). I do this today. It works until someone writes an evaluator that legitimately scores on a scale including negative numbers, or a consumer forgets to filter. A dedicated field is explicit and impossible to misinterpret.

A convention on reason prefix (e.g. "[Could Not Evaluate] ..."). Also what I do today. Brittle, requires string parsing, and invisible to typed consumers. The framework should not require regex on a reason string to compute a correct average.

A separate EvaluationSkip type or exception. More disruptive. Evaluators return list[EvaluationOutput], and that contract is baked into every evaluator subclass and into _run_evaluator's aggregation flow. Adding a union type or exception would require every consumer to handle a new branch. An optional field on the existing model is the smallest possible change.

Additional Context

• Issue [BUG] Judge-based evaluators silently score 0 when a trace exceeds the judge's context window #342 identifies the same root cause from the harness-error angle (context-window overflow producing score=0). This feature gives [BUG] Judge-based evaluators silently score 0 when a trace exceeds the judge's context window #342 a clean landing spot on the model layer. The two are complementary.
• Issue [FEATURE] Skill evaluation: with vs without skill batch-run, aggregation, comparison, analysis #247 (skill evaluation) discusses aggregation of paired runs. Aggregation that correctly excludes non-gradable results is relevant there too, though [FEATURE] Skill evaluation: with vs without skill batch-run, aggregation, comparison, analysis #247 does not strictly depend on this landing first.
• The field is a plain string rather than an enum to allow future extension without a breaking change. A Literal["graded", "could_not_evaluate", "informational"] type annotation is fine for documentation and tooling while remaining forward-compatible.

[pdebjyot]

Happy to take this on — it's the harness-side companion to #342 (use case #4 is exactly what that issue calls for), and it composes cleanly with #343: TraceIndex lets the judge read a large trace, could_not_evaluate handles the cases that still can't be scored. Instead of _run_evaluator's error-isolation catch recording score=0, test_pass=False, it sets status="could_not_evaluate" and the aggregate stops lying.

One thing worth flagging from the code: there are three averaging sites, not one — _default_aggregator (evaluator.py), the Experiment rollup (overall_score = sum(scores)/len(scores)), and EvaluationReport.flatten(). Filtering only in _default_aggregator fixes the per-evaluator score but not the experiment-level total, so status needs to propagate into the report's cases/scores for all three. Also worth handling the all-non-graded case explicitly (return a CNE result, not 0.0/fail), so a fully-skipped evaluator doesn't read as a failure.

Agreed on plain str defaulting to "graded" — keeps it fully backward compatible. If you're good with it, I'll open a PR: the field, filtering across all three sites, _run_evaluator setting could_not_evaluate on the error path, and tests for mixed-status + all-skipped. This is independent of #343, so it can land on its own.

[max-rattray-aws]

Implementation PR: #359