[FEATURE] Progressive disclosure for large reference knowledge, not just traces (extend TraceIndex with a KnowledgeIndex)

[pdebjyot]

Summary

#343 gave judge-based evaluators progressive disclosure over a large trace: instead of inlining a whole Session into the prompt (which overflows the judge and gets scored as a failure — #342), TraceIndex builds a compact overview plus list_spans/get_span/search_spans tools and a for_judge() composition helper, so the judge reads only the spans it needs.

The same overflow problem exists for a second kind of input that is not in the trace: a large reference corpus the judge must consult to score a rubric. This issue asks for the same progressive-disclosure treatment for arbitrary keyed reference material.

The gap

Grounding, feasibility, and instruction-adherence rubrics frequently need the judge to check the trace against knowledge that lives outside it, for example:

• the catalog of tools/skills the agent could have used and their full input/output schemas or contracts,
• API specs, policy/guideline documents, or domain reference text.

Two things make this distinct from TraceIndex:

1. It overflows for the same reason. A realistic capability catalog (dozens of skills, hundreds of tool schemas) or a policy corpus does not fit inline any more than a large trajectory does — and inlining all of it degrades judge accuracy even when it does fit (lost-in-the-middle, position bias; see the references in [BUG] Judge-based evaluators silently score 0 when a trace exceeds the judge's context window #342).
2. The trace can't answer capability questions. A Session only contains the tools/skills a given run actually invoked. A rubric like "was there a capability that would have satisfied this request?" or "does the plan conform to the documented contract for the tools it called?" needs the full catalog, which is a standalone knowledge source, not a slice of the trace. available_tools on the trace tells you what that run was handed, not the universe.

So TraceIndex (bound to a Session) cannot serve these rubrics, and inlining the corpus reintroduces #342.

Proposed direction — a KnowledgeIndex sibling to TraceIndex

Add a KnowledgeIndex over an arbitrary collection of keyed text documents, with the same shape as TraceIndex so it composes with OutputEvaluator the identical way:

from strands_evals.tools.knowledge_index import KnowledgeIndex

# entries: key -> document text (tool schemas by tool name, skill docs by domain, …)
index = KnowledgeIndex(catalog)                      # a large keyed corpus

# --- Mode A: retrieve-then-inject (deterministic, one LLM call) ---
# The metric selects the trace-relevant keys and injects only those.
reference_block = index.render(keys=plan_tool_names) # a bounded <Knowledge> block
judged_output = f"{agent_answer}\n{reference_block}"

# --- Mode B: agentic discovery (judge pulls what it needs) ---
prompt_section, tools = index.for_judge()            # overview + list/get/search tools

Concretely:

• overview(offset) — one line per entry (key, optional description, size, truncated preview), paged so it always fits in max_read_chars.
• list_entries / get_entry(key) / search_entries(pattern, …) — the discovery tools, same paging/windowing/max_read_chars bound and case-insensitive literal-or-regex search (with line-anchored ^/$) as TraceIndex.
• for_judge() → (prompt_section, tools) — the atomic composition helper.
• render(keys) — a deterministic, tool-free path that renders just the selected entries as a bounded <Knowledge> block for the common retrieve-then-inject case (metric knows which entries are relevant from the trace, so no extra round-trips).

Two usage modes, and when to use which

• Retrieve-then-inject (default). When the relevant reference keys are derivable from the trace (the tools the plan called, the domains it touched), the metric selects those keys and injects only that slice via render(). One LLM call, deterministic, cheap.
• Agentic discovery (reserve). When the needed reference can't be predetermined — e.g. "does any entry cover this request?" when the trace names no matching tool — hand the judge the tools via for_judge() and let it look things up. More round-trips and harder to make deterministic, so use it only where open-ended lookup is genuinely required.

Why not just reuse TraceIndex

TraceIndex is bound to a Session and keys by span index; a reference corpus is an arbitrary keyed document set with no spans and no timeline. The list/get/search/paging machinery is shared (worth extracting so both indexes stay identical in behavior), but the surface — keyed entries, render() for slice injection — is different enough to warrant a sibling class rather than overloading TraceIndex.

Scope

• Domain-agnostic: KnowledgeIndex takes any key -> text mapping; how a corpus is assembled/loaded stays with the caller.
• No change to TraceIndex's observable behavior; only a shared-helper extraction.
• Follow-up PR with the implementation, A/B evidence (a judge with an oversized catalog scored via inline vs. render/for_judge), and tests is ready if this direction looks right.