feat(tools): add KnowledgeIndex for progressive disclosure over reference knowledge

README.md

@@ -200,6 +200,76 @@ evaluator = TrajectoryEvaluator(
 )
 ```
 
+### Evaluating Large Traces with Progressive Disclosure
+
+When a session is too large to inline into a judge prompt (large tool results,
+many turns), give the judge a compact overview plus discovery tools instead of
+the full trajectory. The judge loads only the spans the rubric requires:
+
+```python
+from strands_evals.evaluators import OutputEvaluator
+from strands_evals.tools.trace_index import TraceIndex
+from strands_evals.types import EvaluationData
+
+index = TraceIndex(session)  # session: a Session from any provider/mapper
+
+# for_judge() hands back both halves together so neither is forgotten:
+# the overview to put next to the answer, and the discovery tools for the judge.
+prompt_section, tools = index.for_judge()  # tools: list_spans, get_span, search_spans
+
+evaluator = OutputEvaluator(
+    rubric=(
+        "Every factual claim must be supported by tool-result evidence in the trace. "
+        "Use the trace tools to verify each claim against the evidence before scoring."
+    ),
+    tools=tools,
+)
+
+# Compose the overview into the judged output instead of the full trajectory:
+judged_output = f"{agent_answer}\n{prompt_section}"
+evaluator.evaluate(EvaluationData(input=user_prompt, actual_output=judged_output))
+```
+
+The overview is one line per span (index, type, tool name, sizes, preview);
+`list_spans`, `get_span`, and `search_spans` all page or cap their output at
+`max_read_chars` so no single tool return can overflow the judge's context. The
+rubric must tell the judge to verify claims with the tools — otherwise it scores
+off the previews alone.
+
+> **Note:** this composes with `OutputEvaluator`, whose prompt is caller-controlled.
+> It does **not** work with `TrajectoryEvaluator`, which inlines the full
+> `actual_trajectory` unconditionally and would re-create the overflow this pattern
+> exists to prevent.
+
+#### Large reference knowledge with `KnowledgeIndex`
+
+Some rubrics need the judge to check the trace against knowledge that lives
+**outside** it — the catalog of tools/skills the agent could have used, API
+specs, or policy documents. That corpus overflows the judge for the same reason a
+large trace does, and the `Session` can't answer capability questions (it only
+holds what a run *actually invoked*). `KnowledgeIndex` gives the same list / get /
+search treatment to any keyed corpus, in two modes:
+
+```python
+from strands_evals.tools.knowledge_index import KnowledgeIndex
+
+index = KnowledgeIndex(tool_catalog)  # {key: document text}, e.g. tool schemas by name
+
+# --- Mode A: retrieve-then-inject (default; deterministic, one LLM call) ---
+# The metric selects the trace-relevant keys and injects only that bounded slice.
+knowledge_block = index.render(keys=plan_tool_names)   # a <Knowledge> block
+judged_output = f"{agent_answer}\n{knowledge_block}"
+
+# --- Mode B: agentic discovery (reserve for open-ended lookup) ---
+# When the needed entry can't be predetermined ("does *any* tool cover this?").
+prompt_section, tools = index.for_judge()              # overview + list/get/search
+```
+
+Use **Mode A** whenever the relevant keys are derivable from the trace (the tools
+the plan called, the domains it touched) — it's one call and deterministic. Reach
+for **Mode B** only when the lookup is genuinely open-ended. Both honor the same
+`max_read_chars` bound and case-insensitive literal-or-regex search as `TraceIndex`.
+
 ### Trace-based Helpfulness Evaluation
 
 Evaluate agent helpfulness using OpenTelemetry traces with seven-level scoring:

src/strands_evals/tools/_progressive.py

@@ -0,0 +1,160 @@
+"""Shared machinery for progressive-disclosure indexes.
+
+`TraceIndex` (over a `Session`) and `ReferenceIndex` (over a keyed reference
+corpus) expose the same list / get / search shape to a judge agent. The paging,
+windowing, and search logic that keeps every tool return inside a `max_read_chars`
+budget is identical between them and lives here so the two indexes can't drift.
+
+None of these helpers know about spans or entries — callers pass in the already
+rendered lines and per-item text accessors, plus the noun to use in the paging
+markers (``span``/``spans`` or ``entry``/``entries``).
+"""
+
+import re
+from typing import Callable
+
+PREVIEW_CHARS = 120
+DEFAULT_MAX_READ_CHARS = 8_000
+
+
+def preview(text: str, limit: int = PREVIEW_CHARS) -> str:
+    """Collapse whitespace and truncate ``text`` to ``limit`` chars with an ellipsis."""
+    text = re.sub(r"\s+", " ", text).strip()
+    return text if len(text) <= limit else text[: limit - 3] + "..."
+
+
+def window(text: str, offset: int, max_read_chars: int) -> str:
+    """Return ``text`` from ``offset``, capped at ``max_read_chars``, with a paging marker.
+
+    Oversized content is windowed rather than returned whole so a single large
+    item can't overflow the judge's context; the marker says the next offset.
+    """
+    if offset < 0:
+        return f"ERROR: offset {offset} is negative; use offset >= 0"
+    if offset >= len(text):
+        return f"ERROR: offset {offset} beyond content length {len(text)}"
+    win = text[offset : offset + max_read_chars]
+    if offset + len(win) < len(text):
+        remaining = len(text) - offset - len(win)
+        win += f"\n[TRUNCATED: {remaining} chars remain; call again with offset={offset + len(win)}]"
+    return win
+
+
+def paged_listing(
+    header: str,
+    lines_all: list[str],
+    offset: int,
+    max_read_chars: int,
+    *,
+    unit_sg: str,
+    unit_pl: str,
+) -> str:
+    """Render a one-line-per-item overview, paged by item index to fit ``max_read_chars``.
+
+    Args:
+        header: A leading line describing the whole collection.
+        lines_all: Pre-rendered per-item lines (already prefixed with ``[id]``).
+        offset: Item index to start from.
+        max_read_chars: Cap on the returned listing.
+        unit_sg / unit_pl: Singular/plural noun for markers (``span``/``spans``).
+    """
+    total = len(lines_all)
+    if offset < 0:
+        return f"ERROR: offset {offset} is negative; use offset >= 0"
+    if total and offset >= total:
+        return f"ERROR: offset {offset} beyond last {unit_sg} index {total - 1}"
+
+    lines: list[str] = []
+    used, end = len(header), offset
+    for i in range(offset, total):
+        line = lines_all[i]
+        if lines and used + len(line) + 1 > max_read_chars:
+            break
+        lines.append(line)
+        used += len(line) + 1
+        end = i + 1
+    shown = f"Showing {unit_pl} {offset}-{end - 1} of {total}." if lines else f"0 {unit_pl} (of {total})."
+    parts = [header, shown, *lines]
+    if end < total:
+        parts.append(f"[MORE: {total - end} {unit_pl} remain; call again with offset={end}]")
+    return "\n".join(parts)
+
+
+def search_matches(
+    count: int,
+    haystack_of: Callable[[int], str],
+    ident_of: Callable[[int], str],
+    pattern: str,
+    max_matches: int,
+    is_regex: bool,
+    max_read_chars: int,
+    *,
+    unit_pl: str,
+) -> str:
+    """Search ``count`` items, returning matching identifiers with excerpts.
+
+    Case-insensitive; literal by default, regex when ``is_regex`` (with ``re.MULTILINE``
+    so ``^``/``$`` anchor to line boundaries in the newline-joined haystacks). Output is
+    bounded by ``max_read_chars`` as well as ``max_matches``, whichever comes first.
+
+    Args:
+        count: Number of items to search (indices ``0..count-1``).
+        haystack_of: Maps an item index to its searchable text.
+        ident_of: Maps an item index to the identifier shown in brackets (span index or key).
+        unit_pl: Plural noun for the stop markers.
+    """
+    if is_regex:
+        try:
+            # MULTILINE so ^/$ anchor to line boundaries in the newline-joined haystack —
+            # LLMs write anchored regexes and would otherwise read a silent "No matches"
+            # as "claim unsupported".
+            rx = re.compile(pattern, re.IGNORECASE | re.MULTILINE)
+        except re.error as exc:
+            return f"ERROR: invalid regex {pattern!r}: {exc}. Retry with is_regex=False for a literal search."
+
+        def matcher(text: str) -> list[tuple[int, int]]:
+            return [(m.start(), m.end()) for m in rx.finditer(text)]
+    else:
+        needle = pattern.lower()
+
+        def matcher(text: str) -> list[tuple[int, int]]:
+            out, low, start = [], text.lower(), 0
+            while (i := low.find(needle, start)) != -1:
+                out.append((i, i + len(needle)))
+                start = i + max(len(needle), 1)
+            return out
+
+    hits: list[str] = []
+    stop_reason: str | None = None
+    used = 0
+    for i in range(count):
+        if len(hits) >= max_matches:
+            stop_reason = "max_matches"
+            break
+        text = haystack_of(i)
+        positions = matcher(text)
+        if not positions:
+            continue
+        s, e = positions[0]
+        excerpt = preview(text[max(0, s - 60) : e + 60], 160)
+        n = len(positions)
+        suffix = f" ({n} matches)" if n > 1 else ""
+        line = f"[{ident_of(i)}]{suffix} ...{excerpt}..."
+        # Bound the whole response by max_read_chars, not max_matches alone: a generous
+        # max_matches on a large collection would otherwise blow past the per-call
+        # budget every other tool honors. Always keep at least one hit.
+        if hits and used + len(line) + 1 > max_read_chars:
+            stop_reason = "budget"
+            break
+        hits.append(line)
+        used += len(line) + 1
+    if not hits:
+        return f"No matches for {pattern!r}"
+    if stop_reason == "max_matches":
+        hits.append(f"[stopped at {max_matches} {unit_pl}; refine the pattern or raise max_matches for more]")
+    elif stop_reason == "budget":
+        hits.append(
+            f"[budget reached at {max_read_chars} chars ({len(hits)} {unit_pl} shown); "
+            f"refine the pattern to narrow results]"
+        )
+    return "\n".join(hits)