Skip to content

Pull requests: strands-agents/evals

New pull request New
33 Open 238 Closed
33 Open 238 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

fix: improve root cause detector parsing returned json area-detectors Failure detection and root cause analysis of agent sessions bug Something isn't working
#401 opened Sep 15, 2026 by ybdarrenwang Collaborator Loading…
9 tasks done
10
Structured Evaluator for Strands-Eval with Stickler area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#400 opened Sep 14, 2026 by sromoam Loading…
9 tasks done
1
feat: report judge context overflow as could-not-evaluate, not score-0 area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#399 opened Sep 14, 2026 by pdebjyot Contributor Loading…
feat(output_evaluator): add cache_config support for prompt caching area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#397 opened Sep 11, 2026 by shssrd Loading…
8
feat(tools): add KnowledgeIndex for progressive disclosure over reference knowledge area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#396 opened Sep 9, 2026 by pdebjyot Contributor Loading…
1
fix: expose and serialize user tools on TrajectoryEvaluator area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics bug Something isn't working
#394 opened Sep 7, 2026 by AmirF194 Loading…
8 tasks done
1
1
fix: make EvaluationReport.to_file() reject non-strict JSON area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-devx Developer experience: papercuts, confusing public APIs, error messages, ergonomics, usability bug Something isn't working
#393 opened Sep 4, 2026 by serenearyal Loading…
9 tasks done
1
1
feat(providers): add session discovery + evaluate_sessions batch helper (#143) area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL enhancement New feature or request
#391 opened Sep 1, 2026 by lifelmy Loading…
4 tasks done
1
1
feat(store): add per-run keying and config-drift guard to EvaluationDataStore (#347) area-core Core eval framework: Case, Experiment, task handler, evaluation data stores enhancement New feature or request
#390 opened Sep 1, 2026 by lifelmy Loading…
6 of 7 tasks
2
feat(experiment): add ExperimentTrendAnalyzer for cross-run regression detection (#186) area-core Core eval framework: Case, Experiment, task handler, evaluation data stores enhancement New feature or request
#389 opened Sep 1, 2026 by lifelmy Loading…
6 of 7 tasks
1
1
feat(classification): add classify_risk utility for multi-run results (#349) area-core Core eval framework: Case, Experiment, task handler, evaluation data stores enhancement New feature or request
#388 opened Aug 31, 2026 by lifelmy Loading…
3
feat(evaluators): add FuzzyEquals deterministic evaluator area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#387 opened Aug 31, 2026 by lifelmy Loading…
1
ci: update langfuse requirement from <4,>=2.0.0 to >=2.0.0,<5 area-community Repo health, governance, contributor process, release process, and CI dependency bumps chore Maintenance tasks, dependency updates, CI changes, refactoring with no user-facing impact dependencies Pull requests that update a dependency file python Pull requests that update python code
#367 opened Aug 13, 2026 by dependabot Bot Loading…
2
feat: add ToolEfficiencyEvaluator for session-level tool usage analysis area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#362 opened Aug 10, 2026 by max-rattray-aws Contributor Loading…
8
feat: add evaluator metadata method and types area-devx Developer experience: papercuts, confusing public APIs, error messages, ergonomics, usability area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#361 opened Aug 10, 2026 by max-rattray-aws Contributor Loading…
11
feat: add failure cohort analysis for evaluation reports area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#360 opened Aug 10, 2026 by max-rattray-aws Contributor Loading…
7
feat: add status field to EvaluationOutput area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#359 opened Aug 10, 2026 by max-rattray-aws Contributor Loading…
9
feat: add inclusive language evaluator area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#358 opened Aug 10, 2026 by max-rattray-aws Contributor Loading…
15
feat(tools): add TraceIndex for progressive trace disclosure in judge-based evaluators area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL enhancement New feature or request
#343 opened Aug 3, 2026 by pdebjyot Contributor Loading…
5 tasks done
20
feat(experiment): add opt-in model routing for evaluators area-core Core eval framework: Case, Experiment, task handler, evaluation data stores area-evaluators Evaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics enhancement New feature or request
#325 opened Jul 23, 2026 by pdebjyot Contributor Loading…
4 tasks done
1
10
fix: convert Langfuse AGENT observations regardless of parent area-tracing Trace/session ingestion: providers, session mappers, extractors, telemetry/OTEL bug Something isn't working
#314 opened Jul 15, 2026 by poshinchen Contributor Loading…
7 of 9 tasks
1
7
test(redteam): cover RedTeamCase metadata sync and _build_system_prompt assembly area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation chore Maintenance tasks, dependency updates, CI changes, refactoring with no user-facing impact
#301 opened Jul 7, 2026 by zeroshotmind Loading…
9 of 11 tasks
1
feat(redteam): reframe attacker prompts to survive aligned attacker models area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation enhancement New feature or request
#298 opened Jul 2, 2026 by kevmyung Contributor Loading…
6 of 9 tasks
1
fix(redteam): separate errored attacks from breaches in ASR area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation bug Something isn't working
#296 opened Jul 2, 2026 by kevmyung Contributor Loading…
7 of 9 tasks
1
feat(redteam): redesign risk categories for agent-centric evaluation area-redteam Red teaming: adversarial generation, attack strategies, attack success evaluation enhancement New feature or request
#290 opened Jul 1, 2026 by kevmyung Contributor Loading…
9 tasks done
1
ProTip! Type g i on any issue or pull request to go back to the issue listing page.