Skip to content

feat: align evaluation workflows with the latest CalypsoAI API - #14

Merged
f5-aryan merged 1 commit into
mainfrom
feature/7-latest-api-compatibility
Oct 27, 2026
Merged

f5-aryan merged 1 commit into
mainfrom
feature/7-latest-api-compatibility

Conversation

@f5-aryan

@f5-aryan f5-aryan commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This brings scan and LLM-response evaluation in line with the latest CalypsoAI API while keeping existing entry points and output defaults.

  • Apply consistent detection scoring across evaluation, saved-result analysis, exports, and PDF reports, with visible error and exclusion counts.
  • Add bounded requests, lazy configuration, project/provider selection, safe output checks, preserved result evidence, and corrected concurrent timing.
  • Add offline API/workflow coverage and required CI checks, refresh public documentation and the PDF quick reference, and remove retired GitLab CI/templates.
  • Keep generated evaluation data and credentials out of Git. The only committed results entry is the empty results/.gitkeep placeholder.

Validation

  • make test-core: all 146 curated tests passed, including 105 contract/workflow cases with 97.20% combined shared-module coverage.
  • CLI feature checks: 10/10 passed; concurrent basic checks passed.
  • Critical Flake8, scoped Mypy, changed-module Black checks, and staged whitespace checks passed.
  • Full legacy discovery: 146 passed and 11 failed. Those reader/migration failures match the previously reproduced untouched-source baseline; this PR does not claim the broader legacy suite is green.

The documentation identifies existing legacy conversion/migration limitations and supported alternatives. This change does not claim to repair every legacy dataset utility.

GitHub CI: all six required test jobs and the security job passed. The overall workflow succeeded. The non-blocking lint job fails on the repository-wide black --check --diff . formatting check; its log shows broad formatting differences rather than a syntax failure. No unrelated whole-tree formatting sweep is included.

Review notes

Related to #7 and #10; neither issue is automatically closed. The shared-utility extraction uses focused modules rather than claiming the complete proposed package layout is implemented.

This touches evaluator/runner files also changed by #13. Coordinate merge order with that separate scanner-name enhancement.

The large specification snapshot supports offline contract fixtures. Public release highlights are in CHANGES_SUMMARY.md.

Context

Implementation plan

Agent conversation

Co-Authored-By: Warp agent@warp.dev

Unify outcome scoring, request handling, result formats, and reporting across evaluation workflows. Add offline contract coverage and refresh user documentation. Remove retired GitLab configuration and templates while extending GitHub Actions checks.

Related to #7 and #10.

Co-Authored-By: Warp <agent@warp.dev>
@f5-aryan

Copy link
Copy Markdown
Collaborator Author

all good to merge

@f5-aryan f5-aryan self-assigned this Oct 27, 2026
@f5-aryan
f5-aryan merged commit 52e292b into main Oct 27, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant