Skip to content

test(plugin): claude plugin eval suite for skill routing and SQL - #143

Open
jrosskopf wants to merge 1 commit into
mainfrom
feat/plugin-eval
Open

jrosskopf wants to merge 1 commit into
mainfrom
feat/plugin-eval

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #139.

make check-docs runs the SQL in the skills and make check-plugin checks the manifests, but nothing tested whether Claude picks the right skill for a request and writes SQL that runs.

Change

  • plugins/anofox-tabular/evals/: 11 cases in the claude plugin eval format (https://code.claude.com/docs/en/plugin-evals), one per skill plus five boundary requests (daily volume drop, unusual rows per customer, email validation vs masking, schema drift). Prompts name only the tables and the task. Graders need no judge model: tool_used on Skill for the expected skill, and a regex on the answer.sql the agent writes.
  • scripts/run_plugin_evals.py: claude plugin eval has no custom-code graders, so the script runs the suite with --keep-temp, then executes each run's answer.sql after evals/fixtures/fixture.sql (loader of check_doc_snippets.py). A run passes when the script runs without an error and returns rows.
  • make eval-plugin and .github/workflows/plugin-eval.yml: weekly (Mondays 05:17 UTC) and on demand, never on PRs. It uses the released v2.0.0 extension (sha256 checked) and the DuckDB 2.0.0 CLI, no build, and stops at --max-cost-usd 5. It needs the ANTHROPIC_API_KEY repository secret; without it the job prints a notice and succeeds.
  • docs/dev/testing.md, README.md and CLAUDE.md describe the suite.

Local run

11/11 cases passed (expected skill invoked, SQL ran with rows). It took 195 s and cost about 2.10 USD at list price.

Limits

  • The agent in a case does not run SQL itself (Bash in evals needs a sandbox with socat); execution is checked afterwards.
  • The boundary-unusual-rows-per-customer regex only requires customer_id, since a correct answer can be plain SQL.
  • The workflow installs the latest Claude Code with the default model, so scores can move when a new model ships.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

check-docs runs the SQL in the skills and check-plugin checks the manifests,
but nothing tested whether Claude picks the right skill for a request and
writes SQL that runs. A vague or overlapping description silently stops a
skill from loading.

- plugins/anofox-tabular/evals/: 11 cases in the claude plugin eval format
  (prompt.md + graders/*.md), one per skill plus five boundary requests
  (daily volume drop, unusual rows per customer, email validation vs.
  masking, schema drift). Prompts name only tables and the task. Graders
  are judge-free: tool_used on Skill for the expected skill, regex on the
  answer.sql the agent writes for the expected functions.
- scripts/run_plugin_evals.py: claude plugin eval has no custom-code
  graders, so the script runs the suite with --keep-temp and then executes
  each run's answer.sql after evals/fixtures/fixture.sql with the loader of
  check_doc_snippets.py; a run passes without an error and with rows.
- make eval-plugin, and a weekly/on-demand workflow (plugin-eval.yml) with
  the ANTHROPIC_API_KEY secret, the released v2.0.0 extension (sha256
  checked) and the DuckDB 2.0.0 CLI; it skips with a notice without the
  secret. A full run costs about 2 USD and 3 minutes.
- docs/dev/testing.md, README.md and CLAUDE.md describe the suite.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regression-test Claude Code skill selection with a plugin eval suite

1 participant