Repository navigation
Conversation
check-docs runs the SQL in the skills and check-plugin checks the manifests, but nothing tested whether Claude picks the right skill for a request and writes SQL that runs. A vague or overlapping description silently stops a skill from loading. - plugins/anofox-tabular/evals/: 11 cases in the claude plugin eval format (prompt.md + graders/*.md), one per skill plus five boundary requests (daily volume drop, unusual rows per customer, email validation vs. masking, schema drift). Prompts name only tables and the task. Graders are judge-free: tool_used on Skill for the expected skill, regex on the answer.sql the agent writes for the expected functions. - scripts/run_plugin_evals.py: claude plugin eval has no custom-code graders, so the script runs the suite with --keep-temp and then executes each run's answer.sql after evals/fixtures/fixture.sql with the loader of check_doc_snippets.py; a run passes without an error and with rows. - make eval-plugin, and a weekly/on-demand workflow (plugin-eval.yml) with the ANTHROPIC_API_KEY secret, the released v2.0.0 extension (sha256 checked) and the DuckDB 2.0.0 CLI; it skips with a notice without the secret. A full run costs about 2 USD and 3 minutes. - docs/dev/testing.md, README.md and CLAUDE.md describe the suite.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #139.
make check-docsruns the SQL in the skills andmake check-pluginchecks the manifests, but nothing tested whether Claude picks the right skill for a request and writes SQL that runs.Change
plugins/anofox-tabular/evals/: 11 cases in theclaude plugin evalformat (https://code.claude.com/docs/en/plugin-evals), one per skill plus five boundary requests (daily volume drop, unusual rows per customer, email validation vs masking, schema drift). Prompts name only the tables and the task. Graders need no judge model:tool_usedonSkillfor the expected skill, and a regex on theanswer.sqlthe agent writes.scripts/run_plugin_evals.py:claude plugin evalhas no custom-code graders, so the script runs the suite with--keep-temp, then executes each run'sanswer.sqlafterevals/fixtures/fixture.sql(loader ofcheck_doc_snippets.py). A run passes when the script runs without an error and returns rows.make eval-pluginand.github/workflows/plugin-eval.yml: weekly (Mondays 05:17 UTC) and on demand, never on PRs. It uses the released v2.0.0 extension (sha256 checked) and the DuckDB 2.0.0 CLI, no build, and stops at--max-cost-usd 5. It needs theANTHROPIC_API_KEYrepository secret; without it the job prints a notice and succeeds.docs/dev/testing.md, README.md and CLAUDE.md describe the suite.Local run
11/11 cases passed (expected skill invoked, SQL ran with rows). It took 195 s and cost about 2.10 USD at list price.
Limits
socat); execution is checked afterwards.boundary-unusual-rows-per-customerregex only requirescustomer_id, since a correct answer can be plain SQL.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.