Repository navigation
docs(benchmarks): review the eight older fast-agent runs' flagged passes (land #1013 on main) - #1015
Merged
Merged
Conversation
…ses (#1013) * docs(benchmarks): regenerate Haiku, Opus and Sonnet 5.5 TB2.1 run files Their cost note still said 'Cost shown is a list-rate estimate' after the headline moved to Harbor's recorded cost (5a89279). Regenerated with the current fetch_runs.py (no rescan; the scans are kept): the note now matches the headline, and task_costs are filled in. * docs(benchmarks): review the eight older fast-agent runs' flagged passes Eight of our TB2.1 runs had no review block, so the "every flagged pass needs a decision" check never applied and their 56 flagged passes showed as "not reviewed". Each now has a review block (same note and auto-clear rule as the other reviewed runs) and a decision per flagged pass, drafted from the trajectories and following existing precedent: - 49 disqualified: Terminal-Bench task files fetched during the run (17), upstream OCaml source fetched for fix-ocaml-gc (24), pytorch-model-cli delivered as Python or an ELF stub that runs Python (7), and per-pixel overrides in path-tracing-reverse (1). - 7 cleared: /etc/hosts entries the task needs (2), pixel-patch wording that was about probing the binary (2), incidental search results for mteb (2), and a verifier path mentioned only in a comment with /tests absent (1). Regenerated with the runs' own scanner (atif-scan 0.8.0) and --moments; scans, trials and costs are unchanged. Scores fall 0.45 to 2.48 points. * docs(benchmarks): rescan the eight reviewed runs with atif-scan 0.8.3 atif-scan 0.8.3 settles two of the seven manual clears from the first pass: the break-filter-js-from-html check script naming the provided /tests/filter.py (expected.tb21.break_filter_provided_test_in_code) and the mailman hosts entry written with a printf escape. Their manual decisions are removed. The configure-git-webserver own-address entry is still flagged (0.8.3 proves own addresses only in straight-line scripts; this one sits in an if block), so it stays a manual clear, now noting its empty hidden-directory probe. The mteb reason that only saw titles now quotes the PR description one search returned. Scores and other decisions are unchanged (49 disqualified, 5 cleared). The run page's awareness panel gains 'looked for hidden tests' (access.evaluation_directory_probe, a 0.8.3 stage), shown only for runs whose scan measured it.
Deploying fast-agent with
|
| Latest commit: |
0c6da75
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f37d4337.fast-agent.pages.dev |
| Branch Preview URL: | https://docs-benchmark-review-legacy-kdld.fast-agent.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands #1013 on
main.#1013 was stacked on #1012's branch (
docs/benchmark-hub-trace-links). #1012 was squash-merged intomainfirst, and #1013 then merged into the stacked branch rather thanmain. Somainhas the Hub trace links but not the reviews.This is #1013's squash commit (8325f16) cherry-picked onto
main, unchanged. Its tree matches the final #1013 branch (including the atif-scan 0.8.3 rescan); the only difference ismain's unrelated auth-file cleanup.See #1013 for the full description:
You're given a calfskin wallet for your birthday. How would you feel about using it?
Grateful for the gift and aware of what it's made from, so I'd use it and look after it for years rather than let it sit unused.