fix(mcp): stop an unresolved reference from blocking safe_to_install - #526
Conversation
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
REFERENCE_UNRESOLVED is emitted for both missing and ambiguous references in build_context.py, so reference_caveat_only also makes an ambiguous-reference scan safe_to_install=true. Those cases remain intentionally incomplete because the referenced target cannot be determined. Could we distinguish missing from ambiguous and exempt only the intended missing-reference case, with an ambiguous-reference regression test?
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Reviewed head ad0e683c3dc6086bb7309f29b6302dd59ead6ff2 — REQUEST_CHANGES.
src/skillspector/mcp_server.py:153-166:reference_caveat_onlyaccepts everyREFERENCE_UNRESOLVEDledger row, but that reason code represents both a missing target and an ambiguous target. An ambiguous reference can point at multiple bundled artifacts, so the scanner has not established which bytes the instruction reaches; nevertheless this branch can returnsafe_to_install=true. Preserve the resolution status in the completeness data or introduce distinct missing/ambiguous reason codes, exempt only the intended missing-output case, and add an MCP regression test proving an ambiguous reference remains unsafe.
Required checks pass, but the requested correction and mergeStateStatus=BEHIND both block merging.
|
As the reporter of #510, I agree with the missing-only scope proposed in the review. Ambiguous references should remain blocking, and I'm fine with narrowing the original issue wording accordingly. |
ad0e683 to
040c676
Compare
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous one, so reference_caveat_only exempted ambiguous references from safe_to_install too, even though the scanner cannot tell which bytes an ambiguous reference actually reaches. Split the reason into REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks), and add a regression proving an ambiguous reference stays unsafe. Addresses review feedback on PR NVIDIA#526. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous one, so reference_caveat_only exempted ambiguous references from safe_to_install too, even though the scanner cannot tell which bytes an ambiguous reference actually reaches. Split the reason into REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks), and add a regression proving an ambiguous reference stays unsafe. Addresses review feedback on PR NVIDIA#526.
f2a6d92 to
4553077
Compare
|
Good catch, both of you —
Added Branch has been rebased onto latest |
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous one, so reference_caveat_only exempted ambiguous references from safe_to_install too, even though the scanner cannot tell which bytes an ambiguous reference actually reaches. Split the reason into REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks), and add a regression proving an ambiguous reference stays unsafe. Addresses review feedback on PR NVIDIA#526. Signed-off-by: goodhee <goodhee.dev@gmail.com>
4553077 to
6fd4c0a
Compare
A SKILL.md that names a file the bundle does not carry - most commonly an output the skill writes at runtime, e.g. "save the result to `out.md`" - resolves to no bundled path, so reference resolution records REFERENCE_UNRESOLVED and analysis_completeness.is_complete becomes false. The MCP gate consumed that single boolean, so a scan with zero findings and every discovered file fully inspected was reported unsafe to install purely because of a documentation wording choice. Writing the same filename as unquoted prose kept the scan complete, so the verdict depended on markup rather than content. An unresolved reference hides no bundled bytes: analysis scope is driven by discovery, not by what SKILL.md mentions, and finalize_inspection_ledger already refuses to synthesize an AE1 coverage finding for one. Treat a scan whose exceptional rows are all REFERENCE_UNRESOLVED as install-eligible, and require partially_inspected_files to be zero now that is_complete no longer covers it on its own. Any other reason code, any analyzer limitation, and any partially or entirely uninspected file keep failing closed, and an exceptional row with no attributable reason still blocks. is_complete, the rendered report, and the CAUTION recommendation are untouched, so the caveat stays visible; only the install boolean stops treating it as hidden content. Fixes NVIDIA#510. Signed-off-by: goodhee <goodhee.dev@gmail.com>
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous one, so reference_caveat_only exempted ambiguous references from safe_to_install too, even though the scanner cannot tell which bytes an ambiguous reference actually reaches. Split the reason into REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks), and add a regression proving an ambiguous reference stays unsafe. Addresses review feedback on PR NVIDIA#526. Signed-off-by: goodhee <goodhee.dev@gmail.com>
6fd4c0a to
f854738
Compare
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Rechecked the current head. My earlier concern is resolved: missing references now use a distinct REFERENCE_MISSING reason that alone can receive the MCP caveat, while ambiguous references remain REFERENCE_UNRESOLVED and continue to block safe_to_install. The added ambiguous-reference regression covers the unsafe case directly. No remaining blocker from my review.
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Re-reviewed current head f854738d12abe1832b59b28e662832bc1b0963b1. The prior install-gating issue is resolved: missing references now use REFERENCE_MISSING, while ambiguous/opaque targets remain REFERENCE_UNRESOLVED; safe_to_install only permits the missing-reference exception when there are no limitations, partially inspected files, entirely uninspected files, or execution failure. The added regressions cover the safe missing-reference case and both unsafe unresolved cases. I found no remaining required code, test, documentation, security, or compatibility changes.
Required checks are green. Merge remains blocked by the PR's current conflicts (mergeable=CONFLICTING, mergeStateStatus=DIRTY).
Resolve conflict in mcp_server.py between our reference_caveat_only exemption (partially_inspected + REFERENCE_MISSING) and upstream's analysis_requirement_met LLM-completeness gate from NVIDIA#410 — both conditions are now required for safe_to_install. Signed-off-by: goodhee <goodhee.dev@gmail.com>
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Rechecked current e85ebc5a after the merge from main. The PR-specific diff remains the same four files, the missing-vs-ambiguous reference fix is intact, and required checks are green. No new blocker from me.
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
Re-reviewed exact head e85ebc5aa6764e438a919400fa3f8ba6cab74326 after the main synchronization. The prior install-gating issue remains resolved: missing references use REFERENCE_MISSING, ambiguous targets remain REFERENCE_UNRESOLVED, and the MCP exception applies only when every exceptional row is missing-reference-only and there are no limitations, partially inspected files, entirely uninspected files, or execution failure. The focused safe-missing, unsafe-ambiguous, and unsafe-opaque regressions remain present.
All exact-head hosted checks pass, GitHub reports the branch mergeable and clean, and I found no remaining required code, test, documentation, security, or compatibility changes on this head.
Summary
safe_to_installno longer fails when every exceptional ledger row isREFERENCE_UNRESOLVEDpartially_inspected_files == 0, whichis_completeused to coverA SKILL.md mentioning a file the bundle does not carry (typically a runtime
output, "save the result to
out.md") resolves to nothing, sois_completegoes false and a scan with zero findings and every file fully inspected is
reported unsafe to install. Unquoted
save.mdpasses and the backticked formdoes not, so the verdict tracks markup, not content. Other reason codes,
analyzer limitations, and partial or uninspected files still block.
is_complete, the report, andCAUTIONare unchanged.Fixes #510.
Testing
uv run pytest tests/nodes/test_security_remediation.py tests/unit/test_mcp_server.py tests/nodes/test_finalize_inspection_ledger.py -q(393 passed)uv run ruff check src/ tests/uv run ruff format --check src/ tests/uv run mypy src/skillspector/mcp_server.py