Skip to content

fix(mcp): stop an unresolved reference from blocking safe_to_install - #526

Merged
rng1995 merged 3 commits into
NVIDIA:mainfrom
goodhee:fix/510-output-reference-caveat
Sep 18, 2026
Merged

rng1995 merged 3 commits into
NVIDIA:mainfrom
goodhee:fix/510-output-reference-caveat

Conversation

@goodhee

@goodhee goodhee commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • safe_to_install no longer fails when every exceptional ledger row is REFERENCE_UNRESOLVED
  • require partially_inspected_files == 0, which is_complete used to cover
  • regression tests for the caveat case and for the opaque-target case that must still block

A SKILL.md mentioning a file the bundle does not carry (typically a runtime
output, "save the result to out.md") resolves to nothing, so is_complete
goes false and a scan with zero findings and every file fully inspected is
reported unsafe to install. Unquoted save.md passes and the backticked form
does not, so the verdict tracks markup, not content. Other reason codes,
analyzer limitations, and partial or uninspected files still block.
is_complete, the report, and CAUTION are unchanged.

Fixes #510.

Testing

  • uv run pytest tests/nodes/test_security_remediation.py tests/unit/test_mcp_server.py tests/nodes/test_finalize_inspection_ledger.py -q (393 passed)
  • uv run ruff check src/ tests/
  • uv run ruff format --check src/ tests/
  • uv run mypy src/skillspector/mcp_server.py

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REFERENCE_UNRESOLVED is emitted for both missing and ambiguous references in build_context.py, so reference_caveat_only also makes an ambiguous-reference scan safe_to_install=true. Those cases remain intentionally incomplete because the referenced target cannot be determined. Could we distinguish missing from ambiguous and exempt only the intended missing-reference case, with an ambiguous-reference regression test?

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Reviewed head ad0e683c3dc6086bb7309f29b6302dd59ead6ff2 — REQUEST_CHANGES.

  • src/skillspector/mcp_server.py:153-166: reference_caveat_only accepts every REFERENCE_UNRESOLVED ledger row, but that reason code represents both a missing target and an ambiguous target. An ambiguous reference can point at multiple bundled artifacts, so the scanner has not established which bytes the instruction reaches; nevertheless this branch can return safe_to_install=true. Preserve the resolution status in the completeness data or introduce distinct missing/ambiguous reason codes, exempt only the intended missing-output case, and add an MCP regression test proving an ambiguous reference remains unsafe.

Required checks pass, but the requested correction and mergeStateStatus=BEHIND both block merging.

@ikaros-0117

Copy link
Copy Markdown

As the reporter of #510, I agree with the missing-only scope proposed in the review. Ambiguous references should remain blocking, and I'm fine with narrowing the original issue wording accordingly.

@goodhee
goodhee force-pushed the fix/510-output-reference-caveat branch from ad0e683 to 040c676 Compare September 15, 2026 01:15
goodhee added a commit to goodhee/SkillSpector that referenced this pull request Sep 15, 2026
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous
one, so reference_caveat_only exempted ambiguous references from
safe_to_install too, even though the scanner cannot tell which bytes
an ambiguous reference actually reaches. Split the reason into
REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks),
and add a regression proving an ambiguous reference stays unsafe.

Addresses review feedback on PR NVIDIA#526.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
goodhee added a commit to goodhee/SkillSpector that referenced this pull request Sep 15, 2026
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous
one, so reference_caveat_only exempted ambiguous references from
safe_to_install too, even though the scanner cannot tell which bytes
an ambiguous reference actually reaches. Split the reason into
REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks),
and add a regression proving an ambiguous reference stays unsafe.

Addresses review feedback on PR NVIDIA#526.
@goodhee
goodhee force-pushed the fix/510-output-reference-caveat branch from f2a6d92 to 4553077 Compare September 15, 2026 01:31
@goodhee

goodhee commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Good catch, both of you — REFERENCE_UNRESOLVED was indeed overloaded for two different situations. Fixed by splitting it:

  • Added a new LedgerReason.REFERENCE_MISSING for the "target doesn't exist in the bundle" case (e.g. a file written at runtime).
  • LedgerReason.REFERENCE_UNRESOLVED now means only "ambiguous" — matched more than one bundled artifact, target undetermined.
  • reference_caveat_only in mcp_server.py now checks for REFERENCE_MISSING specifically, so an ambiguous reference keeps blocking safe_to_install like any other exceptional reason.

Added test_ambiguous_reference_caveat_still_blocks_mcp_install: two files sharing a basename (docs/guide.md, extra/guide.md) referenced by the bare name, asserting safe_to_install is False and reason_code == "reference_unresolved". Updated the existing missing-reference test to expect the new reference_missing code.

Branch has been rebased onto latest main.

goodhee added a commit to goodhee/SkillSpector that referenced this pull request Sep 15, 2026
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous
one, so reference_caveat_only exempted ambiguous references from
safe_to_install too, even though the scanner cannot tell which bytes
an ambiguous reference actually reaches. Split the reason into
REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks),
and add a regression proving an ambiguous reference stays unsafe.

Addresses review feedback on PR NVIDIA#526.

Signed-off-by: goodhee <goodhee.dev@gmail.com>
@goodhee
goodhee force-pushed the fix/510-output-reference-caveat branch from 4553077 to 6fd4c0a Compare September 15, 2026 01:56
A SKILL.md that names a file the bundle does not carry - most commonly
an output the skill writes at runtime, e.g. "save the result to
`out.md`" - resolves to no bundled path, so reference resolution
records REFERENCE_UNRESOLVED and analysis_completeness.is_complete
becomes false. The MCP gate consumed that single boolean, so a scan
with zero findings and every discovered file fully inspected was
reported unsafe to install purely because of a documentation wording
choice. Writing the same filename as unquoted prose kept the scan
complete, so the verdict depended on markup rather than content.

An unresolved reference hides no bundled bytes: analysis scope is
driven by discovery, not by what SKILL.md mentions, and
finalize_inspection_ledger already refuses to synthesize an AE1
coverage finding for one. Treat a scan whose exceptional rows are all
REFERENCE_UNRESOLVED as install-eligible, and require
partially_inspected_files to be zero now that is_complete no longer
covers it on its own. Any other reason code, any analyzer limitation,
and any partially or entirely uninspected file keep failing closed,
and an exceptional row with no attributable reason still blocks.

is_complete, the rendered report, and the CAUTION recommendation are
untouched, so the caveat stays visible; only the install boolean stops
treating it as hidden content.

Fixes NVIDIA#510.

Signed-off-by: goodhee <goodhee.dev@gmail.com>
REFERENCE_UNRESOLVED covered both a missing target and an ambiguous
one, so reference_caveat_only exempted ambiguous references from
safe_to_install too, even though the scanner cannot tell which bytes
an ambiguous reference actually reaches. Split the reason into
REFERENCE_MISSING (exempted) and REFERENCE_UNRESOLVED (still blocks),
and add a regression proving an ambiguous reference stays unsafe.

Addresses review feedback on PR NVIDIA#526.

Signed-off-by: goodhee <goodhee.dev@gmail.com>
@goodhee
goodhee force-pushed the fix/510-output-reference-caveat branch from 6fd4c0a to f854738 Compare September 15, 2026 07:16

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the current head. My earlier concern is resolved: missing references now use a distinct REFERENCE_MISSING reason that alone can receive the MCP caveat, while ambiguous references remain REFERENCE_UNRESOLVED and continue to block safe_to_install. The added ambiguous-reference regression covers the unsafe case directly. No remaining blocker from my review.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed current head f854738d12abe1832b59b28e662832bc1b0963b1. The prior install-gating issue is resolved: missing references now use REFERENCE_MISSING, while ambiguous/opaque targets remain REFERENCE_UNRESOLVED; safe_to_install only permits the missing-reference exception when there are no limitations, partially inspected files, entirely uninspected files, or execution failure. The added regressions cover the safe missing-reference case and both unsafe unresolved cases. I found no remaining required code, test, documentation, security, or compatibility changes.

Required checks are green. Merge remains blocked by the PR's current conflicts (mergeable=CONFLICTING, mergeStateStatus=DIRTY).

Resolve conflict in mcp_server.py between our reference_caveat_only
exemption (partially_inspected + REFERENCE_MISSING) and upstream's
analysis_requirement_met LLM-completeness gate from NVIDIA#410 — both
conditions are now required for safe_to_install.

Signed-off-by: goodhee <goodhee.dev@gmail.com>

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked current e85ebc5a after the merge from main. The PR-specific diff remains the same four files, the missing-vs-ambiguous reference fix is intact, and required checks are green. No new blocker from me.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Re-reviewed exact head e85ebc5aa6764e438a919400fa3f8ba6cab74326 after the main synchronization. The prior install-gating issue remains resolved: missing references use REFERENCE_MISSING, ambiguous targets remain REFERENCE_UNRESOLVED, and the MCP exception applies only when every exceptional row is missing-reference-only and there are no limitations, partially inspected files, entirely uninspected files, or execution failure. The focused safe-missing, unsafe-ambiguous, and unsafe-opaque regressions remain present.

All exact-head hosted checks pass, GitHub reports the branch mergeable and clean, and I found no remaining required code, test, documentation, security, or compatibility changes on this head.

@rng1995
rng1995 merged commit 07cce5a into NVIDIA:main Sep 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[False positive] Output references like "save to save.md" make a clean skill partial: is_complete=false / safe_to_install=false

4 participants