Skip to content

[6/9] Add pinned CODA contract conformance tooling - #3571

Open
Dmitrii Shilov (bk201-) wants to merge 5 commits into
dev/dshilov/account-overview-05-cost-estimatesfrom
dev/dshilov/account-overview-06-coda-conformance
Open

Dmitrii Shilov (bk201-) wants to merge 5 commits into
dev/dshilov/account-overview-05-cost-estimatesfrom
dev/dshilov/account-overview-06-coda-conformance

Conversation

@bk201-

@bk201- Dmitrii Shilov (bk201-) commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add integrity-checked acquisition of an immutable CODA revision, Gherkin scenario parsing, production-evaluator bindings for DX-001 through DX-024, synthetic tooling regressions and local conformance reports.

Ordinary builds and unit tests do not fetch or execute private upstream contracts. Private checkouts and reports are ignored, excluded from lint/format and VSIX packaging, and are not uploaded as public artifacts. Public Actions execution of private scenarios is refused without an explicit trusted-private-runner opt-in.

Stack

Part 6 of 9 in the Account Overview advisory stack. Depends on #3570. Base: dev/dshilov/account-overview-05-cost-estimates.
Each subsequent PR targets its predecessor branch so GitHub can represent the chain as a native PR stack.

Validation

The integrated stack passed localization generation, oxfmt, lint, all TypeScript build targets, and all unit suites. The full unit command used VITEST_MAX_WORKERS=2 to avoid local resource-contention timeouts. Intermediate runtime branches also passed builds and focused regressions.
Review fixes are committed to their originating PR layers; descendants are rebased onto the updated predecessors.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Integration Tests

  • Shell MCP Integration Tests: ✅ success
  • Integration Tests (extension host): ✅ success

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Build (Compile, Lint, Prettier, l10n, Package)

🧱 Step Results

  • Validate Version: ✅ success
  • Localization: ✅ success
  • Lint: ✅ success
  • Prettier: ✅ success
  • Compile (tsc): ✅ success
  • Package: ✅ success
  • External Skills: verification ✅ success; tests ✅ success

📥 Artifacts (run)

@github-code-quality

github-code-quality Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: TypeScript

TypeScript / code-coverage/vitest

dev/dshilov/account-... 5c8102d dev/dshilov/account-... 18518f1 +/-
77% 78% +1%
View coverage of the 10 most impacted files.
File dev/dshilov/account-... 5c8102d dev/dshilov/account-... 18518f1 +/-
src/panels/acco...onalFixtures.ts 0% 67% +67%
src/panels/acco...nce/bindings.ts 0% 73% +73%
src/panels/acco...onalBindings.ts 0% 86% +86%
src/panels/acco...nce/features.ts 0% 93% +93%
src/panels/acco...s/backupCost.ts 0% 95% +95%
src/panels/acco...mance/shared.ts 0% 95% +95%
src/panels/acco...rorsDetector.ts 0% 96% +96%
src/panels/acco...stManagement.ts 0% 99% +99%
src/panels/acco...ostEstimates.ts 0% 100% +100%
src/panels/acco...tionDetector.ts 0% 100% +100%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.


Updated November 19, 2026 17:32 UTC

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🧪 Unit Tests

🧪 Results

  • Unit Tests (vitest): ✅ success

📈 Coverage

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🎭 E2E Tests (Playwright + VS Code)

🧪 Result

  • E2E Tests: ✅ success

⚠️ Flaky tests (3)

These tests passed only after retrying. The test run is still successful unless other tests failed.

  • document-concurrency.spec.ts › document editor optimistic concurrency › discards stale editor changes and refreshes the latest server version — document-concurrency.spec.ts:64 (default, 2 attempts)
  • migration.spec.ts › Migration Assistant › explains a stale upstream file without changing successful run history — migration.spec.ts:984 (default, 2 attempts)
  • queryEditor-result-toolbar-stats.spec.ts › queryEditor-result-toolbar-stats › Copy and Export actions are invocable without error — queryEditor-result-toolbar-stats.spec.ts:84 (default, 2 attempts)

📥 Artifacts (run)

Tip: the HTML report artifact contains a self-contained Playwright report.
Download the zip, extract, and open index.html — or run
npx playwright show-report <extracted-dir> for the interactive view.

@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from 89409b9 to 937236d Compare November 11, 2026 18:57
@bk201- Dmitrii Shilov (bk201-) changed the title [6/7] Add pinned CODA contract conformance tooling [6/8] Add pinned CODA contract conformance tooling Nov 12, 2026
@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from 937236d to f5165c7 Compare November 13, 2026 11:09
Copilot AI balanced review requested due to automatic review settings November 13, 2026 11:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The pinned private CODA scenarios are unavailable here, so a maintainer with private-source access must verify fixture fidelity and the claimed 379-scenario conformance pass.

No open issues 🎉

What changed in this PR

Adds optional, pinned CODA contract verification for the Account Overview advisory stack, keeping private upstream scenarios separate from ordinary builds and unit tests.

Changes:

  • Adds immutable checkout acquisition, integrity checks, and explicit private-runner safeguards.
  • Parses Gherkin scenarios and binds DX-001 through DX-024 to production evaluators, with local conformance reports.
  • Adds synthetic tooling regressions, contributor documentation, and private-file exclusions.
File Description
vitest.coda.config.ts Defines a separate, single-worker conformance suite.
tsconfig.vitest.json Includes the conformance test configuration in type checking.
tsconfig.eslint.json Includes the conformance configuration in linting scope.
src/​panels/​accountOverview/​services/​advisories/​conformance/​shared.ts Defines shared fixture state and assertion helpers.
src/​panels/​accountOverview/​services/​advisories/​conformance/​metricsFixture.ts Supplies synthetic Monitor responses without network access.
src/​panels/​accountOverview/​services/​advisories/​conformance/​features.ts Loads and compiles Gherkin scenarios.
src/​panels/​accountOverview/​services/​advisories/​conformance/​features.test.ts Tests parsing, binding execution, and rejection paths.
src/​panels/​accountOverview/​services/​advisories/​conformance/​coda.conformance.ts Executes selected contracts and writes scenario reports.
src/​panels/​accountOverview/​services/​advisories/​conformance/​bindings.ts Resolves steps and checks evaluator–registry agreement.
src/​panels/​accountOverview/​services/​advisories/​conformance/​bindingCoverage.test.ts Adds synthetic coverage across detector bindings.
src/​panels/​accountOverview/​services/​advisories/​conformance/​additionalFixtures.ts Maps scenario context steps to production-shaped fixtures.
src/​panels/​accountOverview/​services/​advisories/​conformance/​additionalBindings.ts Connects additional detectors, readers, and outcome assertions.
scripts/​coda.test.mjs Tests pin validation, acquisition, and cache integrity.
scripts/​coda.mjs Fetches verified revisions and launches guarded conformance runs.
package.json Adds the CODA pin, commands, dependencies, and tooling-test integration.
package-lock.json Locks the added Gherkin dependencies.
eslint.config.mjs Excludes private CODA files from ESLint.
CONTRIBUTING.md Documents conformance commands, scope, and privacy boundaries.
.vscodeignore Excludes private CODA files from VSIX packaging.
.oxlintrc.jsonc Excludes private CODA files from Oxlint.
.oxfmtrc.json Excludes private CODA files from formatting.
.gitignore Ignores private checkouts and conformance reports.

Review effort: Balanced

@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from f5165c7 to 38b53d7 Compare November 13, 2026 12:22
Copilot AI balanced review requested due to automatic review settings November 13, 2026 12:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The DX-011 direct evaluation omits scope capacity, causing a severity mismatch with the registered detector that rejects a valid scenario before its outcomes run.

1 open issue

Review effort: Balanced

@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from 38b53d7 to 09ace6d Compare November 13, 2026 13:33
Copilot AI balanced review requested due to automatic review settings November 13, 2026 13:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The private pinned CODA scenarios and original fixture bindings are unavailable, so an upstream reviewer must verify fixture fidelity and the claimed 379-scenario pass before contract conformance can be established.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch 2 times, most recently from 6d85435 to dc48ebd Compare November 13, 2026 14:34
Copilot AI balanced review requested due to automatic review settings November 13, 2026 14:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

An authorized reviewer must verify fixture and binding fidelity against the unavailable private pinned CODA contracts and confirm the claimed full 379-scenario run before relying on conformance.

No open issues 🎉

Review effort: Balanced

@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from c641f5c to 8906a5a Compare November 13, 2026 19:42
Copilot AI balanced review requested due to automatic review settings November 13, 2026 19:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The pinned private CODA corpus is unavailable, so an upstream reviewer must verify binding fidelity and the claimed 379-scenario pass, which synthetic regressions alone cannot establish.

No open issues 🎉

Review effort: Balanced

@bk201- Dmitrii Shilov (bk201-) changed the title [6/8] Add pinned CODA contract conformance tooling [6/9] Add pinned CODA contract conformance tooling Nov 18, 2026
@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from 8906a5a to 47409de Compare November 19, 2026 11:55
Copilot AI balanced review requested due to automatic review settings November 19, 2026 11:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

DX-011 ignores autoscale fixtures that use series instead of peaks, incorrectly reporting insufficient_data for a fixture with 10% utilization.

1 open issue

Review effort: Balanced

@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from 47409de to ce0a8bf Compare November 19, 2026 12:47
Copilot AI balanced review requested due to automatic review settings November 19, 2026 12:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The pinned private CODA contracts and feature files are unavailable, so a reviewer with private-source access must confirm binding and fixture fidelity and the claimed 379-scenario pass.

1 open issue

Review effort: Balanced

Add integrity-checked local source acquisition, Gherkin scenario bindings, synthetic tooling regressions, and private cache/report exclusions without fetching contracts during ordinary builds or tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the same relative materiality denominator as the registered DX-011 detector.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the pin, test command and scenario claims from PR two to the layer that actually adds them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Do not replace supplied upstream partition histories with synthesized peak samples.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 17:24
@bk201-
Dmitrii Shilov (bk201-) force-pushed the dev/dshilov/account-overview-06-coda-conformance branch from ce0a8bf to 18518f1 Compare November 19, 2026 17:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Binding fidelity and the claimed 379-scenario conformance pass require verification by a reviewer with authorized access to the unavailable pinned private CODA scenarios.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants