Skip to content

Support machine/project-prefixed PACE archive snapshots - #415

Merged
tomvothecoder merged 2 commits into
E3SM-Project:mainfrom
tomvothecoder:fix/414-pace-archive-snapshots
Oct 22, 2026
Merged

tomvothecoder merged 2 commits into
E3SM-Project:mainfrom
tomvothecoder:fix/414-pace-archive-snapshots

Conversation

@tomvothecoder

Copy link
Copy Markdown
Collaborator

Description

  • Closes [DevOps]: Support machine/project-prefixed PACE archive snapshots #414.
  • Share snapshot-name validation between archive scanning and the API.
  • Accept timestamp-only and machine/project-prefixed PACE snapshots, including underscores in prefix values.
  • Extend regression coverage for enumeration, references, pruning, case identity, and checkpoint persistence.

Checklist

  • Code follows project style guidelines
  • Self-reviewed code
  • No new warnings
  • Tests added or updated (if needed)
  • All tests pass (locally and CI/CD)
  • Documentation/comments updated (if needed)
  • Breaking change noted (if applicable): none

Validation

  • 233 isolated schema/discovery/checkpoint/client/workflow tests passed.
  • 190 broader archive-ingestor/configuration tests passed.
  • Isolated runs used --noconftest with dummy test settings; database-backed API tests remain unverified.
  • Changed-file pre-commit hooks passed, including Ruff and type checking.
  • make backend-test blocked by missing .envs/local/backend.env.
  • Full make pre-commit-run passed backend hooks; frontend hooks failed because pnpm is unavailable.
  • Draft opened with these validation gaps explicitly accepted by the requester; CI results remain pending.

Deployment Notes (if any)

Deploy updated API validation and the on-site ingestion checkout before running Chrysalis archive ingestion in apply mode. Verify settled snapshot checkpoints are recorded. No database migration required.

Remaining risks: database-backed API persistence and live Chrysalis checkpoint recording still require validation.

@tomvothecoder
tomvothecoder requested a lite review from Copilot October 22, 2026 19:12
@tomvothecoder tomvothecoder added this to the 1. FY27Q1: Beta Launch milestone Oct 22, 2026
@tomvothecoder
tomvothecoder marked this pull request as ready for review October 22, 2026 19:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Update operator and architecture documentation to describe the newly supported prefixed snapshots before approval.

1 open issue
What changed in this PR

This pull request adds support for timestamp-only and machine/project-prefixed PACE archive snapshots, including prefixes containing underscores, across archive discovery and API checkpoint validation.

Changes:

  • Shares snapshot-name validation between API schemas and archive scanning.
  • Extends discovery, pruning, references, case identity, and checkpoint handling.
  • Adds regression coverage for valid and malformed snapshot names and persistence flows.
File Description
backend/​tests/​features/​ingestion/​test_schemas.py Tests accepted and malformed snapshot-name formats.
backend/​tests/​features/​ingestion/​test_archive_source_directories.py Covers prefixed snapshots in source mappings and checkpoint handling.
backend/​tests/​features/​ingestion/​test_archive_discovery.py Verifies prefixed snapshot discovery, pruning, references, and deduplication.
backend/​tests/​features/​ingestion/​test_api.py Verifies checkpoint persistence and range-filtered retrieval.
backend/​app/​scripts/​ingestion/​archive_layout.py Reuses shared validation during archive directory enumeration.
backend/​app/​features/​ingestion/​schemas.py Defines shared validation for timestamp-only and prefixed snapshots.

Review effort: Lite

Comment thread backend/app/features/ingestion/schemas.py
@tomvothecoder

Copy link
Copy Markdown
Collaborator Author

Chrysalis archive full re-run to establish checkpoints

Status: In Progress
After run completes, look at SQL query to see if checkpoints are populated

SELECT
    m.name AS machine,
    (
        SELECT COUNT(*)
        FROM cases AS c
        WHERE c.machine_id = m.id
    ) AS cases,
    (
        SELECT COUNT(*)
        FROM executions AS e
        JOIN cases AS c ON c.id = e.case_id
        WHERE c.machine_id = m.id
    ) AS executions,
    (
        SELECT COUNT(*)
        FROM ingestions AS i
        WHERE i.machine_id = m.id
    ) AS ingestions,
    (
        SELECT COUNT(*)
        FROM execution_discovery_results AS d
        WHERE d.machine_id = m.id
    ) AS discovery_results,
    (
        SELECT COUNT(*)
        FROM archive_scan_checkpoints AS a
        WHERE a.machine_id = m.id
    ) AS archive_checkpoints
FROM machines AS m

@tomvothecoder
tomvothecoder merged commit 96f49c4 into E3SM-Project:main Oct 22, 2026
1 check passed
@tomvothecoder
tomvothecoder deleted the fix/414-pace-archive-snapshots branch October 22, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DevOps]: Support machine/project-prefixed PACE archive snapshots

2 participants