BE-817: atlas: push fit generations to S3

[indietyp]

🌟 What is the purpose of this PR?

Publish a finished fit to S3 so serving hosts can pull it. We must do this because fitting and serving are distinct: fitting runs occasionally, while serving must serve (pun intended) that data. To coordinate that, we use remote storage, and to support future deployments, we keep the backend generic (for now, only S3-compatible storage and local storage are supported) to push and pull generations.

This PR handles the upload half and infrastructure; BE-816 handles the pull (note that S3-based files are already supported via this PR).

The primary challenge in this PR is coordination, especially since S3 doesn't yet support multi-object commits. We therefore upload all files first, then metadata.json to complete the upload. Once done, we copy to active, then set current and cycle previous.

Directory layout:

 s3://<bucket>/<prefix>/
 └── generations/
     ├── current                      ← one line: the promoted <id>. Moves under If-Match on its ETag.
     ├── previous                     ← advisory, the <id> that current pointed at before. Written unconditionally.
     │
     ├── repository/                  ← every generation a fit UPLOADED, admitted or refused. Never pruned.
     │   ├── <id-A>/
     │   │   ├── coordinates.arr
     │   │   ├── representations.arr
     │   │   ├── knn.sprs
     │   │   ├── landmarks.lndm
     │   │   ├── classifier.clsf
     │   │   ├── … (every artifact the metadata lists)
     │   │   └── metadata.json        ← written LAST. Its digest is <id-A>; its presence means the prefix is complete.
     │   └── <id-B>/
     │       └── …
     │
     └── active/                      ← only PROMOTED generations, copied server-side from repository/<id>/.
         ├── <id-A>/                  ← the one current names
         │   ├── coordinates.arr
         │   ├── …
         │   └── metadata.json        ← again last, same completeness rule
         └── <id-Z>/                  ← an earlier promotion. Stays; nothing deletes it.
             └── …

For review, the properties we rely on are the following:

• current moves under the precondition captured at Upload::prepare, we hard error if current has been modified in any way, as it indicates either corruption or another process having written content.
• Conditional writes, copies and multipart creation make exactly one request attempt (S3::single_attempt): a retry after a lost response could report a false conflict or start a second multipart upload after losing the first one's identifier.
• A re-run after a failure reuses an existing object only after its bytes hash to the expected digest (finish_object → verify_destination); this makes sure that no partial uploads pollute the system (or someone else has written to the file in the meantime).
• We purposefully retain the raw bytes of any generation manifest, reason being that we rely on the checksum. Re-serialisation would risk the hash going out of sync, which could prematurely stop pushing or pulling.
• A failed multipart upload or copy aborts and awaits the abort before returning the original error.

🔍 What does this change?

flowchart LR
    F[fit] -->|seal| G[local generation]
    G -->|artifacts, then metadata.json| R["repository/<id>/"]
    R -->|copy after verify, metadata last| A["active/<id>/"]
    A -->|If-Match on captured ETag| C[current]
    C -.->|advisory| P[previous]

Loading

• file/storage/: FilePath (a filesystem path or an s3:// location), Storage (the optional S3 client and the scratch directory), WriteCondition and Revision, with one operation set over both backends. local/ is the lock-and-rename implementation, s3/ the SDK one, with multipart/ for objects over the single-request bound and path/ for bucket locations that keep the caller's literal key text.
• file/generation/upload/: Upload::prepare captures the current pointer and its revision, upload completes the repository prefix, promote copies into active/ and swaps current. GenerationUploadBackend is the trait the tests implement with a fault-injecting fake, and Storage implements it for real.
• file/generation/document/: GenerationDocument keeps the verified original bytes beside the parsed SaltRepository.
• CLI: S3Args (--s3, --s3-region, --s3-endpoint, --s3-force-path-style, an explicit key pair and session token, each with its HASH_GRAPH_ATLAS_S3_* variable and the rest from the SDK's own AWS_* chain) and --upload on fit. FitCommand::new takes the Storage and resolves remote inputs before the run.

⚠️ Breaking Changes & Known Issues

• We cannot know what the remote state is after a transport issue, and the upload errors out at that point. On a re-run every object write is conditional on the object being absent, so an object that did land rejects the write and we verify its bytes instead of uploading them again.
• We currently do not rely on the previous pointer, as it's advisory.
• Nothing prunes repository/. S3 bucket policies must be used to clean up
• We currently do not delete old active generations, tracked in BE-852

❓ How to test this?

1. yarn compose up -d minio, then create a bucket (the compose credentials are dev-s3-access-key-id / dev-s3-secret-access-key).
2. Run a fit with --s3 --s3-endpoint http://localhost:9000 --s3-force-path-style --s3-region us-east-1 --s3-access-key-id dev-s3-access-key-id --s3-secret-access-key dev-s3-secret-access-key --upload s3://<bucket>/atlas.
3. List s3://<bucket>/atlas/generations/: repository/<id>/ holds the artifacts with metadata.json last by modification time, active/<id>/ exists only if the verdict says activated true, and current reads the id.
4. Run a second fit against the same prefix: current moves to the new id under the ETag the first run left, and previous reads the old one.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 17, 2026 1:30pm UTC
petrinaut	Ready	Preview	Sep 17, 2026 1:30pm UTC
petrinaut-docs	Ready	Preview	Sep 17, 2026 1:30pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Sep 17, 2026 1:30pm UTC

[github-actions]

Dependency Review

The following issues were found:

• ❌ 1 vulnerable package(s)
• ✅ 0 package(s) with incompatible licenses
• ✅ 0 package(s) with invalid SPDX license definitions
• ✅ 0 package(s) with unknown licenses.

See the Details below.

Vulnerabilities

Cargo.lock

Name	Version	Vulnerability	Severity	Patched Version
rustls-webpki	0.101.7	rustls-webpki: Denial of service via panic on malformed CRL BIT STRING	high	N/A
		webpki: Name constraints for URI names were incorrectly accepted	low	N/A
		webpki: Name constraints were accepted for certificates asserting a wildcard name	low	N/A

OpenSSF Scorecard

Package	Version	Score	Details
cargo/rustls-webpki	0.101.7	Unknown	Unknown
cargo/h2	0.3.27	🟢 6.2	Details Check Score Reason Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Code-Review 🟢 4 Found 13/29 approved changesets -- score normalized to 4 Maintained 🟢 10 30 commit(s) and 5 issue activity found in the last 90 days -- score normalized to 10 Packaging ⚠️ -1 packaging workflow not detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Binary-Artifacts 🟢 10 no binaries found in the repo CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 Fuzzing 🟢 10 project is fuzzed License 🟢 10 license file detected Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md Security-Policy 🟢 9 security policy file detected SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
cargo/hyper	0.14.32	🟢 5.9	Details Check Score Reason Security-Policy 🟢 10 security policy file detected Binary-Artifacts 🟢 10 no binaries found in the repo Maintained 🟢 10 30 commit(s) and 8 issue activity found in the last 90 days -- score normalized to 10 Packaging ⚠️ -1 packaging workflow not detected Code-Review 🟢 8 Found 26/30 approved changesets -- score normalized to 8 Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing ⚠️ 0 project is not fuzzed Signed-Releases ⚠️ -1 no releases found Branch-Protection ⚠️ -1 internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
cargo/hyper-rustls	0.24.2	Unknown	Unknown
cargo/rustls	0.21.12	Unknown	Unknown
cargo/sct	0.7.1	Unknown	Unknown
cargo/socket2	0.5.10	🟢 5.4	Details Check Score Reason Maintained 🟢 5 5 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 5 Code-Review 🟢 10 all changesets reviewed Dangerous-Workflow 🟢 10 no dangerous workflow patterns detected Binary-Artifacts 🟢 10 no binaries found in the repo Packaging ⚠️ -1 packaging workflow not detected Token-Permissions ⚠️ 0 detected GitHub workflow tokens with excessive permissions Pinned-Dependencies ⚠️ 0 dependency not pinned by hash detected -- score normalized to 0 CII-Best-Practices ⚠️ 0 no effort to earn an OpenSSF best practices badge detected License 🟢 10 license file detected Fuzzing ⚠️ 0 project is not fuzzed Signed-Releases ⚠️ -1 no releases found Security-Policy 🟢 10 security policy file detected Branch-Protection 🟢 4 branch protection is not maximal on development and all release branches SAST ⚠️ 0 SAST tool is not run on all commits -- score normalized to 0
cargo/tokio-rustls	0.24.1	Unknown	Unknown

Scanned Files

• Cargo.lock

[codspeed]

Merging this PR will not alter performance

⚠️ 6 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

✅ 98 untouched benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
⚠️	as_constant	< 1 ns	< 1 ns	N/A
⚠️	constant_equal	< 1 ns	< 1 ns	N/A
⚠️	constant_not_equal	< 1 ns	< 1 ns	N/A
⚠️	access	< 1 ns	< 1 ns	N/A
⚠️	runtime_equal	< 1 ns	< 1 ns	N/A
⚠️	runtime_not_equal	< 1 ns	< 1 ns	N/A

---

_{Comparing bm/be-817-atlas-push-fit-generations-to-s3 (eef86dc) with bm/be-815-atlas-serve-more-than-one-generation-at-a-time (f455d38)¹}

Footnotes

1. No successful run was found on bm/be-815-atlas-serve-more-than-one-generation-at-a-time (a055b7c) during the generation of this report, so 04425d8 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[codecov]

Codecov Report

❌ Patch coverage is 46.77419% with 627 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.66%. Comparing base (a055b7c) to head (eef86dc).

Files with missing lines	Patch %	Lines
libs/@local/graph/atlas/src/file/storage/s3/mod.rs	6.80%	137 Missing ⚠️
...aph/atlas/src/file/storage/s3/multipart/backend.rs	0.00%	91 Missing ⚠️
libs/@local/graph/atlas/src/cli/fit.rs	0.00%	80 Missing ⚠️
libs/@local/graph/atlas/src/cli/shell.rs	0.00%	58 Missing ⚠️
libs/@local/graph/atlas/src/cli/s3.rs	0.00%	53 Missing ⚠️
...al/graph/atlas/src/file/generation/upload/error.rs	18.36%	40 Missing ⚠️
...s/@local/graph/atlas/src/file/storage/error/mod.rs	40.74%	32 Missing ⚠️
.../graph/atlas/src/file/generation/upload/backend.rs	0.00%	30 Missing ⚠️
libs/@local/graph/atlas/src/file/storage/mod.rs	55.81%	19 Missing ⚠️
...bs/@local/graph/atlas/src/file/storage/path/mod.rs	82.97%	16 Missing ⚠️
... and 11 more

Additional details and impacted files

@@                                     Coverage Diff                                      @@
##           bm/be-815-atlas-serve-more-than-one-generation-at-a-time    #9681      +/-   ##
============================================================================================
- Coverage                                                     66.76%   66.66%   -0.11%     
============================================================================================
  Files                                                          1821     1838      +17     
  Lines                                                        192552   193639    +1087     
  Branches                                                       7879     7919      +40     
============================================================================================
+ Hits                                                         128553   129083     +530     
- Misses                                                        62489    63041     +552     
- Partials                                                       1510     1515       +5     

Flag	Coverage Δ
apps.hash-graph	12.40% <0.00%> (-0.05%)	⬇️
rust.hash-graph-atlas	81.38% <47.09%> (-0.68%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit b32a808. Configure here.}