Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 16 additions & 9 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,13 +140,15 @@ one-way door and a restart of vault-unseal an unrecoverable one.
| Profile | Provisioning | Seal | Proven by |
|---|---|---|---|
| local/CI | docker-compose | Vault Transit (`vault-unseal`) | integration suite, every PR |
| AWS | `terraform/aws` | AWS KMS | `terraform test` (mocked), an emulated apply, and two real applies (2026-09-17, 2026-09-24) — partly proven, see `docs/cloud-apply.md` |
| Azure | `terraform/azure` | Azure Key Vault | `terraform test` (mocked) and one real apply (2026-09-28) — partly proven, see `docs/cloud-apply.md` |
| AWS | `terraform/aws` | AWS KMS | `terraform test` (mocked), an emulated apply, and four real applies (2026-09-17, 09-24, and twice on 09-29) — checklist items 1-10 all observed, see `docs/cloud-apply.md` |
| Azure | `terraform/azure` | Azure Key Vault | `terraform test` (mocked) and two real applies (2026-09-28, 09-29) — items 6, 7 and audit observed; the PKI migration is not, see `docs/cloud-apply.md` |
| bare/other | Ansible alone | Shamir (role default) | not exercised |

**`terraform/aws` has been applied to a real account twice, on
2026-09-17 and 2026-09-24, and `terraform/azure` once, on 2026-09-28.**
Do not describe either as working or proven. `docs/cloud-apply.md`
**`terraform/aws` has been applied to a real account four times
(2026-09-17, 09-24, and twice on 09-29) and `terraform/azure` twice
(2026-09-28, 09-29).** The AWS verification checklist is complete; that
is not the same as the profile being proven, and no identity narrower
than an administrator has driven any of it. `docs/cloud-apply.md`
records what each session settled — auto-unseal, peer discovery, health
checks, the handoff, a restore — what it observed failing, and what it
never reached. The AWS sessions' headline finding was that **the cluster
Expand All @@ -156,10 +158,15 @@ instance id that does not exist until launch. A replacement now signs its
own leaf at boot from a bootstrap CA published to SSM
(`scripts/issue-bootstrap-cert.sh`), and on 2026-09-24 a second apply
watched that work on a real node — blocker 1 is closed, and an instance
refresh kept quorum across all three nodes. What neither session reached
is listed in `docs/cloud-apply.md`: snapshots to the bucket, a restore,
PKI and audit on a real node, and any identity narrower than an
administrator.
refresh kept quorum across all three nodes.

The 2026-09-29 sessions closed the rest: snapshots to the bucket, a
restore checked four ways, audit devices, the PKI migration, and the
backup chain end to end — the object the timer wrote, downloaded,
decrypted through KMS and restored. Across all four sessions **almost no
defect was a crash**; they were silent narrowings and features that had
never worked. Expect that shape. What remains unreached is any identity
narrower than an administrator.

**The Azure apply found nineteen defects, and the profile could not have
applied as written** — not on that subscription, not on any. None was
Expand Down
15 changes: 13 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -147,10 +147,10 @@ rotate-secret-id: ## Issue a fresh AppRole secret_id (ROLE=name)
##@ Tests

.PHONY: test
test: test-fast ## Every suite that needs no cluster (seconds)
test: test-fast ## Every suite that needs no cluster (a few minutes)

.PHONY: test-fast
test-fast: ## Shim suites only — no Docker, no credentials
test-fast: ## Shim suites only — no Docker, no credentials (~2-3 min)
@fail=0; for s in $(FAST_SUITES); do \
printf '\n\033[36m=== %s ===\033[0m\n' "$$s"; \
./tests/$$s/run-tests.sh || fail=1; \
Expand All @@ -170,6 +170,12 @@ test-cluster: ## The integration suite against a real 3-node cluster (minutes)
.PHONY: test-all
test-all: test-fast test-cluster ## Everything, including the real cluster

.PHONY: dr-drill-cloud
dr-drill-cloud: ## The restore drill against a real cloud cluster (CLOUD=aws|azure)
@[ -n "$(CLOUD)" ] || { echo "usage: make dr-drill-cloud CLOUD=aws|azure"; exit 2; }
@echo "Needs VAULT_TOKEN, and a cluster you do not mind restoring."
./scripts/dr-drill-cloud.sh --cloud $(CLOUD)

##@ Cloud, without spending anything

.PHONY: tf-test
Expand Down Expand Up @@ -199,6 +205,11 @@ preflight-cloud: ## Pre-apply checks needing credentials (CLOUD=aws|azure)
@[ -n "$(CLOUD)" ] || { echo "usage: make preflight-cloud CLOUD=aws|azure"; exit 2; }
./scripts/preflight-cloud.sh --cloud $(CLOUD)

.PHONY: teardown-cloud
teardown-cloud: ## Remove what terraform destroy cannot (CLOUD=aws|azure)
@[ -n "$(CLOUD)" ] || { echo "usage: make teardown-cloud CLOUD=aws|azure"; exit 2; }
./scripts/teardown-cloud.sh --cloud $(CLOUD)

##@ Lint — these mirror CI exactly

.PHONY: fmt
Expand Down
66 changes: 46 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,14 @@ that part down.
network, autoscaling group, load balancer, KMS auto-unseal, snapshot
bucket. `terraform/azure` builds the same shape with a VM scale set,
Key Vault auto-unseal, and a blob container. The AWS profile applies
and destroys cleanly against an emulated AWS API on every PR, and was
applied to a real account once, on 2026-09-17 — which found ten
defects; a second on 2026-09-24 watched a replacement node heal
itself. `terraform/azure` was applied once, on 2026-09-28, which found
nineteen more — it could not have applied as written, on any
subscription. See [`docs/cloud-apply.md`](docs/cloud-apply.md) for what
those sessions settled and what they did not.
and destroys cleanly against an emulated AWS API on every PR, and has
been applied to a real account four times — 2026-09-17, 09-24 and twice
on 09-29 — which between them worked through the whole verification
checklist. `terraform/azure` was applied twice, on 2026-09-28 and
09-29; the first found nineteen defects and could not have applied as
written, on any subscription. See
[`docs/cloud-apply.md`](docs/cloud-apply.md) for what those sessions
settled and what they did not.
- **HA by default** — the reference topology is a multi-node Raft cluster
behind a load balancer from the start, not bolted on as a "v2" feature.
- **Operable, not just deployable** — runbooks and disaster-recovery
Expand Down Expand Up @@ -244,8 +245,9 @@ See [`docs/deployment.md`](docs/deployment.md).

## Before a cloud apply

`terraform/aws` has been applied to a real account twice, on 2026-09-17
and 2026-09-24; `terraform/azure` once, on 2026-09-28. The emulated apply
`terraform/aws` has been applied to a real account four times, on
2026-09-17, 09-24 and twice on 09-29; `terraform/azure` twice, on
2026-09-28 and 09-29. The emulated apply
in CI settles that the AWS configuration is one the API accepts; it says
nothing about whether the cluster it describes comes up, which is what
those sessions were for — and what they found is in
Expand Down Expand Up @@ -459,20 +461,36 @@ feature breaks, not that the code exists.

What stands between here and v1.0, in order:

1. **A real AWS apply.** Done twice, 2026-09-17 and 2026-09-24, and
**closed** by the second. Settled: the instance profile, the KMS key
1. **A real AWS apply.** Done four times — 2026-09-17, 09-24 and twice
on 09-29 — and **closed**. Settled: the instance profile, the KMS key
policy and the `seal` stanza agree, peers find each other by tag, the
load balancer keeps standbys in the pool, the Ansible handoff works, a
snapshot restores under the KMS seal, and — the one that kept this
item open — **a terminated leader's replacement now signs its own
certificate at boot, auto-unseals and rejoins Raft with nobody
touching it**, watched on a real node. An instance refresh was watched
too, keeping quorum across all three nodes. Thirteen defects were
fixed between the two sessions, seven of them in code no test here
could reach. Still never reached: snapshots to the bucket, PKI and
audit on a real node, and any identity narrower than an administrator.
See [`docs/cloud-apply.md`](docs/cloud-apply.md).
2. **A real Azure apply.** Done once, 2026-09-28, and **closed**.
too, keeping quorum across all three nodes.

The 2026-09-29 sessions finished the checklist. **Snapshots reach the
bucket** under the instance role; **a restore passes four ways** under
the KMS seal, including the post-snapshot token check that separates a
restore from a merge; **audit devices** come up cluster-wide with a
hashed trail; and **the PKI migration** moved all three nodes onto
Vault-issued certificates without the cluster losing a voter. The
object the timer wrote was also downloaded, decrypted through KMS and
restored — the whole backup chain, which only an emulator had ever
answered for.

Roughly thirty defects across the four sessions, and the pattern is
worth more than the count: **almost none was a crash.** They were
silent narrowings and features that had never worked — a `--force`
flag that could not reconfigure anything, a migration that asked its
own PKI role for a name the role refuses, a health check that trusted
one CA during the phase whose purpose is trusting two. Still never
reached: any identity narrower than an administrator. See
[`docs/cloud-apply.md`](docs/cloud-apply.md).
2. **A real Azure apply.** Done twice, 2026-09-28 and 09-29, and
**closed**.
Settled: **auto-unseal through Key Vault**, **a replacement node
signing its own certificate** from a CA it fetched with its own
managed identity, **Raft discovery by scale-set enumeration**, and
Expand All @@ -487,9 +505,17 @@ What stands between here and v1.0, in order:
all: the storage account and its customer-managed key undid each
other on alternate applies, moving snapshot encryption between Key
Vault and Microsoft-managed keys while everything reported success.
Still never reached: snapshots to the container, a restore, PKI and
audit on a real node, and any identity narrower than an
administrator. See [`docs/cloud-apply.md`](docs/cloud-apply.md).

A second session on 2026-09-29 settled snapshots into the container, a
restore under the Key Vault seal, and audit devices — the first time
any of the three had been reached on either cloud. Getting there took
three systemd fixes that are not about snapshots or audit at all: the
packaged unit leaves `/etc` read-only, so Vault cannot open its own
audit log, and `ProtectHome=true` stops the Vault CLI before it
contacts Vault, because it opens `$HOME/.vault` ahead of any
subcommand. Still never reached: the PKI migration on Azure, and any
identity narrower than an administrator. See
[`docs/cloud-apply.md`](docs/cloud-apply.md).
3. **Off-host audit shipping.** The audit trail now outlives the node,
an edit to it is detectable, and the anchors that make it detectable
now leave the machine. `scripts/ship-anchors.sh` writes each one to an
Expand Down
8 changes: 6 additions & 2 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,12 @@ and which parts are a plausible-looking configuration nobody has run.

## The honest gap

**`terraform/aws` has been applied to a real account twice, on
2026-09-17 and 2026-09-24, and `terraform/azure` once, on 2026-09-28.**
**`terraform/aws` has been applied to a real account four times, on
2026-09-17, 09-24 and twice on 09-29; `terraform/azure` twice, on
2026-09-28 and 09-29.** The AWS verification checklist is complete. That
is not the same as the profile being proven: no identity narrower than an
administrator has driven any of it, and roughly thirty defects across
those sessions say what a fifth one would still be for.

`terraform/aws` and `terraform/azure` are covered by `terraform test`
against mocked providers, and that catches more than it might sound like
Expand Down
Loading