A small distributed file system that keeps file metadata consistent across
a cluster of nodes using the Raft consensus algorithm — no external
database. It is inspired by how Kubernetes' etcd manages cluster state: the
replicated Raft log is the source of truth.
Built to study consensus, replication, persistence, and failure handling in distributed storage. Scope is deliberately narrow and everything claimed below is implemented and exercised by a runnable demo.
DataVault is a compact object storage engine for small, strongly-consistent clusters — and a reference implementation for understanding how modern distributed storage works. It is not a MinIO / Ceph / SeaweedFS replacement, and it doesn't try to be. It competes on clarity, not scale or raw performance: small enough to read end-to-end, architecturally rigorous, and able to show you why distributed storage behaves the way it does — you can literally watch streaming → Raft commit → repair happen in the live console.
The self-hosted object-storage landscape shifted in 2025–2026: MinIO's community edition was archived/read-only (Apr 2026), and cloud repatriation + data sovereignty (e.g. the EU Data Act, effective Sept 2025) are pushing teams toward storage that runs on a handful of commodity nodes, survives node loss, and never phones home to a cloud. That leaves room for small, understandable, strongly consistent engines — a different niche from the hyperscale incumbents, and the reason this architecture is worth studying now.
| DataVault primitive (built, real) | What it provides |
|---|---|
| Raft-backed metadata layer, no external DB | strongly-consistent object index that runs itself |
| Content-addressed blobs (SHA-256) | dedup + integrity + immutability |
| Quorum write, durability-before-visibility | no lost or dangling objects |
| Replica repair / anti-entropy | survive node loss, converge back to full replication |
| Encryption at rest | data-sovereignty / compliance building block |
| Automatic failover | HA on commodity/edge hardware |
| Live observability console | watch streaming → Raft commit → repair happen |
The core storage primitives are implemented; higher-level object-storage APIs (S3 verbs, versioning, multipart, IAM, lifecycle) and operational capabilities remain future work. Storage isn't additive — one missing subsystem can be half the engineering — so this is described by what exists, not by a percentage.
- Path A (recommended): an S3-compatible API on top of the Raft-backed engine. The interesting part is the engine; S3 is just the interface that lets existing tools talk to it. Aimed at small edge/sovereign clusters that want strong metadata consistency — not hyperscale.
- Path B: a content-addressed artifact store for ML — versioned datasets/models with dedup and integrity verification. Content addressing is already the core primitive.
Its real differentiator is legibility + observability: half functional storage engine, half educational reference that explains itself.
Known ceilings (stated plainly): a single Raft group bounds metadata scale (fine for small/edge clusters); full replication is 3× storage vs. erasure coding's ~1.5×; not benchmarked at scale or security-hardened. See DESIGN.md.
Context: MinIO EOL — It's FOSS, MinIO archived — Storm Developments, self-hosted alternatives — OSH, 2026 cloud repatriation — Compare the Cloud, data sovereignty + edge — Edge Industry Review
- Raft consensus via
hashicorp/raft(the same library behind Consul, Nomad, and Vault) — leader election, log replication, and majority commit. - Metadata/blob split. The Raft log carries small metadata records (key, content hash, size, owner, version), not file bytes. Replicating multi-GB blobs through the consensus log would bloat every replication round, election replay, and snapshot.
- Content-addressed blob storage with a quorum write path. On upload the leader stages the blob (SHA-256, fsync'd), replicates it to peers, waits for a write quorum (2/3) of durable + checksum-verified ACKs, and only then commits the metadata through Raft. Durability precedes visibility, so committed metadata can never point at a blob that isn't durable. See DESIGN.md.
- Replica Repair (anti-entropy). A leaderless per-node background loop pulls every committed blob the node is missing (verifying checksums), so a node that was down catches up automatically and the cluster converges to full replication. It also garbage-collects orphaned blobs (abandoned staged uploads past their lease, and committed blobs no live file references). Read-repair on download is the synchronous fast path on top of this.
- Encryption at rest. With
-encryption-key, blobs are stored AES-256-CTR encrypted (random IV per blob). The content hash is over plaintext, so dedup and cross-node verification still work; integrity comes from that hash check. - Persistence (WAL). The log and stable state are on disk in BoltDB. A write is not acknowledged until it is durable.
- Snapshots. The state machine can snapshot and restore, so a node recovers from a snapshot plus recent log entries instead of replaying the whole history.
- Automatic failover + recovery. Kill the leader and a surviving node is elected; a killed node that restarts rejoins and self-heals its blobs.
- Observability.
/cluster/status(leadership, indices, per-file replica health) and/metrics(repairs, orphans collected, under-replication, blob counts). CORS and optional bearer-token auth make it browser/dashboard ready.
- Reads are linearizable by default — metadata reads go to the leader (a
follower proxies to it);
?consistency=eventualopts into a local, possibly stale read. Blob bytes are content-addressed and immutable, so they're served from any replica once the authoritative hash is known. - CP, not AP. With 3 nodes DataVault tolerates one failure; lose the majority and it refuses writes rather than risk divergence. That's the intended consequence of Raft (one history, no split brain), not a shortcoming.
Honest scope, not a feature list:
- TLS/mTLS between nodes. Blobs travel between nodes as plaintext and the
inter-node
/internal/endpoints are unauthenticated — they assume a trusted private network. Put them behind TLS before untrusted use. - Only the latest version's blob is retained (old hashes are GC'd); full version history would need metadata to keep a hash list.
- Dynamic membership beyond join (no runtime remove/rebalance).
- RBAC roles, chunking, configurable replication factor.
There are no "AI", "blockchain", "post-quantum", "Byzantine fault tolerance", or availability-percentage claims here. Raft assumes nodes fail by crashing, not by acting maliciously — so this is crash-fault-tolerant, not Byzantine-fault-tolerant.
# Terminal 1 — bootstrap the first node
make node1
# Terminal 2 & 3 — join the cluster
make node2
make node3
# Check who is leader
make statusAutomated proofs (each is self-contained: builds, starts a cluster, asserts, cleans up):
./scripts/demo_blobs.sh # file upload → quorum replication → download → failover
./scripts/demo_full.sh # the whole backend: encrypted-at-rest cluster, quorum
# write, kill+restart a node → background self-heal,
# leader failover, /cluster/status + /metrics# upload a real file (leader only; followers return the leader's address).
# leader stages -> replicates -> waits for write quorum -> commits metadata.
curl -XPUT '127.0.0.1:8001/files/report.pdf?owner=aditya' --data-binary @report.pdf
# download from ANY node (read-repairs locally if this node lacks the blob)
curl 127.0.0.1:8003/files/report.pdf/content -o report.pdf
curl 127.0.0.1:8001/files/report.pdf # one file's metadata
curl 127.0.0.1:8001/files # list live files
curl -XDELETE 127.0.0.1:8001/files/report.pdf
curl 127.0.0.1:8001/status # basic: state, leader, quorum, peers
curl 127.0.0.1:8001/cluster/status # detailed: indices + per-file replica health
curl 127.0.0.1:8001/metrics # repairs, orphans, under-replication, blob countsA native-feeling operations console lives in web/ — animated cluster
topology (the crown moves on failover), drag-and-drop upload with a staged
pipeline, live metrics, and an activity timeline. Run a cluster, then:
cd web && npm install && npm run dev # http://localhost:5173See web/README.md for the interview demo flow.
- CORS is enabled on every endpoint (
-allow-origin, default*), so a browser dashboard can call the API directly. - Optional auth: start nodes with
-api-token <t>and sendAuthorization: Bearer <t>on client requests (inter-node/internal/endpoints are exempt). - The dashboard's data comes from
GET /cluster/status(topology + per-file replica health) andGET /metrics(counters) — poll them on an interval. - Encryption at rest: pass the same
-encryption-key <passphrase>to every node.
- Consistency model: writes go through the leader and commit only after a
quorum persists the log entry. Reads are linearizable by default (routed to
the leader);
?consistency=eventualallows a local follower read. - Why metadata-only in the log: see
internal/fsm/fsm.go. Blobs are streamed to replicas out of band (internal/api/internal/blobs/), and the metadata commit references only the content hash. - Durability before visibility: the write path waits for a write quorum of durable blob ACKs before proposing the Raft entry, so committed metadata can never point at a missing blob. See DESIGN.md.
- Recovery + convergence: BoltDB WAL + Raft snapshots for the metadata; the
per-node repairer (
internal/repair) converges blobs to full replication.
cmd/datavault entry point + flags
internal/fsm the replicated metadata state machine + node registry (+ tests)
internal/blob content-addressed blob store: stage/commit, fsync, checksum, AES-at-rest (+ tests)
internal/repair Replica Repair (anti-entropy): heal + garbage collection (+ tests)
internal/cluster Raft setup: transport, WAL, snapshots, join, registry, repairer wiring
internal/api HTTP API: membership, metadata, upload/download, status/metrics, CORS/auth
scripts/demo_blobs.sh upload/quorum-replication/download/failover proof (real files)
scripts/demo_full.sh full backend E2E incl. encryption, node restart + self-heal, metrics
scripts/check_api.sh integration check of the endpoints + CORS the console uses
web/ the operations console (Vite + React + Tailwind + Framer Motion)
The correctness contract these phases uphold is in DESIGN.md — read that first; the invariants come before the implementation.
- ✅ Raft foundation: election, log replication, WAL, snapshots, failover.
- ✅ Metadata operations through the log (put / delete / versioning).
- ✅ Blob replication, content-addressed. Write path: stage → write quorum (2/3) durable + checksum-verified ACK → commit metadata via Raft.
- ✅ Replica Repair (anti-entropy): heal missing blobs, self-heal restarted nodes, GC orphaned + expired-staged blobs.
- ✅ Observability:
/cluster/status+/metrics. - ✅ Encryption at rest (AES-256-CTR); CORS + optional bearer-token auth.
- ✅ Recovery: a killed node restarts, rejoins, and self-heals (
demo_full.sh). - ⬜ Chunking: split large blobs so repair transfers only missing chunks.
- ⬜ Wire the dashboard to
/cluster/status+/metrics. - ⬜ TLS/mTLS between nodes; live-inventory replica counts.
Stretch / explicit non-goal for now: dynamic membership (joint consensus), Byzantine fault tolerance.