ZeroFS serves S3-compatible buckets as POSIX filesystems over NFS and 9P, and as raw block devices over NBD. All three servers, plus a web UI, run in a single Rust binary. Everything — file data and metadata — lives on the object store. There's no Postgres, no etcd, no separate metadata service to keep alive.
Why I starred it
"Filesystem on S3" projects are common, and most of them cheat somewhere: they need a database for metadata, they choke on small files, or they quietly drop POSIX semantics like hardlinks and fsync. ZeroFS's README leads with a claim that's easy to check and hard to fake: it runs pjdfstest (8,662 POSIX conformance cases), xfstests, a Jepsen-style consistency checker, and full Linux kernel builds over its own mounts, in CI, per protocol. A project willing to run a kernel make -j$(nproc) against its own filesystem as a smoke test is a project confident in its metadata layer. I wanted to see what that layer actually looks like.
How it works
File contents are chopped into 32 KiB extents (EXTENT_SIZE in zerofs/src/fs/mod.rs:74). Each extent gets compressed and AEAD-encrypted individually, then packed as a length-prefixed frame into an immutable segment object up to 256 MiB. Metadata — inodes, directory entries, and one 32-byte pointer per extent — lives in an LSM-tree on the same bucket.
That LSM tree isn't hand-rolled. ZeroFS depends on a fork of SlateDB (zerofs/Cargo.toml:73), pinned to a specific commit, built with wal_disable and foyer caching. SlateDB is itself an object-storage-native LSM engine; ZeroFS's fork drops the write-ahead log because segment objects and the metadata SST already give it the durability boundary it needs — a WAL would be redundant I/O.
The key layout is where the design gets interesting. zerofs/src/fs/key_codec.rs prefixes every key with a domain tag, b"meta" or b"extent", before the kind byte:
// Kind byte assignments (one byte each):
// 0x01 INODE hot metadata, point-keyed by inode_id
// 0x02 DIR_ENTRY hot metadata, lookup by (dir_id, name)
// 0x03 DIR_SCAN hot metadata, ordered scan by (dir_id, cookie)
// ...
// 0xFE EXTENT bulk file data — the only kind in the extent segment
Metadata and bulk extent pointers land in two independent LSM segments via SlateDB's segment extractor, so a directory listing never shares an L0 list or compaction lifecycle with terabytes of extent-pointer churn. The comment in the file spells out the adjacency reasoning too: INODE (0x01) and DIR_ENTRY (0x02) sit next to each other on purpose, because a lookup() touches both, and adjacent kind bytes land in neighboring SST blocks — so the read reuses the same block-cache and filter entries. That's a metadata layout tuned against an actual access pattern, not just alphabetized.
Segment IDs are (epoch, counter) pairs, and the object key shards on the counter's low byte (segments/{shard:02x}/{epoch:016x}/{counter:016x} in zerofs/src/segment.rs) specifically so sequential segment writes round-robin across 256 S3 prefixes instead of hot-spotting one partition — reads stay exact-key, so the sharding costs nothing on the read path.
Encryption is XChaCha20-Poly1305, with the data key wrapped via Argon2id. zerofs/src/secrets.rs goes further than most projects bother to: key material and password buffers live in mlock'd, core-dump-excluded memory that's zeroed on drop, via a custom LockedAllocation/LockedBuf. The comment is refreshingly honest about the limits of this: HKDF and Argon2 still touch key-derived bytes on the regular stack inside their own crate internals, which isn't locked or scrubbed. They protect the boundary they control and say plainly where the guarantee stops, instead of overselling it.
One file that surprised me: dedup.rs. It's not extent deduplication — it's an idempotency ledger for NFS/9P retries. Every non-idempotent mutation (create, write, setattr, rename) can carry a client-generated 16-byte op ID; ZeroFS remembers the result for MUTATION_RESULT_RETENTION and replays it verbatim on a retried request, so a client retry after a dropped ACK can't double-create a file or double-append a write. That's the kind of correctness detail that only shows up once you've been burned by NFS's at-least-once semantics in production.
For durability under failover, zerofs/src/replication/ implements leader/standby replication over the same bucket — no second copy of the data. zerofs/src/fs/write_coordinator.rs batches commits (transactions, inode allocation, segment counters, replication records, dedup results) through a single ordered worker, and a recent commit ("Coalesce fsync requests during active flushes") merges concurrent flush barriers in fs/flush_coordinator.rs so a burst of fsyncs during an active flush doesn't serialize into N round trips.
Using it
zerofs init # generate zerofs.toml
$EDITOR zerofs.toml # set S3 credentials
zerofs run -c zerofs.toml[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "${ZEROFS_PASSWORD}"
[filesystem]
compression = "zstd-3" # or "lz4", changeable without migrationMount over 9P (the protocol to use if you care about fsync actually meaning "durable" — NFS COMMIT semantics let clients treat buffered writes as stable):
mount -t 9p -o trans=tcp,port=5564,version=9p2000.L,cache=mmap,access=user 127.0.0.1 /mnt/9pOr skip the filesystem entirely and get a block device:
mkdir -p /mnt/zerofs/.nbd
truncate -s 1G /mnt/zerofs/.nbd/device1
nbd-client 127.0.0.1 10809 /dev/nbd0 -N device1 -persist -timeout 600 -connections 4
zpool create mypool /dev/nbd0The NBD handshake advertises FLUSH and FUA, and a FLUSH on any connection covers all connections — write barriers hold even with -connections 4, which matters if you're layering ZFS on top.
There's also a real out-of-tree Linux kernel module (kernel/), written in Rust, that speaks a private 9P2000.L.Z dialect directly over TCP or AF_UNIX — bypassing FUSE and the stock v9fs client entirely. It's a genuine VFS driver: lookup, atomic_open, netfslib-backed writeback, POSIX locks, SEEK_HOLE/SEEK_DATA. When DKMS can't build it for your kernel, zerofs mount falls back to FUSE.
Rough edges
The project is licensed AGPLv3 with a commercial alternative, which is a real consideration if you're embedding this in something you ship. It's young by production-storage standards — the pjdfstest/xfstests/Jepsen CI suite is convincing evidence of correctness testing, not of years of field mileage. The zerofs/src tree runs to about 35k lines of Rust just in the core crate, plus a hand-rolled 9P protocol implementation (ninep/handler.rs alone is 6,650 lines) — this is not a weekend project you'll casually fork and patch. And leaning on a SlateDB fork pinned to a specific commit hash means ZeroFS's durability story is coupled to a dependency it doesn't fully control upstream, even though Barre maintains both.
Bottom line
If you need POSIX semantics on top of object storage — for a CSI driver, a self-hosted NAS replacement, or a ZFS pool that lives entirely in S3 — ZeroFS is the most seriously tested project in this space I've read the source of. The key-layout and locked-memory decisions in particular are worth studying even if you never deploy it.