pNFS Lattice is a user-space metadata server for NFSv4.2 and pNFS being developed by PEAK:AIO, LANL, and CMU. It implements the pNFS metadata server (MDS) which handles file metadata ops (open, create, stat, and unlink) and pNFS file layouts.
Lattice scales pNFS MDSes by storing metadata in a single, shared, distributed metadata catalog stored in RonDB, a distributed key-value store.1 All metadata changes are single RonDB transactions, so metadata throughput scales with more MDSes until the RonDB cluster’s shared commit budget is exhausted.2
Semi-shared-everything
Lattice addresses the single-MDS limitation of pNFS by separating protocol handling from metadata state, but the separation is partial:
- The MDS tier still partitions the namespace and holds protocol state in partitioned volatile memory.3
- The persistence tier (the metadata catalog) is a shared-nothing database.
These two tiers fail independently and have their own failure detection and recovery processes. This means that Lattice still has the server-side failure modes of shared-nothing metadata services (split brains and the like).
Architectural strengths
Because Lattice uses a standard NFS client, compute nodes don’t need any proprietary kernel modules, hold no distributed lock state, and take no part in server-side recovery (unlike Lustre or GPFS). The metadata catalog backend is a pluggable interface as well, though RonDB is the only implementation today.4
Architectural weaknesses
When the cluster splits, RonDB relies on an arbitrator to decide which side survives. If no arbitrator is available, the cluster crashes.5 Lattice’s metadata availability therefore depends on RonDB’s membership and arbitration behavior just like GPFS does.
Lattice’s MDS tier also holds state; the namespace is partitioned across MDSes by subtree, and clients are redirected to the MDS that owns each subtree using NFSv4 referrals. Cross-subtree renames (like moving a directory) require a two-phase commit between MDS daemons.6
NFSv4 session, open, and lock states live in each MDS’s memory only,7 so when an MDS fails, clients must reclaim their state during an NFSv4 grace period that defaults to 90 seconds.8
Lattice’s architecture also requires running two independent failure detectors:
- Each MDS writes a heartbeat to RonDB every 5 seconds,9 and a separate watchdog initiates MDS failover when heartbeating stops.6
- RonDB separately tracks the health of its data nodes, with its own heartbeat parameters and timeouts.10
These two detectors watch different components on different cadences, and Lattice’s MDS health depends on RonDB being alive and healthy. This means the two can disagree about whether a given MDS is alive, and Lattice has to encode logic to reconcile those two differences itself.
Compared to full shared-everything
The client-facing implementation of the pNFS MDS in Lattice has the same benefits of VAST’s disaggregated shared-everything (DASE) architecture because metadata and session state are stored in a shared database. However, Lattice and VAST differ in the implementation of that shared database; Lattice still uses a shared-nothing KV store for its metadata catalog, while VAST’s persistence layer is fully shared-everything. This results in a few major benefits that VAST has over Lattice:
- When a cluster splits, there is no arbitration around who survives in VAST. The survivor is persisted in the shared-everything store as a single source of truth that the split cluster simply honors.
- VAST’s MDS tier (CNodes) hold no state, and there is no namespace partitioning across its NFS servers. NFSv4 session state is persisted in the shared-everything backing store, so recovery is trivial. If an NFS server (CNode) dies, any surviving CNode can take over its clients, complete with session state, without forcing client state reclamation.
- Lattice relies on storing all metadata in DRAM, and it replicates metadata across all nodes in a node group. This means metadata is volatile, stored on very expensive media, and stored very inefficiently (full copies across every server in the group). VAST stores its metadata on nonvolatile storage-class memory, and only stores two or three copies by default.
- If every node in one Lattice node group fails, the cluster loses an entire partition and can no longer serve a complete copy of the data.11 VAST CNodes are not grouped into failure domains, so the whole system remains 100% available as long as a single VAST node is online.
At the end of the day, Lattice is moving towards a good architecture for a pNFS MDS layer. However its reliance on a shared-nothing database for its shared-everything metadata still carries all the operational headaches of shared-nothing, and the lack of integration between the MDS and the backing store (as evidenced by two different failure detectors with an ugly interdependence) is a gap where problems are likely to occur in operations.
Lattice’s pluggable backing store gives it the opportunity to choose a better architecture for its persistence layer, and the vision of Lattice includes fully distributing client session state rather than localizing it in shards across MDS memory.3