Two planes, many groups, no special node.
AnimusDB is a Dynamo-lineage store built out of ordinary Raft groups rather than a novel consensus protocol. This page is the whole shape of it: where the metadata lives, where your data lives, what a write costs, how you deploy it, and what it still can't do.
On this page
A map, and the data the map points at.
The control plane is a single small Raft group. It holds cluster membership, the tablet map (which node owns which range of which table), and table definitions including indexes and stream configuration. It is written to when the topology changes — a node joins, a tablet splits, an index is added — and that is a rare event measured against your request rate.
The data plane is one independent Raft group per tablet. Each has its own leader, its own log and its own quorum. A write is durable and acknowledged when a quorum of that tablet's replicas has it; nothing else in the cluster is involved.
Three failure modes the split removes.
No global log
A design that orders every write through one consensus group is bounded by that group forever. Here, adding tablets adds independent groups, and the work spreads across leaders that live on different nodes.
No special node
There is no coordinator to lose. Losing a node costs you leader elections on the tablets it happened to lead; the rest of the cluster does not notice, and no operator action is needed to repair the replicas.
Metadata off the hot path
The control plane is consulted for topology, not for traffic. A hot control plane would put a second quorum round trip in front of every request; this one is asked when the map changes and otherwise left alone.
How a table becomes a set of Raft groups.
A table is a partitioned, sorted map of maps — items addressed by a partition key and an optional sort key, with schemaless attributes. Partition keys are hashed to tokens; a contiguous range of the token space is a tablet, and a tablet is the unit of replication, movement and consensus.
When a tablet grows past its threshold it splits: the range is divided and the halves continue as separate Raft groups, online. Placement is topology-aware, so replicas of one tablet are spread across failure domains rather than piled onto neighbours. Tablets also move — that is what rebalancing after a join or a drain actually is.
Two consequences worth knowing before you design your keys. First, scans are ordered by token, not by partition-key value; ordering within a partition, by sort key, is preserved as you would expect. Second, tablets never merge today, so a table that grows and then shrinks keeps the tablet count it earned at its peak.
Storage sits behind an abstraction with both an in-memory and an on-disk implementation. The in-memory one is not a toy: it is what lets an entire cluster run inside the simulator at machine speed.
Linearizable by default, and explicit about the exceptions.
| Operation | Guarantee | What it costs |
|---|---|---|
| Single-tablet write | Linearizable | One Raft round trip to a quorum of that tablet's replicas, plus a durable local write |
ConsistentRead: true | Linearizable | Served by the tablet's leader |
ConsistentRead: false | Eventually consistent | Can be served without the leader; may return a stale item |
| Multi-tablet transaction | Serializable | Two-phase commit across the participating tablets' Raft groups, with hybrid-logical-clock commit timestamps and MVCC snapshot reads |
ConsistentRead: false is the single largest lever you have — exactly as it is on DynamoDB.One binary, four ways to run it.
Every node is the same animusd binary. What differs is which planes it hosts, and that is a line in a config file rather than a different build. A single config can mix shapes, so a cluster can migrate from one to another incrementally instead of being rebuilt.
| Shape | Hosts | Use it when | Command |
|---|---|---|---|
| Combined | Both planes | The default. Small and mid-sized clusters. | animusd --config … --node I |
| Control-only | Metadata quorum | Larger clusters, where you want the map on dedicated, boring hardware. | animusd control --config … --node I |
| Data-only | Tablets and client endpoints | Scaling capacity without growing the control quorum. | animusd data --config … --node I |
| Single process | An entire cluster | Development, CI, evaluation. | animusd --cluster N |
Generate a starting config with animusd gen-config --nodes 3 --host 10.0.0.1. Add a node later with animusd join --seed 10.0.0.1:7104 — a new node needs exactly one thing, the intra-cluster address of any node that is already running.
Six consecutive ports per node.
A node claims six ports from its base port, 7100 by default. Two nodes on one host need different bases.
| Offset | Default | Purpose | Exposure |
|---|---|---|---|
| +0 | 7100 | Internal — Raft traffic | Cluster network only |
| +1 | 7101 | Client protocol | Application network |
| +2 | 7102 | DynamoDB HTTP endpoint — and GET /metrics | Application network |
| +3 | 7103 | Admin API and dashboard | Operator network only |
| +4 | 7104 | Intra-cluster communication — the address a joining node needs | Cluster network only |
| +5 | 7105 | Data console | Operator network only |
TLS is available on every port but off by default, config-gated per cluster. Turned on, the internal/intra ports speak mutual TLS (every node presents a cluster-CA-signed certificate and verifies its peer's, so a non-member cannot complete a handshake at all) and the client/admin/console ports speak server-only TLS; turned off — the default — every port is exactly as plain and unauthenticated as the exposure column implies. Caller identity on the DynamoDB endpoint is still opt-in SigV4 either way, unaffected by TLS. The exposure column is a description of what you must additionally enforce at the network layer regardless of your TLS setting.
The whole cluster, inside one process, with time under our control.
Everything the database touches that could vary between runs — the clock, the network, the disk, thread scheduling, random numbers — goes through an environment abstraction. In production that abstraction is the operating system. In the simulator it is a deterministic stand-in: a virtual clock that advances only when the simulation says so, a message bus that decides delivery order from a seeded generator, and a storage layer that can be told to lose a write, reorder a flush, or return an error.
A whole cluster then runs inside a single process, faster than real time, with faults injected on purpose: partitions, crashes and restarts, clock skew, disk faults, message reordering and duplication.
The payoff is the seed. A failing run is described entirely by one number, so the failure replays exactly — on a colleague's laptop, in CI, a year later. That converts the expensive half of a distributed-systems bug, reproducing it, into a constant.
Determinism, and why you should careWhat exists, plainly, and what doesn't.
Available
- Growing — a joining node needs the intra-cluster address of any running node; tablets rebalance onto it online.
- Shrinking —
drainrelocates a node's replicas,drain-statusreports progress,decommissionremoves it. - Node failure — replicas are re-created automatically; no operator action.
- Control-plane membership — voters can be changed at runtime without rebuilding the cluster.
- Dropping a table — tablets are marked for reclamation and each node's reclaim loop stops the groups and deletes the storage.
- Backup and restore — on-demand backups and continuous backups (point-in-time recovery to any second); both outlive a dropped source table and restore into a brand-new one. The backup and stream-segment stores can target a filesystem path, replicate across the cluster (the default), or a real S3-compatible bucket; the Kubernetes operator wires the latter from a pre-existing
Secretholding static credentials (never one it creates itself) or, as on EKS, from the pods' projected service-account token exchanged with STS for temporary credentials (spec.s3.webIdentity). - Metrics — a line-oriented
name valueexport atGET /metricson the DynamoDB port: counters and leadership gauges. Reference Prometheus alert and recording rules, availability SLO burn-rate alerts and a Grafana dashboard ship indeploy/observability/. - Tracing — optional OpenTelemetry export with W3C trace-context propagation.
- Introspection — admin JSON endpoints for health, Raft state, storage internals, in-flight transactions and the tablet map, per node, plus a dashboard that renders them with no external dependencies.
- TLS — mutual on the internal Raft/intra wire, server-only on the client, admin and console ports; config-gated, off by default (a cluster with no TLS configured is unchanged plain TCP). The Kubernetes operator can wire it up from a pre-existing
Secretor a cert-managerCertificate.
Missing
- Operator-driven rolling upgrades — a node-by-node rolling upgrade is supported and CLI-assisted (
animus cluster roll planfor the order, restart one node,animus cluster roll waituntil it is healthy, thenanimus cluster finalize; feature gates keep new wire behaviour dormant until every node supports it), and a full-cluster stop, upgrade and restart is supported too. On Kubernetes the operator drives the roll itself: editingspec.imagerolls the pods one at a time behind the same health gate (the control leader hands leadership off first), stops and says why when a pod does not come back healthy, and leaves the irreversible finalize to you unless you opt in withspec.upgrade.finalize: Auto. Not yet safe: a roll from an older release with multi-item transactions in use (that release's own abort-handling bug, found by the previous-release roll test, still open). - Authentication beyond SigV4 — opt-in SigV4 on the client DynamoDB port is still the only caller-identity story. Credentials are now replicated (a static key file is only a bootstrap source, shadowed the moment a matching id is added through the admin API), rotatable with a dual-secret grace window, and each key can be scoped to a per-table, per-operation-class allow list — but this is a coarse allow list, not IAM: no condition keys, no temporary credentials, no per-item or attribute-level authorization, and the admin port itself still has no authentication at all. TLS (above) adds transport confidentiality and, on the internal ports, cluster-membership authentication — it doesn't add a richer client-identity model.
- Latency histograms and per-operation metrics — the metrics export has no latency histogram and no per-operation labels, so a p99 latency SLO cannot be measured from it yet; the reference alerts and dashboard (above) say which other signals are missing.
The list we'd want to be shown.
- TLS is config-gated and off by default; no authentication beyond opt-in SigV4 on the client DynamoDB port either way.
- Rolling upgrades are either driven by hand (
animus cluster roll plangives the order, you restart one node at a time,animus cluster roll waitblocks until it is healthy, then run the irreversibleanimus cluster finalize) or, on Kubernetes, by the operator from aspec.imageedit (clusters of at least three nodes whose PodDisruptionBudget allows losing one; smaller shapes are refused and need the whole-cluster stop, upgrade and restart). Skipping a release, rolling a node back to an older binary, revertingspec.imagemid-roll, and an old binary joining after the upgrade started are not supported. - Restore always creates a brand-new table, never in-place.
- Tablets never merge, so a table that shrinks keeps its tablet count.
- Not benchmarked at scale, and no performance claims are made for that reason.
- No production use by anyone, including us.