How AnimusDB works

Data model

AnimusDB is a wide-column key/value store in the Dynamo lineage: a partitioned, sorted map of maps. A table holds items; each item is addressed by a partition key and, optionally, a sort key.

• The partition key decides where an item lives. It is hashed to a token on the table's ring.
• The sort key orders items within a partition, which is what makes range queries inside one partition cheap.
• Everything else is schemaless attributes — strings, numbers, binaries, booleans, nulls, lists, maps and sets.

Tables are independent. Each one has its own ring, its own tablets, and its own replication and placement decisions.

Tablets and partitioning

A tablet is a contiguous range of a table's hash ring, and it is the unit of everything: replication, placement, migration and splitting. Every key is stored under a Murmur3 token of its partition key, so a tablet owns a token range rather than a range of your raw keys.

That has a practical consequence worth knowing: scans are ordered by token, not by partition-key value. Sort-key order is preserved inside a partition, which is where ordering actually matters for queries.

A table starts as a single tablet and splits as it grows. Tablet identifiers are never reused, and a tablet's lineage across splits is recorded so stream consumers can follow it.

Replication

Each tablet is replicated to a set of nodes — typically three — and those replicas form a Raft group. One is the leader; it serializes every write for that tablet and answers reads. A write is acknowledged once a majority of the group has it durably on disk.

Which nodes hold which replicas is a placement decision made by the control plane. The placement engine honours the replication factor, spreads replicas across failure domains, and respects residency labels — so you can require that a group's data never leaves a given region or jurisdiction. A node's labels are set at its first registration with --label key=value, --labels-file, or a node entry's labels in the cluster config; on Kubernetes the operator supplies the node's topology.kubernetes.io/region and /zone automatically, and a table created while the members span at least as many zones as its replication factor spreads its replicas across zones (best effort).

When a node fails, the control plane notices through heartbeats, re-plans the affected tablets onto healthy nodes, and those nodes catch up by log replication or snapshot. No operator action is needed for the repair itself.

Consistency model

Single-tablet writes are linearizable: there is one leader per tablet and every write goes through its Raft log. Reads follow DynamoDB's own contract, and the choice is per request. ConsistentRead: true is linearizable — served by the leader behind a read-index barrier that confirms it is still the leader before it answers, so you never get a stale answer because you happened to hit a lagging replica. The default, ConsistentRead: false, is an eventually-consistent read: any replica answers it from its own applied state, with no barrier and no hop to the leader. It is cheaper and it scales across replicas, and it may return a slightly older — but always genuinely committed — version of the item. There is no quorum knob either way.

Multi-tablet transactions are serializable: two-phase commit across the participating tablets' Raft groups, with hybrid-logical-clock commit timestamps and MVCC snapshot reads.

A few things are deliberately not immediately consistent, matching DynamoDB's own semantics:

The two planes

AnimusDB separates cluster metadata from your data, and runs them as different systems with different scaling properties.

The control plane

One small Raft group owns the things every node must agree on: which nodes are members and where to reach them, the tablet map and its placement, and the table schema catalog. It is deliberately small and low-traffic. Its state is applied into a per-node system keyspace, so a node can restart and recover its view of the cluster from disk.

Nodes watch for metadata changes rather than polling. A single per-node reconciler reacts to the replicated state — hosting new tablets, reconfiguring groups, releasing tablets that moved away, reclaiming storage for dropped tables.

The data plane

Your data lives in one Raft group per tablet, each with its own storage engine, its own leader and its own election timers. Groups commit independently, so throughput scales with the number of tablets rather than funnelling through a single log.

An idle group quiesces: after a few seconds with no local activity it stops ticking entirely — no timers, no heartbeats, no apply polling — until a write, a peer message, or the reconciler wakes it. A cluster with many cold tablets costs proportionally little.

Any node answers

There is no routing tier and no client-side topology to configure. Every node knows the tablet map, and forwards a request it does not own to the node that does, retrying through a leader election if one is in progress.

Storage engine

Each hosted tablet gets its own log-structured merge engine: a write-ahead log for durability, memtables flushed into sorted string tables, and levelled compaction in the background. Because the engine is per-tablet, a tablet's identity is its file namespace — dropping it is deleting its files, and moving it does not disturb its neighbours.

When compaction falls behind, the engine slows its own writers (and waits at a hard limit) instead of failing them; the slowdown reaches clients as a retryable ThrottlingException, never as an error that loses a write. A node-wide compaction write budget is available as an opt-in setting.

Every byte of that engine's I/O goes through the same seam the simulator controls, which is why crash recovery is tested by injecting faults rather than by hoping.

Splits and rebalancing

Splitting

When a tablet passes its byte threshold, it splits online, in place: a single log entry mints both halves directly, already fully formed on every node that already holds the parent, so there is no separate copy, freeze or tail phase. The write pause at cutover is sub-second and masked by client retries. Both children start out on the parent's own replicas and are then moved onto their ideal placement afterward by ordinary rebalancing — the same mechanism that spreads a growing cluster.

Tablets are split-only — there is no merge operation, matching DynamoDB's own behaviour — so a table that shrinks keeps its tablet count.

Rebalancing

Replica counts across nodes converge on their own: the control-plane leader moves one replica at a time until the difference between the busiest and quietest node is at most one tablet (or, if nodes declare a placement weight, until each carries its share in proportion to it, never above a declared max_replicas). Adding a node is therefore all it takes to get data onto it — the same mechanism that relocates a freshly split tablet's children onto their final homes.

Indexes, streams and TTL

These three features are built on one mechanism: every tablet has a durable change log, and derived state is a consumer of it that tracks its own offset.

Secondary indexes

A local secondary index lives in the same tablet as its base item and is written atomically with it. A global secondary index is a hidden table maintained asynchronously by draining the base table's change log — which is what makes it eventually consistent, exactly as in DynamoDB.

UpdateTable adds or drops a GSI on a table that already holds data. The new index moves through a Creating → Active lifecycle while a backfill walks the existing rows, and a dropped one goes through Deleting so readers never see a half-built index.

Streams

Change data capture is served through the DynamoDB Streams API: ListStreams, DescribeStream, GetShardIterator and GetRecords. A shard is a sealed epoch of a tablet's own change log, so records are ordered per item and sequence numbers stay stable. When a tablet splits, its shards' lineage is recorded so a consumer can follow the children.

Time to live

A table can declare one attribute holding an absolute expiry time in epoch seconds. A leader-gated reaper deletes expired items through the normal write path, so index, stream and change-log maintenance all happen for free. Reads are AWS-faithful: an expired item remains visible until it is actually reaped.

API compatibility

The DynamoDB endpoint accepts DynamoDB JSON over HTTP with the usual X-Amz-Target header, so AWS SDKs work unmodified. Signature verification (SigV4) is opt-in and off by default; until it is switched on, credentials are accepted and not checked.

DynamoDB operations

Transactions

A transaction that touches a single partition commits through that tablet's Raft group directly. One that spans partitions runs two-phase commit across the participating groups: the coordinator stages intents, each participant durably records its decision in its own log, and a commit timestamp from a hybrid logical clock orders the result against everything else.

Practical consequences:

• Reads see a snapshot. MVCC versions are keyed by commit timestamp, so a transactional read observes one consistent instant.
• Conditions are evaluated at commit. A ConditionCheck or a write action's own condition expression is checked against the committed state, giving real cross-node optimistic concurrency control.
• A crashed coordinator does not strand data. A background resolver finishes or aborts in-doubt transactions it finds.
• Clock skew is handled, not assumed away. Reads inside an uncertainty interval restart rather than returning a value that might be misordered.

Consoles and the admin API

Two web interfaces are compiled into the node binary, plus a JSON API underneath them.

The admin API

Everything the dashboard shows is plain HTTP JSON, and the animus admin subcommands are a thin client over it — usable directly from scripts:

# observe
$ animus admin status|config|health|raft|raftkv|metrics <admin-addr>
$ animus admin peers|txns|backups|restores|backup-store|ttl-reaper|gc|segment-store|control-members|storage-control <admin-addr>
$ animus admin lsm|wal <admin-addr> [tablet]
$ animus admin key <admin-addr> <key> [tablet]
$ animus admin storage-scan <admin-addr> [--tablet <id>] [--start <key>] [--limit <n>]
$ animus admin system-table <admin-addr> [--kind <kind>] [--limit <n>] [--after <cursor>]

# act
$ animus admin split <admin-addr> <tablet> <split-key>
$ animus admin flush|compact <admin-addr> <tablet>
$ animus admin reconfigure <admin-addr> <tablet> <voter,voter,...>
$ animus admin drain|drain-status|remove|decommission <admin-addr> <node-id>
$ animus admin drain <admin-addr> <node-id> --force   # override the guard on the last Active node of a Region a global table pins

# control-plane membership (once the version era is on, start the new control voter first: it must be up to be admitted)
$ animus admin control-add|control-remove|control-grow|control-transfer <leader-admin-addr> …

# cluster version (ADR 0073 Phase 2)
$ animus cluster version <admin-addr> [--json]
$ animus cluster finalize <leader-admin-addr> [--to N] [--yes]
$ animus cluster capacity <admin-addr> [--json]   # declared node weight / max replicas vs. actual replicas

# rolling upgrade supervisor (ADR 0073 Phase 3): you restart each node; the CLI orders, gates and waits
$ animus cluster roll plan <admin-addr> [--json]     # the roll order; exits 1 if a precondition fails
$ animus cluster roll wait <node-admin-addr> [--timeout 10m]   # until that node is healthy and reports the new range
$ animus cluster roll status <admin-addr> [--json]

# global tables: MRSC regions/placement, and MREC replicas + this node's per-peer shipper health (backlog, lag, last error)
$ animus admin global-tables <admin-addr>
$ animus table preferred-leader <admin-addr> <table> <region>

# backup, restore, PITR, TTL, streams
$ animus admin backup-create <admin-addr> <table> <backup-name>
$ animus admin backup-delete <admin-addr> <backup-arn>
$ animus admin restore <admin-addr> <backup-arn> <target-table>
$ animus admin pitr-enable|pitr-disable <admin-addr> <table>
$ animus admin ttl <admin-addr> <table> <attribute> [--disable]
$ animus admin stream <admin-addr> <table> <view-type|off>

# export to S3
$ animus admin export-create <admin-addr> <table-arn> <s3-bucket> [s3-prefix]
$ animus admin export-describe <admin-addr> <export-arn>
$ animus admin export-list <admin-addr> [table-arn]

# import from S3
$ animus admin import-create <admin-addr> <table> <s3-bucket> [s3-prefix] [--gzip|--none] --pk name:TYPE [--sk name:TYPE]
$ animus admin import-describe <admin-addr> <import-arn>
$ animus admin import-list <admin-addr> [table-arn]

Bulk-loading synthetic data is a client tool, not a server action: animus seed generates items and writes them through real BatchWriteItem calls (proxied through the same admin interface) rather than an internal write primitive — one write path, with throttled rows retried and reported instead of silently dropped.

$ animus seed <admin-addr> <table> <count> [--start N] [--key-prefix P] [--value-bytes B] [--concurrency C] [--batch N]

Deployment shapes

A single config file can mix combined nodes with control-only and data-only ones, so a cluster can migrate from one shape to another incrementally.

Kubernetes

An operator (the animus-operator crate) manages an AnimusCluster custom resource: one StatefulSet per cluster, a headless internal Service plus a NetworkPolicy for node-to-node traffic — Raft, forwarding, metadata mirroring, seed and join — a separate client-facing Service for the DynamoDB edge, the only one meant to be reachable outside the cluster, and a PodDisruptionBudget bounding how many pods a voluntary eviction (a node drain, a cluster upgrade) may take down at once. The NetworkPolicy restricts both ingress and egress: inbound, only the cluster's own pods (plus the operator, for admin polling) and the DynamoDB port from anywhere; outbound, the cluster's own pods on the Raft/RPC ports plus DNS by default, and — only when an S3-compatible backup/segment store is configured (spec.s3) — that store's own endpoint port (plus 443 for STS when spec.s3.webIdentity is used), scoped to an explicit, operator-set CIDR list. spec.tls turns TLS on for every port the cluster binds (a pre-existing Secret, or one the operator asks cert-manager to issue) and spec.encryptionKeySecretName mounts a pre-existing Secret holding an encryption-at-rest key onto every pod — both off by default, and neither generated or stored by the operator itself. The PodDisruptionBudget's own maxUnavailable is computed from the cluster's control-voter count and its (currently fixed) data-plane replication factor, never a constant and never user-overridable — a cluster too small to safely lose even one replica gets a budget of zero, blocking every voluntary eviction outright. Scaling a cluster is a spec edit (kubectl patch its nodes field); deleting the custom resource tears down everything it owns through ordinary Kubernetes garbage collection. A CI-gated smoke test exercises a real kind cluster end-to-end — create, bootstrap, scale, delete — on every change to this surface. An image (or other pod-template) edit is a gated rolling upgrade: the operator owns the StatefulSet's update partition and lowers it one pod at a time, only while every member is Active, the cluster-wide roll-health check passes and the previous pod has reported the new version range; the control leader is handed leadership away before its pod restarts. A pod that does not come back healthy stops the roll (UpgradeBlocked names why; fix forward by editing the image again), and the new cluster version is raised by animus cluster finalize, or by the operator itself with spec.upgrade.finalize: Auto. Progress is on status.upgrade and the UpgradeInProgress / UpgradeFinalizePending / RollComplete conditions. Shapes that cannot lose a node (fewer than three nodes, or one control node) are not rolled: use a whole-cluster stop, upgrade and restart.

Day-two operations

Growing

A new node needs one thing: the intra-cluster address of any running node. It mints its own identity, registers with the control plane, mirrors the metadata and starts taking replicas as the rebalancer moves them.

Shrinking

drain moves a node's replicas elsewhere; drain-status reports progress; decommission removes it from membership once it is empty. A node that vanishes without draining is repaired by re-planning its tablets onto survivors.

Changing the control quorum

The control group's voters can be changed at runtime — added, removed or replaced — with control-add, control-remove and control-grow, plus a standalone control-transfer to move leadership without removing anyone. You do not have to rebuild the cluster to move the control plane.

Dropping a table

Dropping a table marks its tablets for reclamation; each node's reclaim loop stops the groups and deletes their storage. The process is convergent — it finishes correctly even if nodes restart partway through.

Runbook

Procedures for replacing or decommissioning a node, losing the control quorum, backup and restore drills, certificate rotation, upgrades, disk sizing and overload live in the operations runbook. It states plainly where a procedure has no tool yet (there is no recovery tool for a permanently lost control quorum, no in-place encryption-key rotation, and disk-full handling is proven in simulation and in a real-process chaos run on size-limited filesystems, not yet on a delayed-allocation one such as ext4 or xfs) and which procedures have not yet been exercised on a real cluster.

Observability

• Metrics — a line-oriented name value text export at GET /metrics on the DynamoDB port. Counters and a leadership gauge, in stable order. Reference Prometheus alert rules, availability SLO recording rules and a Grafana dashboard ship in deploy/observability/.
• Tracing — optional OpenTelemetry export with W3C trace context propagated across forwarded hops, so a request that lands on the wrong node still shows as one trace.
• Admin JSON — health, Raft state, storage internals, in-flight transactions, and the tablet map, all queryable per node.
• Dashboard — the same data, rendered, with no external dependencies to install.

Deterministic simulation testing

This is the engineering commitment the rest of the system is built around. No component outside a single seam may read a clock, spawn a task, touch the network or disk, or generate randomness. Everything goes through that seam, which has two implementations: the real one used in production, and a simulator.

The simulator runs an entire cluster on one thread with a virtual clock, and injects:

• network partitions, message reordering, duplication, delay and loss;
• node crashes and restarts at arbitrary points, including mid-write;
• disk faults and torn writes;
• clock skew between nodes.

Every run is a pure function of its seed. A failing test prints its seed, and that seed replays exactly the same history — the same interleaving, the same crash, the same partition — on any machine.

$ ANIMUS_SEED=8402173355719163220 cargo test linearizable_under_partition

Histories are checked for genuine consistency violations with Elle-style checkers rather than just checked for crashes, and corpora of seeds run in CI — a wide sweep on every push, a deep one nightly.

Known limits

Pre-alpha means real gaps. The ones most likely to matter:

• TLS is available on every port but off by default — mutual on the internal Raft/intra-cluster ports, server-only on the client, admin and console ports, config-gated per cluster. A cluster with no TLS configured is unauthenticated plain TCP, exactly as before; caller identity on the DynamoDB port is opt-in SigV4 either way, not affected by TLS.
• Rolling upgrades are one node at a time, by hand with CLI support or driven by the Kubernetes operator. By hand, animus cluster roll plan prints the order (data-only nodes, then control voters, the control leader last) and refuses if the cluster is not healthy; you restart one node on the new binary; animus cluster roll wait blocks until that node is Active, has reported the new range and the server-side roll-health check passes (no tablet under-replicated, its own groups caught up); then run animus cluster finalize once every member (down ones too) reports the new version range (roll wait --finalize --yes on the last node does it for you, opt-in). animus cluster roll status shows progress. On Kubernetes the operator runs the same gate from a spec.image edit (see the operator section above); finalize stays manual unless spec.upgrade.finalize: Auto is set. Each node advertises the cluster versions its binary supports; animus cluster version shows the active version, every node's range and what blocks raising it; finalize raises it one step, on the leader only, irreversibly, and is refused while any member is down, leaving, never activated or does not support the target. A mixed-version simulation corpus and a real-process previous-release roll test cover it. A roll restarts each node in turn, so on a cluster with spare nodes a slow restart can trigger replica rebuilds (correct, only wasteful). Supported: from today's binaries to this release, and from one release to the next. Not supported: skipping a release, rolling a node back after it ran the new binary, reverting spec.image mid-roll, an old binary joining once versioning has started (refused by name), and, for now, a roll from an older release with multi-item transactions in use (that release's own abort-handling bug, open). The CLI restarts nothing itself.
• Restore always creates a brand-new table, never in-place — matching real DynamoDB, for both on-demand backups (RestoreTableFromBackup) and continuous backups (RestoreTableToPointInTime); a source table's backup history and PITR coverage both outlive the table if it's later dropped.
• Tablets never merge, so a table that shrinks keeps its tablet count.
• The Kubernetes operator's validating admission webhook is opt-in, not installed by default — without it, an invalid spec edit (e.g. shrinking controlNodes after creation, or a malformed spec.s3) is caught by the controller after the fact, surfaced as a status condition, not rejected up front. Installing the webhook (a few extra manifests, a TLS cert either from cert-manager or hand-issued) rejects the same bad writes at the API server itself, before they're ever persisted — see the operator's own deploy docs. Growing controlNodes is honored either way: the controller drives the control group's own admin API to add each new voter.
• Not benchmarked at scale. No performance claims are made here for a reason.

Going deeper

Every significant design decision in AnimusDB is written down as an architecture decision record — the context, the decision, and the consequences knowingly accepted. They are the real documentation for how the internals work, and they are kept current with the code.