Determinism, and why you should care

Start with the class of bug that matters, because it is not the class your test suite is aimed at.

Unit tests catch logic errors. Integration tests catch interface mismatches. Both are worth having and neither finds the bugs that take distributed databases down. Those come from partial failure: the node that is not dead but not answering; the network that delivers the message, late, after the retry has already been delivered; the leader that has been replaced but hasn't noticed; the disk that reports a successful write it did not actually make durable.

These bugs need a specific interleaving of events to appear. In a system with any real concurrency, the space of interleavings is astronomically large, and almost all of them are fine. You do not have a correctness problem in the ordinary case. You have a correctness problem in one arrangement out of ten thousand — which, at production request rates, is a Tuesday.

Why staging doesn't find them

The standard answer is to run a copy of production and hope. It does not work, for two reasons that compound each other.

The first is that you cannot schedule the conditions. You can wait for a partition to occur naturally between two of your zones, and you will wait a long time, and when it happens it will not coincide with a leader election and a slow disk. The interesting failures are conjunctions, and conjunctions of rare events are rarer still.

The second is worse. On the rare occasion staging does produce one, you cannot reproduce it. You have a log fragment, an assertion that fired once, and a system whose behaviour depends on thread scheduling and network timing you do not control. Engineers then spend days trying to provoke the failure again, because a bug you cannot reproduce is a bug you cannot confidently fix. Frequently the outcome is a plausible patch, an unprovable claim that it worked, and a ticket closed on optimism.

What deterministic simulation actually is

The idea is to remove every source of non-determinism from the system, so that a run is a pure function of its inputs.

Everything that could vary between executions — the clock, the network, the disk, task scheduling, random numbers — is accessed through an abstraction rather than directly. In production, that abstraction is the operating system. Under test, it is a deterministic substitute:

• Time is virtual. It advances when the simulation says so, which means a week of cluster life runs in seconds and a timeout can be made to fire at the precise moment that hurts.
• The network is a queue whose delivery order, delays, drops, duplicates and partitions are chosen by a seeded generator.
• The disk can be told to be slow, to fail, to reorder, or to lose data that was never flushed — the way real disks behave when a machine loses power.
• Scheduling is controlled by the simulation rather than by the operating system, so the interleaving is chosen deliberately instead of observed by luck.

A whole cluster then runs inside a single process, faster than real time, with faults injected on purpose rather than awaited. Instead of hoping to encounter one bad interleaving, you sample the space of them systematically, for as many machine-hours as you care to spend.

Then the important part

Because nothing varies except the seed, a failing run is fully described by one number. Hand that number to a colleague and they get the identical failure. Put it in CI and the failure replays a year later, on different hardware, after a refactor.

This is what converts the expensive half of a distributed bug into a constant. The fix is verified by re-running the seed. The regression test is the seed. Nobody has to argue about whether a patch worked.

What it buys you, as the person signing off

• Failures move earlier. The one-in-ten-thousand interleaving is met in development, when it is a test run, rather than in production, when it is an incident.
• Fix times become predictable. The unbounded step — reproduce it — is removed. That changes what an SLA on bug turnaround can honestly promise.
• Fault handling is exercised, not assumed. The recovery paths that normally run for the first time during an outage have run thousands of times.
• Correctness is checked continuously. Invariants such as linearizability can be asserted on every simulated run, not sampled during an occasional chaos exercise.

What it does not buy

Any claim this strong deserves its boundary drawn by the people making it, so here is ours.

It is not a proof. Simulation samples the space of executions; it does not exhaust it. Confidence rises with machine-hours, and it never reaches certainty.

It only tests what was modelled. The simulator injects the faults we thought of. Real environments are more inventive: firmware that lies about flushing, a NIC that corrupts one packet in a billion, a kernel bug, an operator running the wrong command. Nothing under the simulator sees any of that.

It only covers what is under it. A design where half the system bypasses the abstraction is only half tested, and the untested half is exactly where you would not want to be surprised.

It says nothing about performance. Virtual time is what makes the technique work, and it is also why a simulated run tells you nothing about latency on real hardware. Those numbers have to be measured separately — and, for AnimusDB, have not been.

It is not a substitute for production experience. It is the best available substitute for the specific class of bug described at the top of this page, which is a narrower and more defensible claim.

This is not a novel idea

The technique is well established, and it is worth knowing the lineage rather than taking it on trust from us.

FoundationDB is the canonical case: it built a deterministic simulator before it built the database, and its reputation for durability is generally attributed to that decision more than to any other. TigerBeetle applies the same approach to a financial transactions database and publishes its simulator's results openly. Antithesis exists as a company selling deterministic simulation as a service, which is a reasonable signal that the industry considers the problem real.

AnimusDB is in that tradition rather than inventing anything: the simulator came first, and the database is written to live inside it.

How to interrogate anyone who claims this

"Deterministic simulation testing" is becoming a marketing phrase, and it covers a wide range of actual rigour. If a vendor claims it — this one included — these are the questions that separate the claim from the practice.

1. What fraction of the system runs under the simulator? The consensus layer only, or the storage engine and the API layer too? Coverage is the whole game.
2. Is time virtual, or merely accelerated? If timeouts still depend on the wall clock, runs are not deterministic and seeds do not replay.
3. Which faults are injected? Partitions and crashes are the easy ones. Ask about clock skew, disk faults, message duplication and reordering, and partial writes.
4. How many seed-hours run per release, and where do you publish that? A technique that runs nightly on one machine is a different claim from one that runs on a fleet.
5. When a seed fails, does it become a permanent test? If found seeds are not kept and replayed in CI, the same bug can return.
6. What escaped anyway? The most useful answer any project can give is a specific bug that the simulator missed, and what was added to the model afterwards.

Ask us those six. The answers are in the repository and in the design records, which is where they belong — a claim about testing that you cannot inspect is just a claim.

Why this matters more for a young project

Mature databases have a different kind of evidence: years of production use, across thousands of deployments, with the worst bugs found by other people. That is genuinely the strongest evidence there is, and AnimusDB does not have it. It is pre-alpha, and nobody has run it in production, including us.

Deterministic simulation is not a replacement for that evidence. It is the only thing available that partially substitutes for it — a way for a young system to have already survived failures that a mature one accumulated the slow way. When you are weighing a database with no operational history, the right question is not whether it has been tested. It is whether the testing was designed to find the failures that history would have found.

Keep reading