failure modes, from easiest to hardest to handle

Not all failures are equal. Distributed systems literature usually orders them along a spectrum from "easy to detect and handle" to "actively adversarial," and a design's fault-tolerance strategy has to match the failure modes it's actually built to survive.
Failure modeWhat happensWhy it's easier or harder
Fail-stopA node halts permanently, but other nodes can detect that it has stopped (e.g. via a heartbeat that stops responding).Easiest — the failure is clean and detectable.
CrashA node halts silently; other nodes can't immediately tell whether it's dead or just slow.Harder — requires timeouts and health checks to distinguish "dead" from "delayed."
OmissionA node fails to send a message (send omission) or fails to receive/acknowledge one (receive omission).Requires retry and acknowledgment logic to detect and recover.
TemporalThe node produces a correct result, but too late to be useful — often from clock drift or an overloaded queue.Correctness isn't the problem; timeliness is, which is a different class of bug to catch.
ByzantineThe node behaves arbitrarily — sending conflicting messages to different peers, returning wrong results, or acting maliciously.Hardest — requires consensus protocols specifically designed to tolerate lying or malfunctioning participants.
Most internal enterprise systems are designed to tolerate fail-stop, crash, and omission failures, and treat temporal failures as a performance problem to monitor. Byzantine fault tolerance is reserved for environments where a node might be actively malicious or compromised — blockchain consensus protocols are the most common real-world example; it's rarely worth the overhead for a typical internal service mesh where all nodes are operated by the same trusted organization.

fault tolerance technique 1: replication

The most common fault-tolerance technique is replication: keep multiple copies of both services and data, so a failed node can be swapped for a healthy replica without losing availability. The design question replication always raises is how those copies stay in sync, and it maps directly onto the CAP trade-off:
Update strategyBehaviorTrade-off
Synchronous replicationAll replicas confirm the update before it's considered complete.Strong consistency, but write latency grows and availability drops if a replica is slow or unreachable.
Asynchronous replicationThe primary accepts the update immediately; replicas catch up afterward.Low write latency and high availability, but reads can return stale data until replicas converge.

fault tolerance technique 2: checkpointing

Checkpointing periodically saves a system's state to stable storage, so that after a crash the system can resume from the last checkpoint rather than starting over from nothing. A batch data pipeline that processes a billion records and checkpoints every million, for instance, loses at most a million records of progress on a crash instead of the entire run.
A checkpoint is only useful for recovery if it captures a consistent state — one where every update completed before the checkpoint was captured, no messages were left "in flight" (sent but not yet received) at the moment of the snapshot, and the relationships between components match what a normal, non-crashed system would look like. A checkpoint taken mid-transaction, with some effects saved and others not, is often worse than no checkpoint at all, because resuming from it can silently corrupt state rather than just losing progress.

designing for the failures you actually expect

The practical takeaway for a design doc or an interview answer isn't to enumerate every failure mode exhaustively — it's to state explicitly which ones the design handles and which it doesn't. "This design tolerates a single node crash but not a full regional outage" is a far stronger answer than silently hoping nobody asks, because it shows the trade-off was a deliberate choice rather than an oversight.

related topics

reference