High Availability and Fault Tolerance
Published on
Availability is the degree to which a service is usable according to its user contract. Fault tolerance is the ability to continue operating through specified failures. Horizontal scaling can add capacity, but it only improves availability when replicas do not share the same failure mode and traffic can reach healthy capacity.
Design from failure domains
Map process, node, rack or zone, cluster, region, identity, network, data, dependency, and control-plane failures. Place redundant components across the boundaries the service is expected to survive.
- Active/passive: simpler conflict model but requires detection, promotion, routing, and capacity readiness.
- Active/active: uses capacity continuously but requires data and request behavior that is safe under concurrent operation.
- Graceful degradation: preserves a critical journey while disabling optional or expensive capability.
- Cell architecture: limits customers and deployments affected by one fault.
Replication is not backup. Replicas can reproduce corruption or deletion. Define a recovery point objective (RPO), recovery time objective (RTO), retention, isolation from destructive access, and a verified restoration process.
Validate the design
Test failure detection, failover, capacity after loss, dependency behavior, data correctness, and the path back to normal operation. Exercise regional and identity failures, not only process restarts. Record observed recovery time and unresolved manual dependencies.
An availability target should correspond to user need and an error-budget policy, not a generic desire for more nines.