High availability and disaster recovery get mentioned in the same breath so often that people assume they are the same thing. They are not. They protect against different failures, and confusing them leaves a gap that only shows up on your worst day. Here is how the two differ in a ClickHouse® database, and how the building blocks fit together.
Two different problems
High availability keeps the database serving when a single component fails: a server crashes, a disk drops out, a replica goes offline. The aim is that users keep reading and writing without noticing that one machine died.
Disaster recovery deals with the big failures: a whole data center outage, a region loss, a table deleted by accident, data corrupted and then copied everywhere. The aim there is different. It is to restore the system and the data after something catastrophic, usually from backups or a separate environment.
Keeping those two goals distinct is the whole point, because the mechanisms that give you one do not give you the other.
How ClickHouse® delivers high availability
Availability in ClickHouse® rests on replication and distributed query execution. The ReplicatedMergeTree family stores multiple copies of the same data parts across different servers, so if one replica is unavailable, another can keep serving.
Replica A
Replica B
Replica C
If Replica A goes down, B or C carries on. One detail worth internalizing: replication is asynchronous. A write accepted by one replica propagates to the others in the background, so this is not the synchronous replication you may know from some relational databases. Metadata that coordinates all of this, the replicated parts, merges, mutations, and replica state, is managed by ClickHouse Keeper, while the actual data stays on local disks.
Distributed tables sit on top and decide where queries go. Rather than pointing applications at individual replicas, you point them at a Distributed table that balances reads, routes around failed nodes, and keeps querying the healthy replicas. In production, a load balancer usually sits in front of the nodes too, sending traffic only to servers that are up.
Keeper is coordination, not storage
ClickHouse Keeper is the service that makes replication work: replica registration, replication queues, leader election for merges, mutation coordination. It does not store your data. But if Keeper becomes unavailable, replication coordination can stall even though the data itself is intact on disk. For that reason Keeper should run as its own replicated cluster, so it does not become the single point of failure you were trying to design away.
What availability cannot save you from
Even a flawless replicated cluster does not protect against everything. Replication faithfully copies whatever it is given, including your mistakes. It does not help if every replica is deleted, if incorrect SQL removes data on all of them, if corrupted data is replicated to every node, or if an entire region is lost. Each of those needs disaster recovery, not availability.
The disaster recovery side
Backups are the core of disaster recovery. ClickHouse® can back up databases and tables with SQL, writing to local disks, S3-compatible object storage, network storage, or cloud storage. A backup is a recoverable snapshot, and without one, many catastrophic failures are simply irreversible.
Two practices make backups actually protective. First, keep them offsite. A backup on the same server disappears when that server does, so store copies in object storage or another data center. Second, some organizations run entirely separate clusters in different regions:
Primary Cluster
|
Data replication / backup
|
v
Secondary Cluster
If the primary environment is lost, applications fail over to the secondary. ClickHouse® does not provide cross-region disaster recovery automatically. Designing that, and the backup policies and recovery procedures around it, is on you.
Set your recovery targets first
Before designing any of this, define two numbers. The Recovery Time Objective is how quickly the system must be back, whether that is fifteen minutes or four hours. The Recovery Point Objective is how much recent data you can afford to lose, from zero to an hour. Your backup frequency and replication strategy flow directly from those targets, so decide them up front rather than discovering them during an incident.
Three misconceptions to drop
Replication is not a backup. It copies data between replicas, including deletions and corruption, so backups remain necessary. High availability does not prevent disasters. It minimizes downtime from component failures and cannot recover data after a catastrophe. And ClickHouse® does not hand you disaster recovery automatically. It gives you the pieces; the resilience comes from how you assemble them.
Putting it together
A production deployment usually combines all of it: ReplicatedMergeTree for copies, Keeper as a highly available cluster, Distributed tables or a load balancer for routing, regular backups stored offsite, and tested restores. Each piece addresses a different kind of failure. Test your restores on a safe target, because a backup you have never restored is a theory, not a recovery plan. Once you build monitoring and backups into one place, keeping an eye on replica health and backup history stops being a chore. You can see how CHOps handles monitoring and backups as part of that setup.
Disaster recovery is serious work with real consequences. Define your targets, keep backups offsite, test restores regularly, and make sure more than one person knows the procedure.
