This is the seventh article in the series on running the ClickHouse® database on Kubernetes with the Altinity® Kubernetes Operator. Every cluster so far has held a single copy of the data. Fault tolerance means replication, and replication needs a coordination service: ClickHouse Keeper. This article deploys it.
What is ClickHouse Keeper, and why you need it
Run more than one copy of a table and the copies have to agree on what data exists and in what order it was written. Something has to be the single source of truth that every replica consults. ClickHouse historically used Apache ZooKeeper for this. ClickHouse Keeper is the modern, built-in replacement: it speaks the same protocol, uses less memory, and is simpler to operate. It holds the replication metadata, the bookkeeping that keeps replicas consistent, not your table data.
You never query Keeper and never store rows in it. It runs quietly in the background, and your replicated tables lean on it. No Keeper, no replication.
Quorums: why the node count is odd
Keeper stays correct by majority vote, called a quorum, and a group can make progress only while more than half its nodes are healthy. That is why the count is odd. A single node offers no fault tolerance, since if it dies, coordination stops, but it is perfectly fine for learning on a laptop. Three nodes form a quorum of two and survive any single failure, which is the standard production choice. Five nodes tolerate two failures. You rarely need more than three.
This article deploys a single-node Keeper so it fits comfortably on minikube, then shows the three-node version you would run in production.
Deploying Keeper with a ClickHouseKeeperInstallation
The operator manages Keeper through its own custom resource, the ClickHouseKeeperInstallation, or CHK. It resembles a CHI but describes a Keeper ensemble. Save this single-node version as keeper.yaml:
apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
name: keeper
spec:
configuration:
clusters:
- name: keeper
layout:
# Use 1 for a laptop, 3 for production fault tolerance.
replicasCount: 1
settings:
keeper_server/tcp_port: "2181"
defaults:
templates:
podTemplate: keeper-pod
volumeClaimTemplate: keeper-data
templates:
podTemplates:
- name: keeper-pod
spec:
containers:
- name: clickhouse-keeper
image: clickhouse/clickhouse-keeper:26.3
volumeClaimTemplates:
- name: keeper-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1GiA few notes. The image is clickhouse/clickhouse-keeper:26.3, matching the ClickHouse server version we use elsewhere. Keeper listens on the standard port 2181. It gets its own small persistent volume for its logs and snapshots, because Keeper's metadata also has to survive restarts. Apply it:
kubectl create namespace ch
kubectl apply -n ch -f keeper.yamlWatch it come up:
kubectl get chk -n ch -w
kubectl get pods -n chThe chk shorthand lists your Keeper installations. Once the pod is Running, Keeper is ready, and the operator has created a Service that ClickHouse will use to reach it.
The production version: three nodes
For production you change one number and give it real storage. This is the same resource with replicasCount: 3:
spec:
configuration:
clusters:
- name: keeper
layout:
replicasCount: 3
settings:
keeper_server/tcp_port: "2181"The operator deploys three Keeper pods, configures them as an ensemble, and they elect a leader and maintain a quorum on their own. You did not have to set peer addresses or raft IDs by hand; the operator did it.
Connecting ClickHouse to Keeper
Now point a ClickHouse cluster at this Keeper. With the operator, the clean way is to reference the CHK by name and let the operator discover the endpoints, so you never hard-code addresses. Save this as ch-with-keeper.yaml:
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
name: "ch"
spec:
configuration:
zookeeper:
keeper:
name: keeper # the CHK we just deployed
# serviceType: replicas # default; one entry per Keeper replica
clusters:
- name: "main"
layout:
shardsCount: 1
replicasCount: 2
templates:
podTemplates:
- name: clickhouse-pod
spec:
containers:
- name: clickhouse
image: clickhouse/clickhouse-server:26.3The zookeeper.keeper.name field points at our CHK named keeper. The cluster asks for two replicas, which only means something now that Keeper is present to coordinate them. Apply it:
kubectl apply -n ch -f ch-with-keeper.yaml
kubectl get chi -n ch -wWhen it reports Completed, two ClickHouse replicas are wired to Keeper. They are not replicating any data yet, because replication happens at the table level, which is the subject of the next article. But the foundation is in place.
If you ever need to point at an external ZooKeeper or Keeper instead, you can list addresses explicitly under zookeeper.nodes with a host and port 2181. The keeper.name reference is simpler, and it is what we use throughout this series.
Verifying Keeper is reachable
Confirm ClickHouse sees Keeper by querying a system table from inside a pod:
kubectl exec -it -n ch chi-ch-main-0-0-0 -- \
clickhouse-client -q "SELECT * FROM system.zookeeper WHERE path='/' FORMAT Vertical"A list of nodes under the root path means ClickHouse is talking to Keeper successfully.
Clean up
kubectl delete namespace chThis removes the ClickHouse cluster and the Keeper ensemble together.
What is next
Keeper is running and your cluster is connected to it. In the next article we finally build a replicated cluster: create a ReplicatedMergeTree table, insert into one replica, watch it appear on the other, and add a Distributed table to query across the cluster.
