Connecting a Kubernetes Cluster

CHOps connects to ClickHouse® clusters running in Kubernetes. Instead of typing in each host, you point it at an installation and it reads the host list from the cluster and keeps it current.

This page covers AKOC, the Altinity® Kubernetes Operator for ClickHouse®. For the Official ClickHouse® Kubernetes Operator (OCKO), see Connecting a Cluster Managed by OCKO.

It assumes you have not done this before. Every step says what to type, what you should see, and what to do when you see something else.

Time needed: about 20 minutes the first time.

Contents

1. How the connection works
2. Which deployments are supported
3. Before you start
4. Step 1: create a read-only account
5. Step 2: make ClickHouse reachable
6. Step 3: create a ClickHouse user
7. Step 4: add the cluster
8. Platform by platform
9. When something does not work

1. How the connection works

The most important idea on this page, and the one that explains almost every problem people hit.

CHOps makes two separate connections, over two separate paths:

They are independent. One can work while the other does not, and CHOps reports them separately for exactly that reason.

Why this matters. CHOps runs outside your Kubernetes cluster. Inside a cluster, ClickHouse® is reachable at addresses like chi-demo-main-0-0-0.demo.svc.cluster.local. Those resolve only inside the cluster. From outside they mean nothing.

So even once CHOps can read your cluster perfectly, it still needs a separate, externally reachable address to run queries against. Most of the work below is making that second connection possible.

2. Which deployments are supported

Kubernetes discovery works wherever you control the Kubernetes API.

Deployments that use Direct connection instead

These are not gaps. A managed service gives you a database endpoint and nothing else, so there is no Kubernetes API for CHOps to read and nothing missing when it does not. Add them under Direct connection, which takes a minute and gives you everything except the Kubernetes screens.

Checking which operator you have

kubectl get crd | grep -i clickhouse

Both can run in one Kubernetes cluster without conflict. They use different API groups, so an installation managed by one is invisible to the other.

3. Before you start

On the machine you run commands from:

• kubectl, working and pointed at the right cluster. Check with kubectl get nodes and look for at least one node marked Ready
• Permission to create service accounts and roles

Already in the cluster:

• AKOC, checked above
• At least one installation. Check with kubectl get chi -A

Somewhere: CHOps running, and you logged in as an administrator.

4. Step 1: create a read-only account

CHOps needs an identity in your cluster. You are going to create one that can only look at things.

chmod +x scripts/chops-k8s-setup.sh
./scripts/chops-k8s-setup.sh

The script prints exactly what it is about to create and waits for you to type y. Read that list, because the script creates nothing until you confirm.

What it creates

The account may read these objects:

• ClickHouse® installations and Keeper installations, for AKOC and OCKO
• Pods, pod logs, services, config maps, events and resource quotas
• Persistent volume claims and endpoint slices
• Stateful sets, network policies, ingresses and pod disruption budgets
• Pod metrics (metrics.k8s.io), on clusters that have a metrics server
• Namespaces, storage classes and nodes. These are cluster-wide objects, so the account gets them only without -n
• The operator pods and their logs in the operator namespace, only with -o

It may not create, update, patch or delete anything, except a SelfSubjectRulesReview, which only asks Kubernetes what the token may do. It cannot read Secrets. Kubernetes enforces that, not CHOps.

Script options

What it prints

The script prints three values. Keep them on screen.

API address:
https://10.0.0.5:6443

CA certificate:
-----BEGIN CERTIFICATE-----
...

Token:
eyJhbGciOiJSUzI1NiIsImtpZCI6...

API address is where your cluster answers. CA certificate proves the server at that address really is your cluster; without it, anyone able to intercept your traffic could impersonate it, so CHOps always checks it and there is no way to switch that off. Token proves who CHOps is.

A narrower account

./scripts/chops-k8s-setup.sh -n production -n staging

The trade-off: CHOps cannot list namespaces, so you type the name instead of choosing from a dropdown. Volume expansion checks show as unavailable, because storage classes are a cluster-wide object. Zone awareness shows as unknown, because nodes are also a cluster-wide object. Everything else is the same.

Check the values before going further

Ten seconds, and it saves an hour. Save the CA certificate to /tmp/ca.crt and the token to /tmp/token.txt. Then run:

curl --cacert /tmp/ca.crt \
  -H "Authorization: Bearer $(cat /tmp/token.txt)" \
  https://10.0.0.5:6443/apis/clickhouse.altinity.com/v1/namespaces/YOUR-NAMESPACE/clickhouseinstallations

You want JSON containing "kind":"ClickHouseInstallationList". If this fails, fix it here. Nothing later can work until it succeeds.

Run the script again after a CHOps upgrade

A new CHOps version can read more optional objects than the script of an older version allows. These features need grants that older scripts do not give:

A token from an older script keeps working. Features without a grant show as unknown or unavailable. The Kubernetes Insights page shows the banner Optional permissions are missing. with the names of these features.

To get the grants, do these steps after each CHOps upgrade:

1. Run the script again with the options that you used before.
2. Add -o <operator namespace> if you want operator logs.
3. For operator logs, type the operator namespace in the connection. See Step 1 of 4: Connect.

It is safe to run the script again: it applies each object in place, and the token stays the same.

5. Step 2: make ClickHouse reachable

This is the step people miss, for the reason in part 1.

Option A: port forward, for testing only

kubectl port-forward -n YOUR-NAMESPACE service/clickhouse-YOUR-INSTALLATION 8123:8123

Leave that terminal open. In another:

curl http://localhost:8123/ping

You want Ok. Then use localhost, port 8123, TLS disabled.

This is fine for trying things out, but not for anything real, because closing that terminal takes the cluster away.

Option B: LoadBalancer service

Ask the operator for an external address:

spec:
  templates:
    serviceTemplates:
      - name: external
        spec:
          type: LoadBalancer
          ports:
            - name: http
              port: 8123
  defaults:
    templates:
      serviceTemplate: external

Then kubectl get svc -n YOUR-NAMESPACE and look under EXTERNAL-IP. It can take a minute or two to appear.

Before you do this, you are putting a database on the public internet. Restrict it with a firewall, a security group, or ClickHouse® user network rules.

Option C: internal load balancer, recommended

The same as B with the address private to your network, and CHOps running inside that network. Nothing is exposed publicly. The cloud-specific annotation is in part 8.

Option D: Ingress

If you already run an ingress controller, add a rule and use that hostname.

6. Step 3: create a ClickHouse user

Do not use the default user. This catches nearly everybody once.

The operator restricts default to the IP addresses of the cluster's own pods. A connection from CHOps is refused, and the refusal looks exactly like a wrong password, so you will spend twenty minutes checking a password that was always correct.

Leave clickhouse_operator alone too. It belongs to the operator, which restricts it to its own pod.

Create one for CHOps

spec:
  configuration:
    users:
      chops/password: use-a-real-password-here
      chops/networks/ip: "::/0"
      chops/profile: default
      chops/quota: default

networks/ip: "::/0" means connect from anywhere. For production, narrow it to the address CHOps runs from, such as 10.20.30.40/32.

For production, use a hash

echo -n "your-password" | sha256sum | tr -d '  -'

      chops/password_sha256_hex: THE-HASH

Check it

curl "http://localhost:8123/?user=chops&password=your-password" -d "SELECT 1"

You want 1.

7. Step 4: add the cluster

Go to Control panel, Cluster Management. Select the Kubernetes tab, then New Cluster. Only administrators can add a cluster.

If the Kubernetes tab is not there, the Kubernetes feature is turned off for this CHOps instance.

Step 1 of 4: Connect

To use a connection that you saved before, select it in Use an existing connection. To add a new one, do these steps:

1. Type a Connection name that you will recognise.
2. Paste the whole block from part 4 into the box Paste the output of chops-k8s-setup.sh. CHOps fills in API address, CA certificate and Token.
3. Optional: type the Operator namespace. This is the namespace where the operator pod runs. CHOps needs it to show operator logs. AKOC installs into kube-system by default.
4. Select Test connection. You want Kubernetes: Connected.
5. Select Continue. CHOps saves the connection.

Probe checks that CHOps can find the operator pod in the operator namespace. It works only after CHOps saves the connection. To use it, go back to step 1, select the saved connection, and select Probe. The token needs the -o grant for that namespace.

With a token from -n, the test in this step can fail because the token cannot list namespaces. This is expected. Select Continue and type the namespace in step 2. When a namespace is already set, for example when you come back to this step, the test passes and shows "Connected. This token cannot list namespaces, so you will need to type the namespace name."

Step 2 of 4: Installation

In Operator, choose Altinity® Kubernetes Operator for ClickHouse® (AKOC).

Choose or type a namespace and select Find installations. The table shows the name, status, version and number of hosts of each installation. Select yours.

Step 3 of 4: ClickHouse address

Type the address from part 5 in ClickHouse® address. The defaults are port 8443 and TLS Enabled.

Pod addressing

Some pages, for example Current Queries, Merges, Parts and Replication, show data for each node, and for these CHOps must reach each pod directly. Pod addressing sets how CHOps reaches the pods:

If CHOps runs outside the cluster, the internal pod addresses do not resolve. To get data for each node, give each pod its own service. Auto-detect shows an example for AKOC under How to expose each pod so per-node data works. The simplest option is to run CHOps inside the cluster.

When CHOps cannot reach each pod, it asks you to confirm before it adds the cluster. Select Import anyway to continue. Cluster-wide queries in the SQL editor are not affected.

Step 4 of 4: Credentials

Type the user from part 6 in ClickHouse® user and its Password. Display name is the name of the cluster in CHOps. The default is the installation name.

You enter these once for the whole installation. Every host inherits them, which is what keeps things working when somebody scales the cluster and new hosts appear.

Select Add cluster. CHOps tests the credentials before it saves. If they do not work, the dialog ClickHouse® credentials did not work opens. Select Add anyway or Go back and fix. Adding anyway is reasonable: the Kubernetes screens still work, which is what you need while diagnosing why ClickHouse® will not answer. The SQL editor, dashboards, alerts and backups do not work until you fix the credentials, which you do by editing the cluster.

Confirm it worked

Three checks, in order of how much they prove:

1. The cluster appears with the right number of hosts. Each host has the name <cluster>-<shard>-<replica>. Here, <cluster> is the cluster name in the installation, and shards and replicas count from 0.
2. Open it and run SELECT hostName(), version().
3. Create an alert rule against it and open a dashboard. Neither feature knows Kubernetes exists, and both should work unchanged. That is the whole point of the design.

8. Platform by platform

K3s

Easiest, and the recommended way to try this.

sudo chmod 644 /etc/rancher/k3s/k3s.yaml
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'

If CHOps runs on another machine, the address must be the node's real IP and that IP must be in the API server's certificate. Set that at install time:

curl -sfL https://get.k3s.io | sh -s - --tls-san 10.0.0.5

Without --tls-san, connecting by IP gives a certificate error saying the certificate does not cover the address. That message is correct and this is the fix.

K3s ships a load balancer, so a LoadBalancer service gets an address immediately. Its storage cannot grow a volume, so CHOps will correctly report volumes as not expandable.

minikube

minikube ip

The API is on port 8443, so the address is https://<that-ip>:8443.

LoadBalancer services stay <pending> unless minikube tunnel is running. Alternatively use a NodePort and connect to the minikube IP on the port shown.

If CHOps runs in a container, put them on the same Docker network: docker network connect minikube your-chops-container.

Self-managed Kubernetes

This covers kubeadm, Rancher and OpenShift, and it is the cleanest case because you control the network.

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'

Use exactly what it prints. If it is a hostname, use the hostname. Substituting an IP causes a certificate error.

On OpenShift the setup script works unchanged. Use a Route rather than an Ingress: oc expose service/clickhouse-demo -n demo.

Amazon EKS

aws eks describe-cluster --name YOUR-CLUSTER --query 'cluster.endpoint' --output text

The most common blocker:

aws eks describe-cluster --name YOUR-CLUSTER \
  --query 'cluster.resourcesVpcConfig.[endpointPublicAccess,publicAccessCidrs]'

If public access is false, CHOps must run inside the VPC. There is no way around that from outside, and it is not a CHOps limitation. If a CIDR list exists, the CHOps address must be on it.

Internal load balancer:

        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-type: nlb

Recommended: CHOps on an EC2 instance in the same VPC. That solves both problems at once.

Azure AKS

az aks show -n YOUR-CLUSTER -g YOUR-GROUP --query fqdn -o tsv

Prefix with https://. Check authorized IP ranges:

az aks show -n YOUR-CLUSTER -g YOUR-GROUP \
  --query apiServerAccessProfile.authorizedIpRanges

Internal load balancer annotation: service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Recommended: CHOps on a VM in the same virtual network.

Google GKE

gcloud container clusters describe YOUR-CLUSTER --zone YOUR-ZONE --format='value(endpoint)'

Prefix with https://. GKE calls the restriction authorized networks. If they are configured, the list must include the CHOps address.

GKE normally issues short-lived credentials through gke-gcloud-auth-plugin. CHOps uses a service account token instead, which does not expire and needs no plugin on the CHOps machine. That is why the same instructions work on every cloud.

Internal load balancer annotation: networking.gke.io/load-balancer-type: "Internal"

Recommended: CHOps on a Compute Engine instance in the same VPC.

Altinity.Cloud BYOK

The cluster runs in your cloud account and you have access to it, so follow whichever of EKS, AKS or GKE it runs on.

There are two differences. Altinity manages the operator, so do not change operator settings without talking to them. And the ClickHouse® endpoint is already exposed, so use the address from the Altinity.Cloud® console rather than creating your own service.

9. When something does not work

Work out which half failed

Open the cluster in CHOps.

• Kubernetes screens work, queries fail. The ClickHouse® address or the credentials. See parts 5 and 6.
• Queries work, Kubernetes screens fail. The API address, certificate or token. Continue below.
• Both fail. Usually a network route problem.

Cannot reach the Kubernetes API

Three usual causes: the address is wrong, the cluster restricts access by IP, or there is no public endpoint at all. Compare against your kubeconfig, check the command for your platform in part 8, and if the API is private, CHOps must run inside your network.

The CA certificate does not match this server

Copy it again, including the BEGIN and END lines:

kubectl config view --raw --minify \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d

Note --raw. Without it, kubectl hides the value.

The certificate does not cover this address

This is a different problem, and easy to confuse with the one above: the certificate is fine, but the address is not. Use exactly the address from your kubeconfig. On K3s this usually means the cluster was installed without --tls-san.

The token was rejected

The service account was deleted or the token revoked. Run the setup script again.

This token cannot list something

CHOps names the exact permission and namespace. Listing namespaces needs cluster-wide access, so if you used -n that is expected: type the namespace name instead.

No installations found

There are three usual causes. The operator is not installed, the operator does not watch that namespace, or your token cannot see it. If the operator is not installed or the token cannot read the namespace, CHOps shows an error that says so. If the list is only empty, check which namespaces the operator watches.

An installation the operator does not watch shows as unmanaged on the Reconcile screen. It exists and nothing is acting on it.

Authentication failed with a password you know is right

Almost certainly the default user. See part 6. Confirm your user exists:

kubectl exec -n YOUR-NAMESPACE YOUR-POD -- \
  clickhouse-client -q "SELECT name FROM system.users"

A host disappeared

There are two innocent explanations. A pod restarting during maintenance vanishes for a minute or two, and CHOps waits for several missed refreshes before removing a host. See How the host list stays current. Or the operator removed a host that failed to start after a scale-out, which is its default behaviour rather than somebody deleting your machine.

What next

Everything in CHOps works against this cluster. For the Kubernetes screens, see The Kubernetes Page. It explains what each screen shows and why it is worth looking at.