Production Deployment

This guide walks you through deploying CHOps on a Linux server with automatic HTTPS (via Caddy) and automatic startup (via systemd). At the end, CHOps runs as a background service at https://your-domain.com, and restarts automatically if it crashes or the server reboots.

If you are new to Linux servers, read every step. Nothing is skipped.

Prerequisites

You need:

  • A Linux server (tested on Ubuntu 24.04 and 26.04, and Debian 13; other modern distributions should also work).
  • A hostname for the server. Use a public domain (for example, chops.example.com) if you expose CHOps to the internet, or an internal DNS name or plain IP address if you run it on a private network. A domain is not required.
  • SSH access to the server.
  • If you expose CHOps publicly with automatic HTTPS: ports 80 and 443 reachable from the internet, which Let's Encrypt needs to issue certificates. On a private network this is not required. See the internal TLS options in Step 5.
  • Port 8123 reachable from the server to your ClickHouse® host (not from the internet).

Step 1: Install Bun

curl -fsSL https://bun.com/install | bash -s "bun-v1.4.2"
source ~/.bashrc
bun --version

The service user in Step 3 cannot read your home directory. For a source install, copy Bun to a shared path:

sudo cp "$(which bun)" /usr/local/bin/bun

If you build a binary instead, you only need Bun on your build machine, not on the production server.

Step 2: Set Up CHOps

Option A: Run from source

git clone https://github.com/Quantrail-Data/CH-Ops.git /opt/chops
cd /opt/chops
bun install
cp .env.example .env

Edit /opt/chops/.env:

SUPER_ADMIN_1=admin
SUPER_ADMIN_1_PASSWORD=your_strong_password_here
SUPER_ADMIN_1_EMAIL=you@example.com
ENCRYPTION_SECRET=paste_a_64_char_random_string_here
PORT=3000
NODE_ENV=production

SUPER_ADMIN_1, SUPER_ADMIN_1_PASSWORD, SUPER_ADMIN_1_EMAIL, and ENCRYPTION_SECRET are all required. The server exits on startup if any is missing. See Configuration for the full list.

Generate a strong ENCRYPTION_SECRET (it must be at least 32 characters):

openssl rand -hex 32

Build the frontend. The db:migrate step is optional, because the server also runs the migration at each start:

bun run db:migrate
bun run build

Test that it starts:

bun src/backend/server.js
# Should print: CHOps v<version> listening on http://localhost:3000
# Press Ctrl+C to stop

Option B: Run from binary

You can download a prebuilt binary instead of building your own. We publish prebuilt binaries for Linux, macOS, and Windows on the Releases page. Download chops-linux-x64 and skip to "Copy the binary and .env to your server" below (the copy step already renames it to /opt/chops/chops).

To build it yourself, do so on your development machine, and run bun install first to install the dependencies. See Building a Binary for details.

bun install
bun run build:binary:linux

Copy the binary and .env to your server:

ssh your-server 'sudo mkdir -p /opt/chops && sudo chown "$USER" /opt/chops'
scp chops-linux-x64 your-server:/opt/chops/chops
scp .env your-server:/opt/chops/.env

On the server:

chmod +x /opt/chops/chops
cd /opt/chops
./chops
# Should print: CHOps v<version> listening on http://localhost:3000
# Press Ctrl+C to stop

Step 3: Create a systemd Service

systemd is the process manager built into Linux. It starts CHOps automatically on boot, restarts it if it crashes, and lets you manage it with simple commands.

Create a dedicated system user (no login, no home directory, only to run the service):

sudo useradd --system --no-create-home --shell /usr/sbin/nologin chops
sudo chown -R chops:chops /opt/chops

Create the service file:

sudo nano /etc/systemd/system/chops.service

For source installs:

[Unit]
Description=CHOps - ClickHouse® Administration Dashboard
Documentation=https://github.com/Quantrail-Data/CH-Ops
After=network.target
 
[Service]
Type=simple
User=chops
Group=chops
WorkingDirectory=/opt/chops
ExecStart=/usr/local/bin/bun src/backend/server.js
Restart=on-failure
RestartSec=5
 
# Environment file
EnvironmentFile=/opt/chops/.env
 
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/chops/data
PrivateTmp=true
 
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=chops
 
[Install]
WantedBy=multi-user.target

For binary installs:

[Unit]
Description=CHOps - ClickHouse® Administration Dashboard
Documentation=https://github.com/Quantrail-Data/CH-Ops
After=network.target
 
[Service]
Type=simple
User=chops
Group=chops
WorkingDirectory=/opt/chops
ExecStart=/opt/chops/chops
Restart=on-failure
RestartSec=5
 
# Environment file
EnvironmentFile=/opt/chops/.env
 
# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/chops/data
PrivateTmp=true
 
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=chops
 
[Install]
WantedBy=multi-user.target

What each setting does:

  • After=network.target: wait for networking before starting.
  • User=chops: run as the dedicated user, not root.
  • Restart=on-failure: restart automatically if CHOps crashes.
  • RestartSec=5: wait 5 seconds between restarts.
  • EnvironmentFile: loads your .env variables into the process.
  • NoNewPrivileges=true: the process cannot gain extra permissions.
  • ProtectSystem=strict: makes the filesystem read-only except for allowed paths.
  • ReadWritePaths=/opt/chops/data: CHOps can only write to its data directory. This directory holds chops.db, crypto.salt, the export files, and the backups.
  • PrivateTmp=true: gives the service its own /tmp (isolated from other processes).

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable chops
sudo systemctl start chops

Check that it runs:

sudo systemctl status chops

You should see active (running). If it failed, check the logs:

sudo journalctl -u chops -f

Useful systemd commands

sudo systemctl start chops      # Start the service
sudo systemctl stop chops       # Stop the service
sudo systemctl restart chops    # Restart after config changes
sudo systemctl status chops     # Check if it is running
sudo journalctl -u chops -n 50  # View last 50 log lines
sudo journalctl -u chops -f     # Follow logs in real time

Logging

CHOps outputs structured JSON logs to stdout and stderr, which systemd's journald captures automatically. Each log entry is a single JSON line with timestamp, level, message, and context:

{"ts":"2026-05-18T10:30:00.000Z","level":"info","msg":"GET /api/alerts/rules 200 12ms","ctx":{"method":"GET","path":"/api/alerts/rules","status":200,"duration":12,"user":"admin","ip":"::1"}}

Viewing logs:

# Human-readable (default)
sudo journalctl -u chops -f
 
# JSON output for piping to jq or a log aggregator
sudo journalctl -u chops -o json | jq '.MESSAGE | fromjson'
 
# Filter by log level (errors only)
sudo journalctl -u chops -f | grep '"level":"error"'
 
# Last hour of logs
sudo journalctl -u chops --since "1 hour ago"

Log levels: set LOG_LEVEL=debug in your .env file to enable debug-level logging. The default is info. Levels: debug, info, warn, error.

What is logged: CHOps logs every API request (method, path, status, duration, username, IP), scheduler events (backup started, completed, failed, and alert notifications), server startup, and errors.

What is NOT logged by the request logger: passwords, tokens, request bodies, or response bodies. Unhandled errors and some diagnostic output still go directly to stdout and stderr, outside the structured logger, so treat the journal as content that may contain raw error detail.

Step 4: Install Caddy

Caddy is a web server that handles HTTPS certificates automatically (via Let's Encrypt). You do not need to generate or renew SSL certificates by hand. Automatic public certificates require the server to be reachable from the internet. For a private-network deployment, Caddy can instead issue certificates from its own local certificate authority, or serve plain HTTP (see Step 5).

Ubuntu/Debian

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

Verify the installation:

caddy version

Step 5: Configure Caddy

Edit the Caddyfile:

sudo nano /etc/caddy/Caddyfile

Replace the contents with:

chops.example.com {
    reverse_proxy localhost:3000
}

Replace chops.example.com with your actual domain name.

That is the entire configuration. Caddy will:

  1. Obtain a free TLS certificate from Let's Encrypt.
  2. Renew it automatically before it expires.
  3. Redirect HTTP (port 80) to HTTPS (port 443).
  4. Proxy all requests to CHOps on port 3000.

Restart Caddy to apply:

sudo systemctl restart caddy

Check that Caddy runs:

sudo systemctl status caddy

Open https://chops.example.com in your browser. You should see the CHOps login page with a valid HTTPS certificate.

Running without a public domain (private networks)

If CHOps runs on a private network with no public domain, you do not need Let's Encrypt. Caddy can serve an internal DNS name or an IP address with a certificate from its own local certificate authority. Use tls internal:

chops.internal.example {
    tls internal
    reverse_proxy localhost:3000
}

Replace chops.internal.example with the internal DNS name your users resolve to the server. You can also use the server's IP directly:

10.0.0.5 {
    tls internal
    reverse_proxy localhost:3000
}

With tls internal, Caddy generates its own root CA and issues the certificate locally. Browsers warn that the certificate is untrusted until you distribute and trust Caddy's root CA on client machines. It lives under Caddy's data directory, typically at /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt.

On a trusted internal network where TLS is not needed, you can have Caddy serve plain HTTP. Prefix the address with http:// (this also disables the automatic HTTP-to-HTTPS redirect):

http://chops.internal.example {
    reverse_proxy localhost:3000
}

You can also skip Caddy entirely and expose CHOps directly on its port (http://server-ip:3000), though a reverse proxy is still useful for TLS and access control. For browser-trusted certificates on an internal domain without open ports 80 and 443, use a DNS-challenge certificate, which requires a Caddy build that includes your DNS provider's plugin.

Advanced Caddy Configurations

Basic authentication in front of CHOps

Add a second layer of authentication at the reverse proxy level (in addition to CHOps's own login):

chops.example.com {
    basicauth {
        admin $2a$14$yourhashedpasswordhere
    }
    reverse_proxy localhost:3000
}

Generate a bcrypt hash for the password:

caddy hash-password --plaintext 'your-password'

IP allowlisting

Restrict access to specific IP addresses (for example, your office network):

chops.example.com {
    @blocked not remote_ip 203.0.113.0/24 198.51.100.42
    respond @blocked 403

    reverse_proxy localhost:3000
}

Custom headers

Add extra security headers, or override ones set by CHOps:

chops.example.com {
    header {
        X-Robots-Tag "noindex, nofollow"
    }
    reverse_proxy localhost:3000
}

Rate limiting at the reverse proxy

Caddy can rate-limit before requests reach CHOps:

chops.example.com {
    rate_limit {
        zone dynamic_zone {
            key {remote_host}
            events 100
            window 1m
        }
    }
    reverse_proxy localhost:3000
}

Note: the rate_limit directive requires the caddy-ratelimit plugin.

Run one CHOps instance only

Run exactly one CHOps process for one database. Do not put two CHOps instances behind a load balancer, and that includes Kubernetes replicas: 2 or more. CHOps keeps these items in the memory of its process:

  • password reset codes and reset tokens,
  • failed login counts and rate limits,
  • the alert scheduler and the Kubernetes health alerts.

Two instances cause these problems:

  • A reset code from one instance does not work on the other instance.
  • The login limits count only half of the attempts.
  • CHOps sends each alert two times.
  • SQLite does not support two writers.

For high availability, use a standby instance that you start only when the first instance stops. It must use the same database file and the same data/crypto.salt. For example, use a systemd service on a second server that you start manually, or a Kubernetes Deployment with replicas: 1 and strategy: Recreate.

Step 6: Verify the Deployment

Run through this checklist:

  1. Open your CHOps address (https://chops.example.com, an internal DNS name, or the server IP). It should load the login page. With a public domain you get a trusted certificate. With tls internal you see a certificate warning until Caddy's root CA is trusted on the client.
  2. If you configured public HTTPS, http://chops.example.com should redirect to HTTPS automatically.
  3. Log in with your super admin credentials.
  4. Go to Control panel > Cluster Management, add a ClickHouse® node, and test the connection.
  5. Reboot the server (sudo reboot) and check that CHOps comes back automatically.

Automated Database Backups

Add a cron job to back up the SQLite database daily:

sudo crontab -u chops -e

Add this line:

0 2 * * * cd /opt/chops && /usr/local/bin/bun run db:backup

This runs bun run db:backup every day at 2:00 AM. Each run saves two files to /opt/chops/data/backups/: chops-<timestamp>.db and chops-<timestamp>.salt. To restore, you need both files and the same ENCRYPTION_SECRET.

The binary does not have a db:backup subcommand. ./chops only starts the server. For binary installs, use the in-app Control panel > App Data Backup (admin), which uploads the database and the salt to an S3 storage profile. Or use a SQLite backup on a schedule:

0 2 * * * sqlite3 /opt/chops/data/chops.db ".backup '/opt/chops/data/backups/chops-$(date +\%Y\%m\%d).db'"

This needs sqlite3 on the host, and /opt/chops/data/backups/ must exist. The command does not copy data/crypto.salt, so keep a copy of that file too. It does not change after the first start.

To delete backups that are older than 30 days, add a cleanup line:

5 2 * * * find /opt/chops/data/backups -name "chops-*" -mtime +30 -delete

Updating CHOps

Source install

sudo systemctl stop chops
cd /opt/chops
git pull
bun install
bun run build
sudo systemctl start chops

Binary install

sudo systemctl stop chops
# Copy new binary to /opt/chops/chops
sudo systemctl start chops

The server runs the database migration at each start. The migration is safe on an existing database. At the first start of a new version, CHOps also encrypts any secret that an older version stored in plain text, such as the login signing keys, notification channel secrets, and storage profile keys.

Troubleshooting

Caddy shows "connection refused": CHOps is not running. Check sudo systemctl status chops and sudo journalctl -u chops -n 20.

Certificate errors: make sure your domain's DNS A record points to your server's public IP. Caddy needs ports 80 and 443 open to complete the ACME challenge.

"ENCRYPTION_SECRET must be at least 32 characters": your .env file has a ENCRYPTION_SECRET shorter than 32 characters. Generate a new one with openssl rand -hex 32.

"Missing required env" on startup: one of SUPER_ADMIN_1, SUPER_ADMIN_1_PASSWORD, SUPER_ADMIN_1_EMAIL, or ENCRYPTION_SECRET is unset. Under systemd, confirm they are in the EnvironmentFile. Under Docker, confirm they are passed to the container.

Permission denied errors: make sure the chops user owns the data directory: sudo chown -R chops:chops /opt/chops/data.

Port 3000 already in use: change the PORT in .env to something else (for example, 3001) and update the Caddyfile reverse_proxy line to match.

Docker Deployment

CHOps ships with a Dockerfile and docker-compose.yml for a containerized deployment. The image uses oven/bun:1.4.2-alpine with a multi-stage build. Published images are also on Docker Hub (quantrailadmin1/ch-ops) and GitHub Container Registry (ghcr.io/quantrail-data/ch-ops).

Quick Start

The bundled docker-compose.yml reads your configuration from a .env file (through env_file: .env), so create that file first.

# Clone or extract the project
cd CH-Ops
 
# Create your .env file and edit it
cp .env.example .env
nano .env
# Set ENCRYPTION_SECRET (openssl rand -hex 32), SUPER_ADMIN_1,
# SUPER_ADMIN_1_PASSWORD, and SUPER_ADMIN_1_EMAIL
 
# Build and run
docker compose up -d
 
# Check that it runs
docker compose logs -f

CHOps is now at http://localhost:3000. The SQLite database persists in the chops-data Docker volume.

Because the compose file reads from .env, put your values in that file. Exporting them only in your shell does not pass them to the container.

Build Only (without Compose)

docker build -t chops .
docker run -d \
  --name chops \
  -p 3000:3000 \
  -e ENCRYPTION_SECRET=$(openssl rand -hex 32) \
  -e SUPER_ADMIN_1=admin \
  -e SUPER_ADMIN_1_PASSWORD=your_strong_password_here \
  -e SUPER_ADMIN_1_EMAIL=you@example.com \
  -v chops-data:/app/data \
  chops

Environment Variables

Pass environment variables through the docker compose env file or docker run -e:

  • ENCRYPTION_SECRET (required): a random string that makes the key for stored credentials. You cannot change it after the first start.
  • SUPER_ADMIN_1, SUPER_ADMIN_1_PASSWORD, SUPER_ADMIN_1_EMAIL (required): the first super admin, seeded on first startup.
  • DISABLE_ENV_LOGIN=true (optional): disable the .env login fallback in production.
  • PORT=3000 (default): the HTTP port inside the container.
  • LOG_LEVEL=info (default): debug, info, warn, error.
  • FRONTEND_LINK (optional): the address that users open. CHOps uses it in the one-time links that set a password.
  • TRUST_PROXY (optional): the number of reverse proxies in front of CHOps.

The bundled docker-compose.yml loads your whole .env with env_file: .env, so every variable in the file reaches the container, including SUPER_ADMIN_1_EMAIL and FRONTEND_LINK. It then sets NODE_ENV=production and PORT=3000 in its own environment: section, which override anything in .env. Because it reads from the file, put your values in .env rather than only in your shell.

Data Persistence

Inside the container, CHOps stores the SQLite database at /app/data/chops.db and the encryption salt at /app/data/crypto.salt. The docker-compose.yml maps /app/data to a named volume (chops-data), so the data survives container restarts and rebuilds.

To back up the database, stop the container first. A copy of a running database can miss data that is still in the WAL file. Copy the salt file too:

docker stop chops
docker cp chops:/app/data/chops.db ./chops-backup.db
docker cp chops:/app/data/crypto.salt ./chops-backup.salt
docker start chops

Updating

docker compose down
docker compose build --no-cache
docker compose up -d

A rebuild does not delete the database volume, so your data, users, alerts, dashboards, and cluster config persist across updates.

Health Check

The container has a built-in health check that pings /api/health every 30 seconds. Check the status with:

docker inspect --format='{{.State.Health.Status}}' chops

Image Details

  • Base: oven/bun:1.4.2-alpine (Alpine Linux and the Bun runtime).
  • Multi-stage build: dependencies and the frontend build in stage 1, a slim runtime in stage 2.
  • Non-root user (chops) for security.
  • Includes: the built React frontend, the backend source, the shared source (src/shared), production node_modules, and the docs.
  • Health check: /api/health answers when the server runs. /api/ready also checks that the database can be read.

Kubernetes Deployment

The repository has example manifests in its root folder: configmap.yaml, statefulset.yaml, service.yaml, and ingress.yaml. They run one CHOps pod with a persistent disk for /app/data.

  1. Put the image in a registry that your cluster can pull from. Then set the image: field in statefulset.yaml. The manifest names the published image quantrailadmin1/ch-ops:<version>, with the version of this release from version.json (for example quantrailadmin1/ch-ops:1.4.0). The tag must match the CHOps release that you deploy.

  2. Make a Secret named chops-secrets with the four required values:

    kubectl create secret generic chops-secrets \
  --from-literal=ENCRYPTION_SECRET="$(openssl rand -hex 32)" \
  --from-literal=SUPER_ADMIN_1=admin \
  --from-literal=SUPER_ADMIN_1_PASSWORD=your_strong_password_here \
  --from-literal=SUPER_ADMIN_1_EMAIL=you@example.com

    Keep a copy of ENCRYPTION_SECRET. A new value makes the stored passwords unreadable.

  3. Apply the manifests:

    kubectl apply -f configmap.yaml -f statefulset.yaml -f service.yaml

  4. Change the host name in ingress.yaml. Then apply it with kubectl apply -f ingress.yaml. The ingress example is for the NGINX ingress controller.

What the manifests set:

  • replicas: 1. Do not raise it. See Run one CHOps instance only.
  • DB_PATH=/app/data/chops.db on the persistent disk. crypto.salt is also in /app/data.
  • EXPORT_DIR=/app/exports on an emptyDir volume with an 8Gi limit. Export files do not survive a pod restart. Keep EXPORT_MAX_TOTAL_BYTES below the volume limit.
  • A liveness probe and a startup probe on /api/health, and a readiness probe on /api/ready.