A good alert wakes you up for the right reason. A bad alert wakes you up so often that you stop reading it, and then misses the one that mattered. After years of tuning ClickHouse® alerting, I have landed on a simple model that produces good alerts: an alert is just a SQL query and a threshold. This post is how I set them up and, just as importantly, how I keep them quiet.
An alert is a query plus a condition
The model is deliberately simple. You write a SQL query that returns a number. You set a condition on that number. When the condition is true, you get an email. That is the whole thing.
Because the query is ordinary ClickHouse® SQL against your system tables or your own data, you can alert on anything you can measure. In CHOps alerting you define the query, the schedule, and the threshold, and it checks on that schedule and emails you when the condition trips.
Here is a real one. Alert me if more than ten queries took longer than five seconds in the last five minutes:
SELECT count() AS slow_queries
FROM system.query_log
WHERE type = 'QueryFinish'
AND event_time > now() - INTERVAL 5 MINUTE
AND query_duration_ms > 5000;Condition: slow_queries > 10. That is it. No special alerting language, no metric to pre-define. If you can write the query, you can alert on it.
The alerts I actually run
A handful of alerts cover most of what goes wrong.
Replication falling behind. A query against system.replicas for any replica where the queue is growing, with a threshold that ignores small transient lag and fires on sustained lag.
Part counts climbing. A query against system.parts that fires when any table crosses a part count that means merges are not keeping up. This is the early-warning alert I trust most, and I explained why in monitoring merges and mutations.
Disk filling up. A simple threshold on free space, because a full disk takes a ClickHouse® node down hard and you want a lot of warning.
Query failure spikes. A count of failed queries over a short window, which catches bad deploys and schema problems quickly.
Each of these is a query and a number. None of them is complicated.
The hard part is not firing too often
Writing alerts is easy. Writing alerts that you do not learn to ignore is the real skill. A few rules have served me well.
Alert on sustained problems, not instant blips. A single slow query is not worth an email. Ten slow queries in five minutes might be. Build the time window into the query so transient noise does not trip it.
Set thresholds from real data, not guesses. Look at what your cluster normally does, then set the threshold above the normal range, not at it. An alert that fires every day at peak load is not an alert, it is a daily annoyance.
Have fewer alerts that mean something. I would rather run five alerts I trust than fifty I skim. Every alert you add that fires for no reason makes the whole set less trustworthy.
Email is the OSS channel
In CHOps, alerts go to email, which is honestly enough for most teams. Email is reliable, everyone has it, and you can forward or filter it however you like. CHOps Pro adds Slack, Microsoft Teams, PagerDuty, and Google Chat for teams that want alerts routed into their existing on-call tooling, but you do not need them to run a solid alerting setup. A well-tuned email alert that fires only when something is genuinely wrong beats a noisy alert in any channel.
The natural source for alert queries is your monitoring dashboards. When a panel reliably shows a problem, promote it to an alert. The alerting feature page has more on scheduling and conditions.
Alerting is about protecting your attention as much as your cluster. Keep the set small, tune the thresholds to reality, and the alerts you get will be ones worth reading.
