CPU is at 100% and all queries become slow under load

Your ClickHouse server uses 100% of its CPU. Queries that normally finish in seconds now take much longer. This happens when several queries run at the same time.

The server may have enough CPU cores to handle the workload. However, each query can request several execution threads. When many queries run together, the total number of active threads can become too high.

This is a thread oversubscription problem. It can increase scheduling overhead and reduce the CPU time available to each query.

Start by checking whether ClickHouse runs too many execution threads under load. Do not assume that high CPU use alone proves thread oversubscription.

This article explains how to inspect thread concurrency and control it with server settings.

Why it happens

ClickHouse uses multiple threads to execute queries in parallel. The max_threads setting controls the maximum number of query-processing threads that a query can use for relevant execution stages.

For example, consider a server with 16 CPU cores. Each query can use up to 16 threads when its workload and settings allow it.

Now suppose 20 queries run at the same time. If each query tries to use 16 threads, the workload can request up to 320 execution threads.

The server has only 16 CPU cores. It cannot run all 320 threads simultaneously on those cores. The operating system and ClickHouse must schedule the available CPU time across runnable work.

This example illustrates the risk. It does not mean that every query creates all its allowed threads or keeps them active at the same time.

Excessive concurrency can increase scheduling overhead. It can also make queries compete for CPU time, memory bandwidth, caches, and other resources. As a result, query latency increases across the server.

ClickHouse provides three settings to control query-thread concurrency:

• concurrent_threads_soft_limit_num sets an absolute soft limit for concurrent query-processing threads.
• concurrent_threads_soft_limit_ratio_to_cores sets a soft limit relative to the available CPU cores.
• concurrent_threads_scheduler selects the scheduling policy for sharing available thread slots.

These settings work together. The soft limit controls the concurrency budget. The scheduler controls how queries share that budget.

A soft limit is not the same as a hard limit on every thread in the server. Background tasks and other thread pools can also consume CPU resources.

The max_thread_pool_size setting controls the maximum number of threads in a server thread pool. It does not directly limit the number of execution threads that every query can request.

Therefore, reducing max_thread_pool_size is not the first fix for this problem. Start with query concurrency and max_threads.

How to confirm

Collect data while the problem occurs. A sample taken when the server is idle may not show the cause.

Check CPU and query activity

First, inspect the server's current CPU usage with your normal monitoring tools. Then check how many queries run concurrently.

Run this query:

SELECT
    count() AS running_queries
FROM system.processes;

This gives the number of queries currently listed in system.processes. It does not show the number of execution threads used by those queries.

Next, inspect the configured value of max_threads:

SELECT
    name,
    value,
    changed,
    description
FROM system.settings
WHERE name = 'max_threads';

This query shows the setting for the current session. It does not necessarily show the effective value for every user or query. Check the relevant user profile and query-level overrides.

Inspect server concurrency metrics

ClickHouse exposes server metrics through system.metrics. Look for metrics related to ConcurrencyControl and compare their values during normal operation and under load.

The exact metric names and meanings must be checked against the target release. Verify the relevant names in ClickHouse version before using them in dashboards or alerts.

Run this query to inspect available metrics:

SELECT
    metric,
    value,
    description
FROM system.metrics
WHERE positionCaseInsensitive(metric, 'ConcurrencyControl') > 0
ORDER BY metric;

If this query returns no rows, do not assume that concurrency control is inactive. The metric names may differ in your build. Check the metric names exposed by your target server.

Also inspect the server settings that control concurrency:

SELECT
    name,
    value,
    changed,
    description
FROM system.server_settings
WHERE name IN
(
    'concurrent_threads_soft_limit_num',
    'concurrent_threads_soft_limit_ratio_to_cores',
    'concurrent_threads_scheduler',
    'max_thread_pool_size'
)
ORDER BY name;

Use system.server_settings to check the effective server configuration. Compare these values with the defaults documented for your exact release.

Interpret the results

The following combination supports the oversubscription hypothesis:

• CPU usage remains near 100% during the incident.
• Many queries run at the same time.
• Queries become slower as concurrency increases.
• The server's concurrency-control metrics show thread-slot pressure, if the relevant metrics are available and verified.

No single observation proves the cause. Check query profiles, background activity, and other resource limits before changing production settings.

The settings

These server settings control query-thread concurrency and the global thread pool.

The defaults below come from ClickHouse master, commit 9f79cc1, dated 30 October 2026. The target version for this article is 26.9. Verify each default on your server before you apply a change.

How the soft limit works

The absolute and relative limits work together. When both limits are nonzero, ClickHouse uses the lower limit.

For example, consider a server with 16 CPU cores. The default ratio gives a soft limit of 32 query-processing threads when the absolute limit remains zero.

This does not mean that ClickHouse can run only 32 threads across the entire server. The limit applies to query-processing threads covered by concurrency control. It excludes some work, such as threads used to retrieve data from remote servers.

The limit is also soft. Each query can retain at least one thread when the limit is reached. A query can gain more threads when slots become available.

How the scheduler shares slots

The concurrent_threads_scheduler setting selects the scheduling policy. The default, max_min_fair, aims to distribute available CPU slots fairly among competing queries.

This matters when queries have different parallelism requirements. A large analytical query may benefit from many threads. A small dashboard query may need only a few.

The scheduler manages the available slots. It does not guarantee equal query latency or equal CPU consumption. Query complexity, data access, and other resource limits still affect performance.

You can change the scheduler at runtime without restarting the server. Keep the default unless measurements show a reason to test another supported policy.

How the global thread pool differs

The max_thread_pool_size setting limits the global thread pool. ClickHouse uses this pool for query processing and background operations.

A value of 10000 does not mean that ClickHouse runs 10,000 threads at once on the CPU. It is a maximum pool size, not a CPU concurrency target.

Do not lower this setting simply because CPU usage is high. A low value can prevent ClickHouse from creating threads that it needs for queries or background operations.

How to fix it

Start with a controlled change. Keep the relative soft limit near its default. Then reduce query parallelism if your workload creates too much CPU contention.

Test each change separately. Use the same representative workload before and after each change.

Change 1: Keep the relative soft limit near the default

Create this file:

/etc/clickhouse-server/config.d/concurrency-control.xml

Use the following configuration:

<clickhouse>
    <concurrent_threads_soft_limit_num>0</concurrent_threads_soft_limit_num>
    <concurrent_threads_soft_limit_ratio_to_cores>2</concurrent_threads_soft_limit_ratio_to_cores>
</clickhouse>

The absolute limit remains zero. The relative limit remains two times the available CPU cores.

This is a sensible starting point because it preserves the default concurrency budget. Do not lower the ratio without evidence that the default allows excessive contention.

Restart required: No. Reload the server configuration using your normal configuration-management process. Confirm that the server has applied the values.

If your server already uses these values, do not change them just to make a change. Proceed to the next step.

Change 2: Lower max_threads in the relevant settings profile

The max_threads setting controls the maximum query-processing parallelism for an individual query. It is a user or query setting, not a server setting.

Use a settings profile to reduce parallelism for the affected users. For example, create or update a profile in:

/etc/clickhouse-server/users.d/production-profile.xml

The following example sets max_threads to 8 for a profile named production:

<clickhouse>
    <profiles>
        <production>
            <max_threads>8</max_threads>
        </production>
    </profiles>
</clickhouse>

Assign the production profile to the intended users through your existing user configuration. Do not create a duplicate profile if one already exists.

Why choose 8? It is a starting value for testing, not a universal recommendation. It reduces the maximum parallelism of each query and may allow more concurrent queries to make progress.

For a server with many cores, a value of 8 may be too low. For a server with limited CPU capacity and many concurrent analytical queries, it may be a useful starting point.

Measure query latency and throughput before you choose a final value.

Restart required: Usually no, if the server reloads the updated user configuration successfully. Confirm that the profile change has taken effect. Existing sessions may require attention, depending on how your users and profiles are configured.

If the server does not apply the change through configuration reload, follow your deployment procedure to activate the configuration.

Change 3: Keep the scheduler at its default

Keep concurrent_threads_scheduler set to max_min_fair unless testing identifies a better policy.

You can include the setting in the server configuration if you want to make the intended value explicit:

<clickhouse>
    <concurrent_threads_scheduler>max_min_fair</concurrent_threads_scheduler>
</clickhouse>

The default policy is a reasonable starting point for mixed workloads. Do not change the scheduler and the thread limits at the same time. Otherwise, you may not know which change caused the result.

Restart required: No.

Change 4: Leave max_thread_pool_size unchanged initially

Keep the global thread-pool size at its current value unless you have evidence of thread-pool exhaustion.

If you need to configure it explicitly, use:

/etc/clickhouse-server/config.d/thread-pool.xml

<clickhouse>
    <max_thread_pool_size>10000</max_thread_pool_size>
</clickhouse>

The value 10000 preserves the supplied default. It is not a recommendation to increase or decrease the pool.

Restart required: Yes. Plan a restart if you change this setting.

The global thread pool serves more than query execution. A lower limit can delay other work, including background operations. Investigate pool exhaustion before changing the limit.

How to verify the fix

After the configuration reload, run the same workload that caused the incident. Keep the query mix, data, and concurrency as similar as possible.

Check the effective server settings:

SELECT
    name,
    value,
    changed,
    description
FROM system.server_settings
WHERE name IN
(
    'concurrent_threads_soft_limit_num',
    'concurrent_threads_soft_limit_ratio_to_cores',
    'concurrent_threads_scheduler',
    'max_thread_pool_size'
)
ORDER BY name;

Confirm that the server reports the expected values.

Next, check the query setting in a session that uses the intended profile:

SELECT
    name,
    value,
    changed
FROM system.settings
WHERE name = 'max_threads';

Confirm that the result reflects the intended setting for that session. If it does not, check profile assignment and query-level overrides.

Compare these measurements before and after the change:

• CPU usage during the same workload.
• Query latency, including median and high-percentile latency.
• Query throughput over the same period.
• The number of running queries.
• The verified concurrency-control metrics in system.metrics.

Use your existing query logs and monitoring dashboards to compare latency and throughput. Check the relevant metric names against the target release before you add alerts.

A successful change should reduce harmful CPU contention while preserving useful parallelism. CPU usage may remain high if the server continues to process more useful work. That alone does not mean the fix failed.

If latency remains high, investigate query plans, expensive joins, data scans, background merges, and other resource bottlenecks. Do not keep lowering max_threads without measuring the effect.

Risks and trade-offs

The main risk is setting the soft limit too low. ClickHouse may leave CPU capacity unused when queries cannot obtain enough slots to use the available cores efficiently.

This can reduce throughput and increase the latency of large analytical queries. It can also make a server appear underutilized even when queries have work to do.

A lower max_threads value has a similar trade-off. It can improve concurrency for a mixed workload, but it can slow down queries that benefit from high parallelism.

Do not confuse thread concurrency with query admission. These settings do not replace limits on the number of simultaneous queries.

Also check whether the server has a workload configuration that defines CPU resources. When a CPU resource is declared, workload scheduling can replace the server-level concurrency controls for participating queries. Check your workload configuration before changing these settings.

Summary

• Check CPU usage, running queries, and concurrency-control metrics before changing settings.
• Keep concurrent_threads_soft_limit_ratio_to_cores near its default of 2.
• Reduce max_threads in the relevant settings profile when queries create excessive CPU contention.
• Do not lower max_thread_pool_size unless you have evidence of thread-pool exhaustion.
• Compare latency, throughput, and CPU usage after each change. Avoid a soft limit that leaves useful CPU capacity idle.

References

ClickHouse Server Settings source code