Architecture & Cloud Economics · Concurrency
Latency spikes, random timeouts and capacity that sits idle while being throttled are usually not capacity problems. They are what happens when fast, user-facing work queues behind slow background work — and the fix is normally a configuration change, not a bigger cluster.
availableProcessors() and your container has no CPU limit, you are sizing for the whole node. We show how to check this in about thirty seconds.Who this is for: CTOs and engineering leaders whose platform does meaningful background processing — media encoding, ML inference, document generation, data exports — alongside interactive traffic.
A media platform we worked on ran three kinds of background work: text-to-speech, video processing with FFmpeg, and transcription. Early on, one shared executor handled all three. It worked, and it was the right call at the time — there was no contention to solve.
Then traffic grew, and the shape of the problem changed:
This is head-of-line blocking, and its defining property is that the affected work is not the work that is slow. Your TTS endpoint is fast. Its code is fine. Its p99 is terrible, because it is standing in a queue behind something else entirely.
That distinction matters commercially, because it explains why buying capacity feels like it helps and then stops helping. More servers do drain the queue faster, so the symptom softens. But you are now paying to keep a structurally inefficient allocation running, and the cost scales linearly with traffic while the underlying contention never goes away.
Aggregate CPU utilisation looks healthy — often suspiciously low — because the bottleneck is a pool's thread count, not the machine. Per-endpoint latency looks fine at p50. The damage concentrates in p99 and in timeouts, which get written off as network blips or client issues. Meanwhile the fix keeps getting deprioritised because nothing is technically "down".
You can usually tell without instrumentation. If four or more of these are true, workload contention is worth an afternoon of investigation:
When latency spikes hit, the reflex is to give each workload its own pool:
# the "just add more pools" instinct
TTS Pool
Video Pool
Transcription Pool
Image Pool
PDF Pool
Cache Warming Pool
...
This does remove head-of-line blocking. It also creates six independent things competing for one CPU allocation, each sized by guesswork.
The blocking goes away, which is why teams stop here and declare victory. What replaces it is harder to see and harder to debug:
Isolation is not free — it costs tuning surface and cognitive load. It pays for itself only where two workloads genuinely have different resource profiles. Splitting workloads that behave identically buys you nothing and bills you for the complexity.
The useful line is not the feature boundary. It is the workload boundary: the line between jobs with similar resource and latency profiles and those without.
Incoming Requests
│
┌────────────┴────────────┐
│ │
TTS Workload Media Workload
│ │
▼ ▼
TTS Executor Media Processing Executor
│ │
Short jobs, bounded Long jobs, CPU-saturating
│ │
┌─────────┴─────────┐
│ │
FFmpeg Transcription
Two pools, not five. FFmpeg and transcription share one because their resource behaviour is the same — splitting them adds overhead without adding isolation.
The reasoning behind each decision:
This is the highest-yield thing in this article, and it takes half a minute to check.
Almost every Java service sizes its pools from the core count:
// This line means something different depending on your Kubernetes manifest. int cores = Runtime.getRuntime().availableProcessors(); executor.setCorePoolSize(cores / 2);
Sound, portable, and correct — but only if the container has a CPU limit.
Since JDK 10 the JVM is container-aware: availableProcessors() reads the cgroup CPU quota rather than the host's core count. That is the behaviour everyone assumes. The part people miss is the fallback — with no CPU limit set, there is no quota to read, so the JVM reports the entire node.
On a 64-core node, a pod with no limits.cpu sizes that pool to 32 threads. Those 32 threads then run CPU-saturating work inside a pod the scheduler may only be prepared to give two cores' worth of time to. The result is heavy CFS throttling: threads are runnable, the quota is exhausted, and everything in the pod — including its health checks — stalls in a way that looks nothing like a thread pool problem.
Two commands tell you where you stand:
# What the JVM will actually see $ kubectl exec deploy/your-service -- nproc # What you have actually granted it $ kubectl get deploy/your-service \ -o jsonpath='{.spec.template.spec.containers[0].resources}'
If nproc reports your node's core count instead of your intended budget, every pool sized from it is wrong — and has been since the day it was written.
Adding resources.limits.cpu makes the quota real, which makes availableProcessors() honest, which makes every pool derived from it correct — without touching application code. It is rare to find a change with this ratio of blast radius to effort.
One caveat worth knowing, because it catches teams after they set the limit: native libraries generally do not respect cgroups. FFmpeg's default threading and ONNX Runtime both size their internal thread pools from the host's core count regardless of your quota. The quota still caps their total CPU consumption, so the host stays protected — but inside the container you get context-switch churn rather than clean scaling. Capping the number of concurrent child processes is what actually helps; the CPU limit is what stops the container from affecting its neighbours.
Getting this classification wrong is the most common cause of a pool that is sized confidently and still wrong.
| Profile | Examples | Sizing | Queue & rejection |
|---|---|---|---|
| CPU-bound, long-running | Video encoding, image processing, local ML inference, PDF rendering | Fixed, at or below your CPU budget. Keep core and max equal. | Short queue. Reject early and visibly. |
| I/O-bound, short | Outbound API calls, database reads, cache and queue operations | Can exceed the core count — threads are waiting, not computing. | Deeper queue is acceptable; jobs drain quickly. |
| Mixed or unknown | Anything that sometimes blocks and sometimes computes | Measure before deciding. If clearly 80/20, treat it as the dominant profile. | Start conservative; widen using production data. |
Speed and cost are different axes. Local text-to-speech is a good example — a job finishes in one to three seconds, which reads as lightweight, but it is neural inference: it is brief precisely because it is saturating CPU while it runs. Sizing it like an I/O-bound workload, at two or three times the core count, oversubscribes the machine at exactly the moment it is busiest. The question is never "how long does it take?" It is "what is it doing while it takes that long?"
Reasonable question, and the honest answer is: for one of these categories, largely yes — for the other, no, and reaching for them there will cost you.
Virtual threads solve thread scarcity. Where a task spends its life blocked on I/O, a platform thread is expensive and idle, and Loom removes that waste elegantly. If your bottleneck is "we cannot afford enough threads to wait on all these calls", this is the right tool.
But for CPU-bound work, threads were never the constraint. Making them cheaper does not make cores cheaper. Worse, the idiomatic executor is unbounded:
// Unbounded by design — there is no queue and no rejection policy.
Executors.newVirtualThreadPerTaskExecutor();
Swapping this in for a bounded pool silently deletes your backpressure.
Drop that in place of a fixed pool that fronts video encoding, and two hundred queued jobs become two hundred concurrent encodes. Your clean "queue is full, retry shortly" rejection is gone; instead every job in flight thrashes and times out together. You have traded a graceful, observable failure for a total one.
If you want Loom's ergonomics on this kind of work, keep the bound explicitly — a semaphore, or a concurrency-limited executor. The bound is the design decision. The thread type is an implementation detail.
The most commonly deleted piece of a good design is the rejection policy, usually because a rejected job looks like a bug in a dashboard.
It is the opposite. An unbounded queue does not add capacity; it converts a fast, visible failure into a slow, invisible one. Work accumulates, latency climbs past every timeout upstream, and the jobs still get dropped — just later, after you have paid to hold them in memory and after your users have already given up.
Bounded + reject → fails in 50ms, caller sees a clear signal,
retry or shed load upstream
Unbounded queue → fails in 90s, caller has already timed out,
memory grows, cause is three layers away
Same outcome for the job. Very different outcome for the system and the on-call engineer.
Two practical notes. First, make the rejection message say which pool rejected and why — a generic executor exception costs real minutes at 3am. Second, check what your framework does to it: Spring's ThreadPoolTaskExecutor, for instance, wraps your rejection in a TaskRejectedException whose message is generic, so a carefully-worded message survives only on the exception's cause. If something surfaces that to a user or a log, it has to unwrap it.
We are deliberately not going to tell you that this saves a specific number of dollars per month. Anyone quoting you a figure before looking at your workload is guessing, and you should discount it accordingly.
What we can give you is the arithmetic. The saving from isolation comes from released overprovisioning — capacity you are buying to absorb latency spikes that are self-inflicted rather than demand-driven.
# Run this against your own numbers overprovision_factor = peak_provisioned_capacity / peak_utilised_capacity # Typical pattern: provisioned at 2.5-3x to absorb contention spikes, # while true utilisation at peak sits far below the limit. # Isolation lets you run closer to real utilisation. recoverable = monthly_compute_spend x (1 - target_factor / current_factor)
If you cannot fill in peak_utilised_capacity confidently, that gap is itself the first finding.
The metrics that tell you whether the design is working — and which most teams are not yet emitting:
| Metric | What it tells you | What "good" looks like |
|---|---|---|
| Queue depth, per pool | Whether a pool is a bottleneck or just busy | Near zero at steady state; spikes drain quickly |
| Active threads vs pool size | Whether sizing matches demand | Headroom at p50, saturation only at genuine peak |
| Rejection count | Real overload, distinct from slowness | Zero normally; non-zero and visible under true overload |
| Fast-endpoint p99 vs background volume | Whether isolation actually holds | Flat — the correlation should disappear entirely |
| CFS throttled seconds | Whether you are fighting your own CPU limit | Near zero; anything sustained means sizing is wrong |
That fourth metric is the one to watch. Before isolation, fast-endpoint p99 tracks background job volume. After, the correlation should vanish. If it does not, the isolation is not real — and that is a far more useful signal than any cost estimate.
Week 1 — Measure. Map workloads to actual resource profiles using execution-time distributions and CPU behaviour under load. Audit container CPU limits and what availableProcessors() reports in each one. This week usually produces at least one genuine surprise.
Week 2 — Design and stage. Consolidate to workload-aligned pools. Size against real CPU budgets. Add rejection policies with messages that name the pool. Deploy to staging under representative load.
Week 3 — Instrument. Queue depth, active threads, rejections, throttling. Let the metrics contradict the design if they are going to — this is the point of doing it in this order.
Week 4 — Tune on production data. Sizing that is right for one platform is wrong for another running similar code, because the workload mix differs. Theory sets the starting point; production sets the number.
Good architecture here does not look like more pools or more infrastructure. It looks like a team that can explain where its contention comes from and has added isolation only where it pays for itself.
So when latency starts rising, queues start growing, or background jobs start behaving differently week to week, the first question should not be "what do we add?" It should be:
Answer that honestly and the fix that emerges is almost always smaller, cheaper and more durable than another layer of infrastructure would have been.
Run the two kubectl commands in the container-sizing section, then plot p99 latency on one fast endpoint against background job volume for the last thirty days. If the two move together, you have contention. Both take under an hour and neither requires us.
Usually both, but weighted towards configuration. Executor definitions and resource limits carry most of the benefit. It is rarely a rewrite, which is what makes the risk profile attractive — changes are small, independently reversible, and verifiable in staging.
The principles do. Head-of-line blocking, workload profiling, bounded concurrency and backpressure are language-independent. The mechanics differ — Go's goroutines with semaphores, Node's worker pools and event-loop blocking, Python's GIL and process pools each have their own traps — but the diagnostic questions are identical.
Frequently, yes, and this article is deliberately specific enough to start. Teams typically bring us in for one of two reasons: nobody has uninterrupted time to do the measurement properly, or the system has enough interacting workloads that the analysis is genuinely hard. If it is the first, an internal engineer with a clear week is often the better answer.
That is a legitimate option and sometimes the right one — it buys isolation by construction. It also changes your cost model from provisioned to per-invocation, adds cold-start latency, and caps execution duration in ways long encodes often exceed. Worth evaluating, but it is a different architecture with different economics rather than a drop-in fix.
Typically two to four weeks of focused work, largely parallel to feature delivery since it touches configuration and infrastructure rather than product code. The measurement phase is entirely non-invasive.
We audit thread pool and executor design, identify exactly where workloads contend, and implement workload-aligned isolation with the instrumentation to prove it worked. If the audit finds nothing meaningful, we will tell you that — it is a better outcome than a project you did not need.
What a review covers:
availableProcessors() check across services