One morning, a bunch of EC shops sharing a single server all tripped their monitors at once with "response timeout."
Let me be honest up front: the outage itself self-recovered in about five minutes. There's no heroic recovery scene here.
The real story is not those five minutes. It's how I chased down the creepy part: every usual suspect was innocent, yet the sites were down.
The stage: one shared web server (Apache prefork) hosting dozens of EC shops.
If you run something like this, your stomach already hurts. One shop's trouble drags the neighbors down with it.
One morning, every shop's monitor went red at once
The alerts fired together. Multiple shops, same instant, "socket timeout."
When every shop on a shared box dies simultaneously, your first suspect is "the foundation died" — CPU, memory, DB, that sort of thing.
Except when I looked, something was off.
The sites were down, but the server was perfectly healthy.
Every usual suspect was innocent
I ruled them out one by one.
- CPU: 84% idle. Bored.
- DB: not saturated even at peak. Not the bottleneck.
- Memory: no sign of the OOM killer. No process restarts.
Every classic load-trouble suspect came back clean.
And yet the monitor kept returning "socket timeout" — meaning the TCP connection was being established, but nothing came back after that.
That's when something nagged at the back of my mind.
If CPU and DB have headroom and memory is fine, then what's running out isn't "compute" — it's the slots that serve responses.
In Apache prefork terms, that's the number of worker processes.
My trusted error.log was, in fact, dead
"If we hit MaxRequestWorkers, error.log would say so."
I opened the global error.log expecting that line, and did a double take.
Nothing. Not a word.
Digging in, the culprit was the logging pipeline.
ErrorLog was piped to syslog, but the fixed tag I'd attached didn't match rsyslog's routing regex.
As a result, server-scope errors (server-wide, not per-VirtualHost) landed nowhere and vanished into thin air.
Even the one line I wanted most right now — "reached MaxRequestWorkers" — simply wasn't there.
This is the quiet but scariest part of the whole thing.
In the middle of an incident, the instrument I was relying on wasn't even plugged in.
(I fixed the routing side afterward so server-scope errors are retained. More on that at the end.)
I sorted the access log by %D and felt a chill
With error.log dead, I went back to the log that was still alive: the access log.
What saved me was the response-time field (%D, in microseconds).
Things invisible when you aggregate by request count show up the moment you sort by %D.
159 requests had taken 15+ seconds each. The worst held a worker hostage for 17 minutes (1040 seconds).
Seventeen minutes, for a single web request. Normally we live in milliseconds, or seconds at worst. This was minutes.
And all of them targeted one single shop, from one single IP.
Count-based aggregation will never float this kind of attacker to the top.
That was the biggest lesson of the day, so I'll put it in bold.
To find a stall, sort by time spent (%D), not by request count.
The culprit wasn't "the one making lots of requests"
The IP turned out to belong to an overseas commercial proxy / scraping vendor — spoofing an old Chrome UA, pretending to be a browser.
And here's the nastiest part.
It sent 217 requests over about 8 minutes — roughly 0.45 req/sec. Low, if anything.
That slid right through the connlimit/hashlimit I had in place (drop above 20 concurrent connections or above 1 req/sec).
Ordinary rate limits are built to catch "too fast," so they were useless against "deliberately slow."
What was it doing?
It fired requests promptly, but deliberately read back the response only in tiny dribbles.
A worker can't be released until the response is fully received, so each one held a slot for 10, 15 minutes.
Do that with many requests at once, and prefork workers fill up one by one until you hit the ceiling (128) — and at that moment every shop's slots are gone.
The foundation (CPU/DB) stays idle the whole time while the sites go down. That's the creepy part, explained.
This is the response-side version of the well-known slowloris (which dribbles out requests to occupy workers).
It's the family known as low-and-slow — it hits you with "slow," not "fast."
There's no flashy traffic spike, so staring at load graphs won't reveal it.
What I did on the spot (stopgap)
Once the culprit was clear, I stopped the bleeding first.
The action was simple: immediately block that single IP.
Concretely, one DROP line at the top of the firewall (iptables). I backed up the pre-change state just in case, then added it, and confirmed the collateral shops were serving 200s again.
But let me be honest here.
This is just swatting the hand that's hitting you right now — not a permanent fix.
The vendor rotates IPs, so IP blocking inevitably becomes whack-a-mole.
And the commonly suggested mod_reqtimeout watches the case where the request side is slow (the original slowloris). The case here — the response-reading side being deliberately slow — can't be caught by a request-side timeout.
It's the same reason a simple rate limit got slipped.
The real fix is "a front layer that absorbs fast and slow," I think
From here on, this is not what I actually did, but the conclusion / the homework (I haven't deployed it yet, honestly).
As long as bare prefork Apache speaks dynamic content directly, the structure itself — "a worker is held until the client finishes reading the response" — never goes away.
So the sensible fix is to put a front layer that buffers the response in front of Apache.
A CDN or reverse proxy receives the full response first, then patiently babysits the slow client on its own.
That way the Apache worker is freed the moment it has flushed the response, and "slow clients" can no longer take slots hostage.
A complementary output-side / idle timeout (cutting off no-progress while streaming a response) helps too.
In short: let the front layer, where slots are plentiful, deal with slow clients — not Apache, where slots are precious. That's the structural answer, the way I see it.
So what did I actually learn
Not crypto, not algorithms — three humble but effective takeaways.
1. If you have CPU/DB headroom but still get socket timeouts, suspect worker/connection exhaustion.
Look at the free slots in the worker pool, not the load graph. Resources can be plentiful while "slots" run dry.
2. Hunt stalls by time spent (%D), not by request count.
Low-rate low-and-slow won't appear in count aggregation. Sort by %D and it finally shows its face.
3. Check that your monitoring logs themselves are alive — in peacetime.
"error.log happened to be dead exactly during the incident" is not funny. On a quiet day, make sure your instruments aren't silently dropping into the void. This one stung the most.
It's precisely because it isn't a flashy attack that it's so easy to miss.
Are the slots in your worker pool actually free right now?
Watch out for the customer who makes few requests but overstays forever.
