Yesterday evening, every one of my internal services and the software I host for my clients became unreachable for several hours. A saturated Postgres database, a high-availability setup that did not play its part, and a Scaleway incident on top. Here is the autopsy, no filter, because an outage you tell honestly is worth more than one you paper over.
The trigger, or rather the triggers
At 10:15 PM (Paris time), Scaleway lost a node in its block storage cluster in fr-par-1 (incident bsp2y5fysy9w). On the infrastructure side the effect was immediate: virtual machines stuck, and above all managed databases frozen on storage that had become unreachable.
At the very same moment, on my end, I had kicked off a batch of contact-enrichment jobs on Argos, my CRM. That kind of batch opens a lot of Postgres connections, and fast. Was it the batch that saturated the pool, or was the database already suffering from the Scaleway storage? I do not have a clean answer, and that is exactly the trap of correlated incidents: two plausible causes landing in the same window, and a doubt that only lifts after the fact. What I do know is that the two fed off each other. A database that responds poorly, connections piling up instead of being released, a batch that keeps asking for more. The pool hits its connection limit, and from there no application can open a single one. Everything falls in cascade.
The real problem: an HA setup that wasn’t one
My database ran in high availability, two nodes. Reassuring on paper. Except that both nodes lived in the same datacenter, fr-par-1. When the block storage layer of fr-par-1 went down, my two replicas left together, in the same second. That is the lesson I hold onto the most: redundancy only protects against what it isolates. Mine covered the failure of a process or a machine, not that of an entire zone. I had taken out insurance against one risk, not against all of them.
Worse, restarting the database was impossible. The managed Postgres engine stayed hooked to the failing storage. I could kill the zombie connections and restart my applications all I wanted, the final recovery no longer depended on me: it was suspended on Scaleway’s repair. Finding yourself a spectator of your own outage, with no lever at all, is the most unpleasant moment of the evening.
Getting out of it
Things unblocked during the night, by combining three things. I cut the connections left open on the Postgres side, I restarted the applications to start again from clean pools, and Scaleway eventually repaired the storage node. That last point was the decisive factor: without it, the rest was useless. My first two actions only set the stage so everything could come back cleanly once the infrastructure returned.
What I take away, and what I have already done
A good incident is the one that finally makes you do what you were putting off. I tackled three things right away.
PgBouncer in front of Postgres, first. A connection pooler I had wanted to set up for a long time without ever getting to it. It caps the number of connections and stops a batch from starving every other application. It is in place.
A read-only failover in another zone, next. My two HA nodes in the same datacenter were the heart of the problem. I added a read-only replica elsewhere, so that a Scaleway zone going down no longer takes everything with it in one block.
Observability on connections, finally. Watching pg_stat_activity and how full the pool is, to see saturation climb and fire an alert before it kills everything, instead of discovering it once the service is already down.
The lesson, in one sentence
An infrastructure dependency, however solid on paper, stays a single point of failure until you have explicitly isolated it. And redundancy you have never tested against the right failure mode is redundancy in name only.
If you have infrastructure you keep calling “highly available” without having checked against which kind of failure exactly, we can take a look together, rather than finding out on some Tuesday evening.