Emergent Failure: A Postmortem of the 2026-07-31 Global OOM Cascade

A host with 24 GiB of RAM and 4 GiB of swap froze twice in under two hours. The proximate cause was memory exhaustion. The interesting part is how three individually reasonable decisions — a fallback price, an eager risk loop, and a container with no memory ceiling — composed into a failure no single component predicted.

Verdict

On 2026-07-31 the primary host (fraotech-llc) entered a global out-of-memory cascade at 14:46 UTC and hard-locked at roughly 16:01, requiring a forced power cycle. The rebooted machine began failing again within minutes. Both freezes traced to a single service: the trading-platform backend. This was not a leak. It was not a misconfiguration in the usual sense. It was a two-bug interaction with a config gap, and it is the clearest current example of why complex systems fail at the seams rather than at the center.

Timeline

The shape of the failure is worth studying before the causes: the first kill was a Rust compiler, not the trading service. The trading service became the victim only after the host was already exhausted, and the restarts then turned a single exhaustion event into a self-sustaining loop.

Root cause: two bugs and a config gap

Bug 1 — the fallback price had no authority

The market-data layer resolves prices through a provider chain: a primary exchange feed, a secondary finance feed, and finally a simulator marked as the "last resort safety net." When the two real feeds are unreachable — rate limits, connection failures — the chain falls through to the simulator. The simulator does not fetch prices. It estimates them from hardcoded tables.

Its estimate for BTC is $45,000.

That constant is badly out of date. Real BTC was trading in the $62,000–$65,000 range. When the real feeds dropped out, the ticker broadcast BTC at $45,000, and the risk engine — which trusts whatever the ticker says — evaluated every open BTC position against that number. A stop-loss at $60,000 is comfortably above a $45,000 "price," so the engine concluded the market had crashed through it and ordered a liquidation. Every such trade "breached" at once because the same fake price hit all of them.

This is the cleanest statement of the gap: a fallback provider was functionally authoritative. It carried no quality, staleness, or authority signal, so downstream consumers could not distinguish a real price from a placeholder.

Bug 2 — the risk loop never said "already done"

The risk engine evaluates each ticker update (roughly once per second) against its in-memory matrix of active trades. For every breach it dispatches a liquidation. There is no dedupe, no cooldown, no "this trade is already being liquidated" marker.

So the same trade fired again, and again, every second, for as long as the fake price persisted. The numbers make it concrete:

• 536,074 liquidation events logged
• 54,110 distinct trade IDs
• the most-fired trade logged 19 liquidations

Each liquidation is dispatched as a background task. The execution path is throttled by a database semaphore capped at 25 concurrent operations; the dispatch path is not throttled at all. When tens of thousands of trades breach in the same second and re-breach the next, the task queue grows without bound. The container ballooned to 13.5 GiB — not from leaked memory, but from queued work.

Config gap — a container with no ceiling

The trading backend container runs with no memory limit; the container runtime reports Memory: 0. Its restart policy is unless-stopped. Together these compose into a crash loop: the kernel kills the container’s main process, Docker restarts it, the startup re-fires the storm, the kernel kills it again. Across three restarts the cadence was 40k, then 170k, then 86k liquidation events, each run consuming more memory before being reaped.

A memory limit would not have fixed the bugs. It would have contained them: the container would have been killed at its own ceiling instead of taking the host down with it.

Why it looked like a leak and was not

The peak grew across runs: 2.8, 6.3, 12.0, 13.5 GiB. That reads like a leak. It is not. One run survived 18 hours and 22 minutes at a peak of 4.5 GiB — a leak does not stay below its own lifetime. The growth across runs is the startup burst (load state, re-fire the storm) being reaped at progressively worse moments while the host is already drowning.

What the evidence says about the forensics

The reconstruction used four data sources worth institutionalizing.

1. journald boot separation. journalctl --list-boots cleanly separated the frozen boot from the current one, and the previous boot’s log held the entire story. The kernel’s OOM killer prints the full process table at each kill; sorting that table by RSS at 14:46:36 showed the compile workers and the trading backend in rank order.

2. The kill cadence. Every trading-backend death was exit code 137 (SIGKILL). systemd logged each container scope’s "Consumed X CPU time over Y wall clock, Z memory peak." Those lines are the evidence for when the burst happens: 13 GiB peak in 70 seconds on a cold boot versus 4.5 GiB over 18 hours of steady state.

3. Session transcripts as state. The orchestration that triggered this ran under the COSAW workflow: a main oversight session plus phase workers in isolated worktrees. Their transcripts survived the freeze and recorded the exact moment of failure — the main session’s last line before lockup was the detection that a phase pane died at 14:46:35, one second before the kernel fired.

4. Service logs at debug level. The trading backend logs at debug with tracing. The "storm" is 536,074 near-identical lines. Counting distinct trade IDs — sort | uniq | wc -l — is what turned a wall of noise into the number that proves the re-fire bug.

The systemic lesson

None of the three components was individually reckless.

• A fallback provider is standard practice.
• An eager, low-latency risk loop is the entire point of a risk engine.
• A container without a memory limit is the default configuration.

The failure emerged from their composition: a fallback with no quality signal, feeding a loop with no dedupe, running in a process with no bound, on a host whose headroom had already been consumed by unrelated parallel build work. Each layer assumed the next would catch it. None did. That is the gap, and it is exactly the shape of failure that the user-facing symptom — a frozen console, dead pings, no keyboard echo — does not explain.

The generalizable audit checklist:

The fix, in progress

The corrective work is being executed under the COSAW workflow as a dedicated main session with advisor review. The design direction, in brief:

1. Risk engine: dedupe and cooldown per trade; dispatch a liquidation once; move the trade out of the active set on dispatch.
2. Fallback guard: tag prices by provider and never let a non-authoritative price drive liquidation.
3. Memory ceiling: add a memory limit to the container so a future bug kills the container, not the host.
4. Build parallelism: bound concurrent phase compiles to protect host headroom.

Why this belongs in the record

This incident is the best live example we have of the thesis that complex systems fail at the seams. Every piece was reasonable; the composition was not. The purpose of this post is to make the failure reproducible — not the bug, but the pattern — so that future audits can ask the questions in the table above and catch the same class of gap before it requires a power cycle to find.

Engineering postmortem. Frao Technologies LLC.