Fixing an 800% Cloud Cost Spike from Runaway Multi-Agent AI

The phone call came with a familiar tone of panic. A startup founder, voice tight with stress, explaining that their cloud bill had just come in at 800% above normal. One month ago they’d deployed a shiny new multi-agent AI pipeline—the kind that promises to automate everything from go-to-market campaigns to bug fixes. Now their runway was evaporating in real-time.

“Our AI vendor says everything’s fine,” they told us. “The logs look normal. But we’re hemorrhaging money and nobody can tell us why.”

We took the case. What we found was straight out of a horror movie: their agents had turned. Not maliciously—worse. They’d started feeding on each other, trapped in endless loops, multiplying their actions until the system was consuming itself from the inside.

28 days. That’s how long the infection spread before anyone noticed.

Patient Zero

The startup was in an impossible position. Costs were spiraling. They needed answers yesterday. But they were staring at a black box—and something inside it had gone very wrong.

Their vendor had built what looked like a sophisticated multi-agent system. Agents handled GTM automation, bug triage, feature releases—all the things that sound great in a pitch deck. The code had been tested. Every agent worked perfectly in isolation. The vendor was adamant: “Everything is functioning within expected parameters.”

That’s the thing about outbreaks. Everything looks fine until it doesn’t.

What the vendor didn’t mention: they’d never tested what happens when all those “perfectly functioning” agents start talking to each other. At scale. In production. Nobody had watched what happened when you put them all in the same room and let them loose.

We inherited a containment nightmare:

Blind investigation. The vendor refused to share code until we could prove there was a bug. Classic catch-22—we needed to find the infection without being allowed inside the lab.

Log chaos. The system generated 1.4 million log entries per hour. Unlabeled. Unstructured. Finding patient zero in that noise would be like tracking a single sneeze through a packed stadium.

Multi-cloud spaghetti. Services spread across providers, hooks triggering hooks, dependencies nested so deep that tracing a single operation felt like tracking a virus through a population.

Hunting the Infected

Traditional consulting would have thrown a team of analysts at this for weeks. The client didn’t have weeks. The infection was spreading—costs doubling, tripling—and the board was starting to ask questions.

So we did something different.

Our CTO Segev pulled in one of their developers for context, and then we built a hunter: an AI agent specifically designed to track anomalies through their logs. We gave it a sliding context window to handle the massive data volume—it could grab fresh records continuously, analyze patterns, and generate hypotheses in real-time.

Then we built a second agent to verify the kills. Its job was to take every hypothesis from the first agent and test it against baseline data from before the outbreak.

We let them loose. For hours, these two agents swept through the data, tracking patterns, eliminating false positives, mapping the spread of whatever was eating their system alive.

And then we found it.

The Horde

1.4 million extra database writes. Per hour.

Not new records—that would have been obvious. These were updates to existing records, invisible unless you knew exactly what to look for. Each write triggered hooks. Those hooks triggered other processes. Those processes made more writes. Each action spawning more actions, spreading through the system like a contagion.

The agents had become zombies. Still running. Still executing their code perfectly. But caught in loops, mindlessly repeating the same actions, infecting each other with triggers that would never stop.

We could see the damage, but not the source. The database was a document store with no query logging. We needed to trace the infection back to patient zero.

We instrumented every write operation with a hook that captured the full call stack—a breadcrumb trail showing exactly how each outbreak started and spread.

And then the picture came into focus.

The Outbreak

Here’s how the infection spread:

A GTM request comes in. Agent A generates a market report and updates the target list. That update triggers Agent B, which runs analytics and updates the market strategy. That update triggers Agent A again to run a comparison analysis. Which generates a new report. Which updates the target list. Which triggers Agent B.

Forever. An endless loop. Agents triggering agents triggering agents, each one doing exactly what it was programmed to do, none of them aware they’d become part of a horde.

This wasn’t one outbreak. We found multiple infection chains, all spreading simultaneously, each generating thousands of writes per minute. Every agent was doing exactly what it was designed to do. The code was correct. The tests passed.

But nobody had asked: “What happens when we put them all together?”

This is the horror of multi-agent AI systems. Unit tests mean nothing when your agents can infect each other in production. Each agent is locally healthy and globally catastrophic. The outbreak doesn’t start until they start communicating—and by then, it’s already spreading.

Containment Protocol

Once we had proof, the vendor finally opened up the codebase. Now we could see exactly where the infection started—and more importantly, how to build immunity.

We delivered seven containment patterns. Think of them as vaccines for your multi-agent system. Any organization running autonomous agents should treat these as survival basics:

1. Correlation IDs: Track the Bloodline

Correlation IDs are trace identifiers that follow every action through a multi-agent call chain, allowing an agent to detect when it’s already in an execution path and stop before creating a loop.

Every action carries a trace ID through the entire call chain. Before an agent acts, it checks: “Am I already in this execution path?” If it sees its own ID in the ancestry, it knows it’s been infected—and stops.

trigger_chain: [gtm_agent] → [analytics_agent] → [gtm_agent] ← QUARANTINE

Simple. Obvious in hindsight. Would have stopped every outbreak we found.

2. Debouncing: Slow the Spread

A 30-60 second quarantine window on document updates. Multiple writes to the same record within that window get coalesced into one. This alone would have contained 90% of the infection.

3. Circuit Breakers: Automatic Quarantine

Circuit breakers enforce hard limits on agent activity — maximum writes per minute, maximum chain depth — and automatically isolate agents that exceed thresholds.

Maximum writes per minute per agent. Maximum chain depth—no trigger chain longer than 5 hops. Automatic lockdown when thresholds approach. When a breaker trips, the system isolates the affected agents and waits for human intervention.

4. Semantic Change Detection: Stop the Redundant

Semantic change detection hashes every payload before writing and skips the operation if the new state is identical to the old state.

Before every write, hash the payload. If the new state equals the old state, skip the write entirely. Half the infected loops we found were agents mindlessly rewriting identical data—zombies going through the motions.

5. Traffic Control: Establish a Chain of Command

Traffic control adds a lightweight orchestrator that sequences agent work — agents request permission to act on a resource instead of reacting to every stimulus.

Add a lightweight orchestrator that sequences agent work. Agents request permission to act on a resource instead of just reacting to every stimulus. Think of it as the CDC for your agent ecosystem—nothing moves without clearance.

6. Full-Ensemble Stress Testing: Simulate the Outbreak

Full-ensemble stress testing puts all agents in the same staging environment under realistic load — the only way to catch feedback loops that are invisible to unit tests.

This is what actually failed here. Unit tests passed. Nobody had ever put all the agents in the same room under realistic load. That’s exactly where outbreaks start.

Build staging environments with all agents active. Run load simulations. Design chaos tests specifically to trigger feedback loops. Find the infections in the lab, not in production. If you can’t break it in staging, production will break it for you.

7. Cost Anomaly Alerts: Early Warning System

Cost anomaly alerts trigger at 110%, 150%, and 200% of baseline spend, with automatic scaling limits and kill switches to contain cost spikes before they become existential.

This outbreak burned money for 28 days before anyone noticed. That’s 27 days too long. Billing alerts at 110%, 150%, 200% of baseline. Automatic scaling limits. Kill switches. The earlier you detect the outbreak, the cheaper the containment.

Survival Guide

We contained the outbreak. We stopped the bleeding. We gave them the immunity patterns to make sure it never happens again.

The engagement finished under budget and ahead of schedule—a two-person team solving what could have taken weeks of traditional consulting. The client was impressed enough to bring us back: we’re now helping them reconfigure their entire infrastructure to be outbreak-proof from the ground up. Sometimes containing one disaster is just the beginning of a longer partnership.

But here’s the unsettling truth: the vendor wasn’t lying when they said everything was working as intended. Every agent was doing exactly what it was designed to do. The code was correct. The tests passed. Each agent, examined in isolation, was perfectly healthy.

The infection wasn’t in any individual agent. It was in the space between them—in the emergent behavior that nobody predicted because nobody tested for it. The outbreak started the moment they began communicating.

This is the horror of multi-agent AI systems. They’re not just code. They’re ecosystems. And ecosystems can turn on you in ways no single component would suggest.

For the startup, this was an expensive lesson—but a survivable one. They caught it before it killed the company. 28 days of infection, contained before it became terminal.

Not everyone is that lucky.

If you’re building multi-agent systems, or deploying them, or trusting a vendor who says everything is fine: test the ensemble. Monitor for outbreaks. Build the containment protocols before you need them.

Because when agents start talking to each other, you never know what they might become.

Technology Stack

Multi-Agent AI Cloud Infrastructure AI-Assisted Debugging Log Analysis