W6W runs a platform for central API management and integration infrastructure. Their users connect to third-party apps through code, a CLI, curl, or whatever else fits the way they already work, then compose those connections into functions and workflows. The platform supports both self-hosted and cloud integrations, ships with a package of provided apps, and lets users bring their own or pull from open source. Its usefulness scales with its catalog — and the catalog is what brought them to us. W6W contracted ProductiveHub to build a harness for autonomous API integration at scale: a system that selects APIs from an inventory of thousands, researches them, builds the connectors, tests them, documents them, composes them into cross-app use cases, reviews the code, and ships.
There was one caveat attached, and it turned out to be the entire design brief: the harness has to operate inside a budget. The budget dictates how many apps get launched. Not the other way around.
The Challenge
As part of their launch strategy, W6W wanted to support an ever-growing number of apps — starting in the hundreds and steadily growing past 2,000 base apps.
The first burst, roughly 35 apps, was hand-sourced for selected partners: someone looked at what those partners needed to run their operations, picked the relevant endpoints, and built the connectors with Claude Code. The coding was already agentic, and that part worked — it produces good connectors. What didn’t scale was everything around it. A person still chose each API, scoped it, steered the build, and checked the result, so human attention stayed the unit of production and every app cost about what the one before it did. Nothing compounds.
So the ask was a harness that could run that whole arc without a person driving each step — selection through launch, including the documentation and the cross-app use cases — all inside a spending envelope.
That constraint is the interesting part. Cost here is the input that determines output, not something you watch on a dashboard and reconcile later. Apps per day is not a target anyone sets; it falls out of how much money is left and what cost-per-app currently is. Getting that relationship right shaped every other decision we made.
Autonomous API Integration at Scale, Gated by Budget
The first thing we changed was the unit of work. The unit is an integration job, not a conversation. Each job is a DAG of stages with a cost estimate attached before it starts. A scheduler admits jobs from the queue only while projected spend still fits the remaining envelope.
Stated plainly: budget is an admission-control problem, not a monitoring problem. A monitoring approach tells you that you overspent. An admission-control approach declines to start the job that would have overspent. Everything else in the architecture follows from that one reframe.
We built the harness to be self-improving, so each cycle makes the app builder better and more efficient without anyone hand-tuning it. And we ran a second, separate agentic system alongside it for early drift detection and QoW/QoP monitoring — quality of work and quality of process — because a system that optimizes itself is exactly the kind of system that can quietly optimize itself in the wrong direction.
The weekly envelope
W6W agreed to our recommendation to target a weekly budget rather than a monthly one: a tighter control loop, a bad week that can’t quietly eat a quarter, and unspent budget rolling into the next batch on a cadence short enough to actually steer.
A percentage n of the base weekly budget is held back as reserve; the rest is the dev allocation. That percentage is always computed off the base weekly budget, never off a rollover-inflated one — otherwise the reserve compounds. Unused reserve rolls forward into the next week’s dev allocation, so week k gets base_dev + rollover(k-1).
The dollar figures below are not W6W’s budget. They are a round base — $1,000, with n = 10% — chosen because the arithmetic is legible:
| Week | Dev alloc | Reserve | Actual spend | Reserve drawn | Rolls over |
|---|---|---|---|---|---|
| 1 | $900 | $100 | $910 | $10 | $90 |
| 2 | $990 | $100 | $1,010 | $20 | $80 |
| 3 | $980 | $100 | — | — | — |
The total ceiling in any week is dev_alloc + reserve — $1,090 in week 2 above.
Two decisions we left open
Two parts of the budget model are recorded in our design notes as open decisions with a recommendation attached, not as settled behavior. We think it’s worth showing them as they stand.
Underspend on the dev line. The rule as specified rolls forward only unused reserve. Using the same example: if a week spends $600 of a $980 dev allocation, the $380 evaporates. There are two viable policies. Reserve-only rollover, as specified, is simple and predictable, but it penalizes efficiency gains at exactly the moment they start working. Capped total rollover rolls unused dev money too, capped at something like 25% of base weekly so a slow week can’t create a spending spike later. Our recommendation is capped total rollover — the whole point of the improvement loop is that cost-per-app falls, and the budget model should let the client convert that into more apps rather than into a forfeited surplus.
Reserve exhaustion. When the reserve hits zero mid-week, our recommended behavior is: hard stop on admitting new jobs, let in-flight jobs finish since that money is already sunk, no rollover, and page a human. Explicitly no borrowing against next week — that is how a one-week overrun turns into a structural deficit.
Guardrails
Three guardrails sit underneath all of it. A per-job cost ceiling stops any single API consuming more than roughly 5% of the weekly envelope; breaching it kills the job and files it as too_expensive for human review. A daily pace target of remaining_budget / days_remaining, with tolerance, keeps spend even. And failed spend is tracked separately — money burned on jobs that never shipped is the honest measure of whether the improvement loop is doing anything.
How We Built the Integration Harness
The harness ProductiveHub delivered is a nine-stage pipeline sitting behind that admission gate.
The pipeline
Nine stages, each independently retryable, independently priced, and independently model-routed.
| # | Stage | Output |
|---|---|---|
| 1 | Selection | Ranked candidate APIs from inventory |
| 2 | Research | Docs, OpenAPI/Postman specs, SDKs, changelogs → normalized internal spec |
| 3 | Connector gen | Auth, pagination, rate limits, error taxonomy, schema mapping |
| 4 | Integration app | Customer-facing surface |
| 5 | Test | Contract tests vs. spec, sandbox calls, replay fixtures |
| 6 | Docs | Generated from the normalized spec, not from the code |
| 7 | Use-case composition | Cross-app workflows against the graph of shipped connectors |
| 8 | Code review | Separate model, adversarial framing |
| 9 | Launch gate | Ship / hold / kill |
Two deserve a note. Docs are generated from the normalized spec rather than the generated code, so they describe the API’s actual contract instead of whatever the connector happened to implement. And code review runs on a separate model with an adversarial framing, because a model reviewing its own output is a formality.
Model routing: open weight for volume, commercial for judgment
We route by stage character, not by preference.
| Tier | Model | Used for |
|---|---|---|
| Bulk | DeepSeek (local) | Doc scraping and chunking, spec normalization, schema mapping, test fixture generation, doc drafting, classification and triage, retry-reason labelling |
| Cheap judgment | Haiku | Routing decisions, structured extraction, validation passes, PR summaries, selection pre-filter, escalation triage |
| Real judgment | Sonnet | Connector architecture for non-trivial APIs, integration app code, adversarial code review, use-case composition, launch gate |
| Break glass | Opus | Escalation only — two failed Sonnet attempts on the same stage, or a supervisor-flagged anomaly. Hard-capped as its own budget line. |
The escalation ladder runs DeepSeek → Haiku → Sonnet → Opus, but stages have a floor rather than a universal start point. Code review never begins below Sonnet. Doc drafting never escalates above Haiku. Escalations are logged as first-class events and charged to the job that caused them — a job that escalates twice is telling you something about the API, not about the model.
The Opus governor disables the top tier automatically once reserve utilization crosses a threshold. That ordering is deliberate: a budget squeeze should degrade to “ship fewer apps,” never to “ship worse apps.”
The ladder also runs downward, which is where most of the cost lever lives. We didn’t fix a target token mix at design time — it gets discovered.
The usage audit file
Every harness run emits an append-only JSONL usage file, one record per model call. It is the source of truth — the accountant reads nothing else, and the improvement loop segments it.
A few fields carry more weight than they look. segment_id is what makes the improvement loop possible at all — the log has to be segmented at write time, because reconstructing segments afterward doesn’t work. agent_version pins every record to the instruction set that produced it; without it, no before/after comparison is defensible. cached_input_tokens is broken out separately, because prompt caching materially changes the arithmetic on a pipeline that re-reads the same pattern library every job. failure_class uses a controlled vocabulary, not free text. And records are written during the run, so a crashed run is still accounted for.
Pricing self-hosted inference honestly. Running DeepSeek locally means there is no per-token price, so cost has to be derived: $/1k tokens = (GPU $/hr ÷ 3600) × (gpu_seconds ÷ tokens) × 1000. We amortize idle GPU time across the week’s actual token volume and re-derive the effective rate weekly. Skip this and the open-weight tier looks free, which makes every downgrade recommendation the loop produces unjustified. A half-idle cluster can genuinely cost more per token than Haiku.
The accountant
A cron process — nightly at minimum, hourly during active batches. It reads the usage JSONL, prices it against a config-driven rate table (prices change; never hardcode them), writes to the ledger, and then acts.
It reports spend sliced by week, day, job, stage, model, agent, and agent version; cost per shipped app and cost per failed app; token mix and spend mix across open-weight and commercial, plus drift in both; cache hit rate and dollars saved by caching; estimate-versus-actual variance per stage and archetype; and reserve draw with projected rollover.
Then it decides. It sets admission control for the rest of the week from remaining budget, days left, and the calibrated cost-per-app estimate. It lowers the escalation ceiling as reserve utilization rises, Opus off first. It recalibrates the cost model from actual variance — the least glamorous and most valuable thing it does. It reweights selection, so realized cost-per-app feeds back into stage 1 ranking and expensive archetypes down-rank on their own. And it holds a kill switch for reserve exhaustion, per-job ceiling breaches, or a cost-variance anomaly — actual spend running past 3× estimate on three consecutive jobs usually means something upstream broke, not that the APIs got harder.
The division of authority matters: the accountant is advisory on quality and authoritative on money. It can stop the pipeline. It cannot approve a launch.
The Self-Improvement Loop
This is not the agent rewriting its own prompt. It’s a structured, evidence-gated loop over segmented execution logs.
Each dev agent’s log is cut into semantically meaningful segments — auth negotiation, pagination discovery, schema mapping, error-path handling, test scaffolding. Segments are the unit of analysis, because a whole-run log is far too coarse to learn from. Each segment is graded on measured efficiency rather than asserted efficiency: tokens burned, retries, tool calls, wall time, and whether its output survived downstream review unchanged. Where one approach consistently beats another on the same segment type, we promote it into the agent’s standing instructions; where an approach consistently wastes tokens, we encode it as an anti-pattern.
Segment grades alone would overfit to the harness’s opinion of itself, so they’re fused with external signal: user feedback on shipped connectors, post-deploy metrics such as real-traffic error rates, latency, connector churn and support tickets, and return on tokens spent — apps shipped, and apps retained, per dollar.
Changes are emitted as small versioned instruction deltas rather than periodic rewrites, because many small validated changes beat one large unvalidated one and each is individually reversible. Every delta is A/B’d against the current agent version on a held-out golden set before promotion. A delta that improves cost but degrades post-deploy quality is rejected, not silently traded off.
The downgrade recommender
The main lever on cost is a distinct output of the same loop. For each call site it asks one question: given the observed distribution of outcomes here, does the lower tier produce equivalent results?
Opus → Sonnet usually becomes available once the failure that triggered the escalation is encoded as a pre-flight check. Sonnet → Haiku once the pattern library holds enough worked examples that the task becomes retrieval and adaptation rather than design. Haiku → DeepSeek for anything that has gone mechanical — structured extraction, normalization, fixture generation.
Demotions are proposed on evidence, shadow-tested with both tiers running and outputs compared, then promoted only if quality holds across a statistically meaningful sample. Any demoted call site is re-evaluated on regression: if quality drops after promotion, it auto-reverts and locks at the higher tier for a cooling period. That’s the mechanism migrating the token mix toward local open weight without anyone hand-editing a routing table.
What accumulates
The connector pattern library is the big one — every shipped connector becomes a retrieval example tagged by auth type, pagination style, and error convention. Late in a program you have hundreds of worked examples; in cycle one you have zero. That accumulation is what makes DeepSeek viable for work that needed Sonnet at the start. Alongside it: a failure taxonomy feeding pre-flight checks, so known failure classes get caught before money is spent; versioned, diffable agent instruction sets, each traceable to the evidence that produced it; and the accountant’s cost model calibration.
The claim the whole loop stands on is narrow and falsifiable: cost per shipped app falls week over week at constant or rising quality. If it doesn’t, the loop is decorative.
Keeping It Honest
The supervisor system runs out-of-band, on a separate model and — importantly — a separate budget line, so it can’t be starved by the thing it’s watching.
QoW (quality of work) tracks test pass rates, review findings per KLOC, doc completeness, post-launch error rates from real traffic, and connector churn. QoP (quality of process) tracks escalation rate, retries per job, cost variance against estimate, stage latency, and model mix drift.
Drift detection periodically replays a fixed golden set of APIs and compares against baseline over a sliding window. Drift shows up as cost-per-app creeping up or quality creeping down while the pipeline still reports green — a pattern library poisoned by one bad shipped connector, or an instruction delta that looked good on segment metrics and is quietly worse in production. Those are the failure modes a self-improving system is uniquely exposed to, and they’re invisible to the system’s own metrics by construction.
The supervisor can pause the pipeline, roll back the last instruction delta, and force-escalate a demoted call site. It has a kill switch, not just a dashboard.
Results: Cost per Shipped App and the 6.2× Claim
The headline result is 10 apps/week → 62 apps/week average. A 6.2× throughput gain.
We don’t think a number like that should stand on its own, so we designed the proof set around defending it. A 6.2× claim invites three obvious objections — the budget grew, the APIs got easier, and quality got worse — and each has to be answered with evidence rather than assurance. What follows is what each chart is built to close, not a claim that the argument is already settled.
Throughput against cost per shipped app, on a dual axis. This is the single slide. Apps shipped per week rising while cost per shipped app falls, on the same time axis, annotated with instruction-delta promotions and tier-demotion events so step changes are attributable to specific interventions rather than to time passing.
A waterfall decomposing where the gain came from. Throughput attributable to budget increase, to cost-per-app reduction via tier demotion, to fewer retries and escalations, and to lower failed spend. This is the chart a skeptical CFO will actually read, and the important thing about it is that it can come back with an uncomfortable answer. If most of the gain came from budget rather than efficiency, the honest move is to say so — the number is still real, but it’s a different claim. Paired with apps shipped per $1,000 spent, which normalizes throughput against budget entirely, the efficiency question either survives or it doesn’t.
Golden-set replay: cost and quality on a fixed set of APIs over time. The strongest chart in the set, because the input is held constant by construction. Selection bias cannot explain a curve on a fixed API set. This is what would answer “the APIs got easier.”
Quality metrics on the same time axis as cost-per-app. Test pass rate, review findings per KLOC, post-deploy error rate per app, connector churn — plotted against cost so the two are read together. Cost falling while quality holds is the claim; either chart alone proves nothing. Human intervention rate per shipped app belongs here too: if it didn’t fall alongside cost, throughput was bought with reviewer hours that never show up in the token budget.
One caveat on all of it. The golden-set replay, the archetype tags, and the post-deploy metrics only exist if they were instrumented from week one. If any weren’t captured at the start, the right answer is to say so rather than reconstruct a baseline after the fact — a retrofitted baseline is the fastest way to lose a technical audience, and it costs more credibility than the missing chart would have bought.
What We’d Carry Forward
Two things we’d move earlier on the next build of this kind.
Instrument the golden set, the archetype tags, and the post-deploy metrics from week one. They cost very little to capture up front and cannot be recovered later. Everything that makes an efficiency claim defensible depends on a baseline that existed before anyone knew the result.
Price self-hosted inference honestly from the start. If the local tier looks free in the ledger, the downgrade recommender is optimizing against a fiction, and every demotion it proposes is unjustified — including the ones that happen to be correct.
Conclusion
W6W needed a catalog that grows without growing linearly in human attention, inside a budget. What ProductiveHub built treats those as the same problem. Making cost an admission-control decision rather than a monthly report lets the pipeline run close to its envelope without breaching it; segmenting execution logs at write time lets the system learn where its money actually goes; and putting an independent supervisor on its own budget line keeps a self-improving system from improving itself somewhere nobody wanted it to go.
The throughput number is the part that fits on a slide. The part worth keeping is the machinery that makes the number arguable — the fixed golden set, the failed-spend line, the quality metrics plotted on the same axis as cost. Autonomous API integration at scale isn’t hard because generating a connector is hard. It’s hard because staying honest about what the connectors cost, and what they’re worth once they ship, is the only thing that lets you keep doing it.
Technology Stack
Transform Your Technology Organization
Ready to achieve similar results? Read about our fractional CTO services, or start a conversation about your engagement.
Related Success Stories
Fixing an 800% Cloud Cost Spike from Runaway Multi-Agent AI
28 days after deploying their multi-agent AI, a startup's cloud bill spiked 800%. Their agents had turned—trapped in infinite loops, generating 1.4 million zombie writes per hour. We contained the outbreak and delivered seven survival patterns.
AI Engineering Mesh: Scaling Agent Teams Like Infrastructure
A newly funded AI platform needed to ship an MVP fast — without the chaos that usually comes with 'fast.' We built a dynamically scaling mesh of Claude-powered agent clusters with a stakeholder map that knows exactly who to call, when, and why.