AI Agent ROI: I Audited My 30+ Loop Fleet to Find Out

Claire Vo stood on stage at Cursor's Compile conference and said the thing nobody running agent fleets wants to hear: "10 years ago we had scrum and story points and backlogs, and now we have agent swarms and token maxing." Her argument (full talk) is that product people spent decades inventing productive-feeling systems for not building what people want, and that the agent era is growing the same theater with better tooling. Then she asked the room whose revenue was up in proportion to their PR count.

I run a fleet of agent loops across three brands. As of this morning's dashboard snapshot, 25 enabled and 7 paused or retired. So she is talking about me, and I went and checked.

She's half right. And her question is really the ROI question everyone running agents gets asked and almost nobody can answer with numbers: what did the fleet return, against what it cost? Here is what my own logs say, including the part that agrees with her.

Where She's Right, With My Receipts

I wrote up the operating system behind our fleet in how to become an AI-native company. That article carries the failure cases in full. Three of them are exactly what Vo is describing.

The loop that ran 140 times and landed nothing. We have a reply loop that had run 140 times as of 2026-07-28 and had landed its last reply on 2026-07-21. A CLI bug meant it couldn't post multi-line text, every draft it was handed was multi-line, so it skipped, correctly, every run. Forty-five drafted replies were sitting in its queue. Every one of those 140 runs recorded ok. By the activity metric this was one of my most reliable loops. By any metric Vo would accept, it had been dead for a week.

$50.71 of gate-passing work I deleted by hand. Our design build engine shipped items that scored 8.0 to 8.6 out of 10 from its own critic, all hard checks passing. I looked at the live output and rejected seven of them by eye. Four of the seven are priced in the run log: $50.71 of metered spend, all four deleted. The loop was optimizing its gate score. The gate score had come apart from the thing it was a proxy for. Run after run, the dashboard was green.

The queue at the human gate. The binding constraint on our fleet has never been production. It's that every loop's output needs a human decision, and the human was already the bottleneck. We had to write this down as a standing rule: when the queue jams, the answer is to kill, defer, or ship what exists, never to add another producer. If your instinct when output stalls is to stand up one more loop, you are doing the thing Vo says you're doing.

So the disease she's naming is real. I have it in my own logs. A fleet can be fully green and mostly theater at the same time, and the operator won't know, because every metric the fleet reports about itself is an activity metric.

Where the Comparison Breaks

Here's what I think her framing misses, having lived on the other side of it.

Scrum's cost was human hours. The ceremony consumed the same engineers who could have been building, which is why process theater was pure loss. A loop's production cost is tokens, and tokens are cheap. Our expensive design loop runs $7.82 to $15.58 per priced run. The reply loop cost roughly nothing to run 140 times. What the fleet actually consumed was my attention at the review gate, and that's a different resource with a different fix.

Which means the answer isn't fewer agents. The loops on my fleet that touch something real do land. The SEO loops carry a rule where every run reports how many of its own shipped PRs are still unmerged, as a number called prs_unmerged, and the run gets told that while that number is nonzero, flat search results are expected and not a failure. That single reporting rule is the difference between a loop that knows it hasn't landed and a loop that thinks skipping is succeeding.

The scrum comparison also can't explain why we keep 7 loops paused or retired. Theater doesn't retire itself. A fleet with a working kill rule shrinks where it isn't landing, and the paused column is evidence the audit is running, not evidence the whole thing is a ritual.

How to Measure AI Agent ROI: Landed, Not Produced

Vo's one-liner for where the constraint moved is "the rate limiter is commercializable ideas." Mine is narrower and easier to operationalize: the rate limiter is landed outcomes per loop, and most fleets don't track it at all.

Landed means the work reached the world. Merged, not opened. Posted, not drafted. Indexed and clicked, not published. Replied to, not sent. Paid, not shipped. Everything upstream of that is production, and production is now nearly free, which is precisely why counting it tells you nothing.

That gives ROI a denominator and a numerator you can actually pull from logs. The numerator is landed outcomes. The denominator is token spend plus the review time the output consumed at the human gate. On our fleet the token side is small and knowable: one daily SEO loop cost about $19 over four days, the expensive design loop runs $7.82 to $15.58 per priced run, and the reply loop cost close to nothing for its 140 runs. The review side is the number that surprised me, because 45 queued replies and seven hand-rejected designs were charged against my attention, and attention was the scarce input all along. Most published agent ROI math counts the tokens and ignores the gate. The gate is where the money goes.

The audit I ran on my own fleet is three columns and an afternoon:

1. List every enabled loop. Mine came to 25. If you can't produce this list, that's the finding.
2. For each loop, write down the last thing it landed and the date. Not the last run. The last outcome that reached the world. The reply loop's landed column said 2026-07-21 while its run column said today. That gap is the whole diagnosis.
3. Count the queue at every human gate. Forty-five replies waiting on me was not a loop problem. It was a me problem, and no additional loop was going to fix it.

Then apply the standing rule to the bottom of the list: kill it, defer it, or ship what it already made. Prompt tuning is not one of the options. A loop that lands nothing for two weeks has a structural problem, and structural problems don't respond to wording.

Two mechanical rules keep the audit honest after the afternoon is over. Every loop reports what it landed, not what it did, the way prs_unmerged does. And a run that skipped must record differently from a run that succeeded, because the two failure stories above stayed invisible for exactly as long as ok covered both. The deeper version of that observability problem, including the outage that all of our loops handled correctly and invisibly, is in how to cold-launch a social presence with agent loops.

The Part That Keeps Me Honest

One more receipt, from the morning I drafted this. Our loop scheduler's API went down at 12:22 local time and stayed down for over two hours. The dashboard that generates my fleet snapshot rendered every loop's outcome as a question mark. The incident report is timestamped and sitting in my notes repo.

For two hours, the fleet I'm defending in this article could not answer the only question this article says matters. I couldn't tell you what had landed, because I couldn't tell you what had run. An audit is only as good as the observability underneath it, and mine has a single point of failure I hadn't priced until it failed while I was writing about auditing.

That's the honest state of running an agent fleet in 2026. Vo is right that the theater is available, cheap, and extremely comfortable. The counter isn't a rebuttal, it's a column in a spreadsheet that says what landed and when. Keep that column current and the swarm has to earn its tokens. Let it go stale and she's describing you, whether your dashboard is green or not.

If you want the full operating system this audit plugs into, the loop order, the autonomy ladder, and the failure catalog are in how to become an AI-native company. The enforcement side, including why rules that agents can route around aren't rules, is in the harness engineering guide.