I Ran a Bank's Risk Model On An AI Agent
Just because it passes functional tests doesn't mean it's financially viable
A couple of months ago, I found myself having to conduct an unenviable and boring task - counting LLM tokens.
The reason?
A review of an AI agent on the cusp of deployment into production at one of Europe’s largest logistics operators was giving a few people the jitters. A warehouse-incident support agent - built as agent steps inside their integration platform, running on Claude Sonnet at API rates - was the culprit. It’d passed every functional test anyone thought to write. Its costs have been unremarkable through months of development. The people in the room like it, which is rarer than it sounds, and the agenda had one item left before everyone goes back to their day jobs.
Then somebody asks the question that ends up saving the project. What happens in a critical outage — two warehouses go down at once?
Nobody had been careless. Testing at demo volume is what every team does, because demo volume is what exists before rollout; the finance people had signed a budget built on the costs the testing produced, which is what finance is supposed to do. The engineers had built an agent that worked. Every step was professional. And every step quietly assumed the thing that the question had just un-assumed: that the agent’s costs on a normal day tell you something about its costs on the worst one.
They don’t, and the reason lives in how agent systems remember.
A conversational agent re-sends its accumulated memory with every turn. If this is how the system’s been designed, there’s no cache holding the story so far; turn ten carries turns one through nine back into the prompt as input tokens, turn twenty carries nineteen. Cost doesn’t grow with a conversation — it compounds with it.
When I pulled the token logs, the worst of the agents was consuming 80,000 input tokens for every 1,200 it produced. More than 90% of the bill was input. The agent wasn’t paying to think. It was paying to remember out loud.
Now put that mechanism inside an agent and follow the logic to its destination. An outage doesn’t just create incidents — it creates long, anxious, multi-turn conversations, many of them at once, each one compounding as it grows. The agent works hardest, and burns most, at the exact moment the business is already having its worst day. The cost curve and the crisis curve are the same curve. A budget built on the demo month meets the difference mid-crisis, as an invoice.
So I did something that had no precedent in that project, though it has decades of precedent one industry over. I priced the agent the way a bank prices a trading desk.
Banks stopped asking “what will this position cost?” a long time ago, because the question has no single answer. They ask for the distribution: the expected case, the bad case, the catastrophic case, each with a probability attached. Value-at-Risk, the instrument is called, and it maps onto an AI agent with almost embarrassing ease. I built a scenario table — a normal month, an elevated one, a critical outage, an extreme one — weighted at 85%, 10%, 4% and 1%, and let the model do what the demo never could: show the tail.
A critical month came out at ten times the base month. An extreme month, twenty. And the figure a finance team actually needed — the P95, the number you budget so that only the worst 5% of years surprise you — landed at roughly five times the expected annual cost. Same agents, same warehouses, same model underneath. The only variable was how rough the year turned out to be.
Leadership looked at the tail and made a decision that would’ve been impossible a week earlier, because a week earlier nobody knew the tail existed. They scoped the rollout — the version whose bad night the budget could absorb — instead of the uncontrolled one, which at those multiples had no route to a return at all. A banking instrument had saved an AI project, and what struck me most was that nobody in the meeting found this odd. Finance people have known for a century how to make decisions about distributions. They’d just never been offered an AI system priced as one.
The law underneath is short: AI agents don’t have a price. They have a price distribution. The demo month is your P50 — the middle of the curve, the month most likely to happen, the number every pilot report contains. Production lives at the P95. Neither number is a lie. They’re different questions, and almost every agent business case I’ve reviewed answers only the comfortable one.
If you’re involved with an AI project today there’s a couple things I’d advise you do with some urgency. Assuming of course that you’ve not already done this.
First, pull the token logs from whatever agent you have running — pilot, proof of concept, anything with history — and compute the ratio of input to output. If input runs above 80 or 90% of the total, and on conversational agents it usually does, you’ve found a compounding architecture, and no amount of user adoption will make it cheaper. That’s a design conversation, not a procurement one.
There’s a footnote here about the most common fix I see attempted. When the token bill frightens, the first instinct is to swap the model — if Claude on AWS Bedrock looks expensive, thinking shifts to a cheaper native Nova model instead. Against the mechanism above, that’s a false economy. The bill wasn’t big because of the rate card; it was big because the architecture re-sends its memory with every turn. A cheaper model inherits the same compounding at a discount — and often needs more turns to reach the same answer, feeding the exact behaviour you were trying to escape. Swapping models slides the whole distribution down a little. It doesn’t change its shape. The critical month still costs ten times the normal one — which is one more reason to price agents as distributions, not rate cards.
Second, take your ROI slide and re-cut its single number into three: a conservative case you’d stake 70% confidence on, a realistic one at 50%, an optimistic one at 30%. Then present the range. I’ve watched a CFO dismantle a beautifully precise projection with one sentence — “those numbers seem precise, for a black box” — and I’ve watched the same CFO lean into a set of honest ranges. Precision reads as naivety. Ranges read as someone who’s run one of these in production.
No IT project or software system (especially one that includes AI) goes through its life without a bad night at some point — that’s what makes it a distribution and not a guess — and it arrives with the invoice already written. The only question is whether anyone priced it before it did.
One question did, in a review room, a week before rollout. Cheapest question that operator ever asked.
Until next time,
Chris
P.S. — The money gate is only the first of the three gates, and most projects that survive it die at another door. How AI Projects Die walks all five — two screens, three gates, ten real engagements, the numbers left in, one move at each. Free, twenty pages: take a copy.