Business problem
A marketplace generates a constant stream of questions that are trivial for the person with database access and slow for everyone else. A buyer wants to know why a call did not reach them. A publisher wants yesterday’s delivered volume by vertical. Someone wants a campaign paused before the next hour of spend. In the ordinary operating model all of these arrive as email or chat messages and consume a human, which means the answers are slow, inconsistent, and expensive at exactly the moments they matter.
An assistant that can only answer questions solves the easy half. The valuable half is the half that acts: pause the campaign, adjust the bid, update the routing target, mark the disposition. And that is where the design problem starts, because the moment an assistant can change production state, the question stops being “is the model helpful” and becomes “on whose authority did that happen, and can we prove what actually occurred”.
Why this was difficult
The difficulty is not making a model call a function. It is that the naive version of tool calling quietly assumes several things that are not true.
Model output is not authorization. A model producing a well-formed tool call is evidence that the tokens were likely, not that the user asked for it, is entitled to it, or understood it. Treating generation as intent is the root defect behind most of the incidents this design exists to prevent.
Intent is ambiguous, and confidently so. “Turn off the campaign that is losing money” requires a judgement the model will happily make. The system has to force that judgement into the open, as a specific proposed change against specific records, before anything happens.
State moves between the proposal and the confirmation. A user takes twenty seconds to read a preview. In that window a cap can be hit, a target can be disabled, a call can complete. An action confirmed against stale facts is a different action from the one the user approved.
Duplicate execution is not idempotent in the real world. A retried write, a double-clicked confirmation and a reconnected socket can all present as the same request. In a system with budgets, payouts and routing capacity, executing twice is a financial event.
Models narrate success. Left to itself, an assistant will tell the user their campaign was paused because that is the natural continuation of the conversation, whether or not the write succeeded. Any design that lets the model speak before the system has confirmed the outcome is a design that will eventually lie to a customer.
Cross-tenant leakage is unacceptable and easy. Competitors are frequently both buyers in the same vertical. A tool that forgets its scope on one code path is a commercial incident, not a bug.
My responsibility
I defined what these agents are for, what they may do, and what they must never do on their own. That means the capability model — the decision that certain actions are reserved to humans regardless of how convenient automation would be — and the interaction model, including the requirement that a user confirming a mutation is shown the server’s computed effect rather than the model’s description of it.
I specified the authorization requirements (persona and organization scoping, role-based permission on every tool, deny by default), the execution correctness requirements (server-held arguments, tamper verification, ownership and TTL checks, stale-state detection, single-consumer execution), the outcome-reporting rule, the spend and budget ceilings, and the audit requirements. I wrote the acceptance criteria each mutating tool had to satisfy before it was allowed into the catalogue, and I tested and reviewed the implementations against them.
Constraints
- Partner-facing means untrusted input. External users talk to Partner Chat. Prompt content is user input, and no part of the safety model may depend on the model behaving well.
- It has to be genuinely useful. A safety model that makes the assistant slower than email fails commercially — people simply stop using it and the design proves nothing.
- Everything is auditable after the fact. A financial or routing dispute weeks later must be answerable from records, not recollection.
- Two surfaces, one doctrine. The partner assistant (~35 tools) and the internal operator MCP (~120 typed tools) have very different blast radii but must not have two different safety stories, because the weaker one becomes the real one.
- Coding agents implement it. The safety model had to be expressible as concrete, testable acceptance criteria, not as design intent.
Architecture
The flow above is the entire argument of this case study, so it is worth reading as a sequence of refusals rather than a sequence of steps.
A user expresses intent. The model selects a tool — and that selection is a proposal, nothing more. Permission and persona/organization scope are evaluated server-side against the authenticated session, not against anything the model asserted. The server then computes the preview: it resolves the actual records, the actual current values and the actual effect, and it stores the pending action server-side. The arguments that will execute live on the server from this moment; the model never holds them again.
The user sees that computed preview and confirms. Confirmation triggers verification, not execution: TTL, ownership, argument hash and record version are all checked, because the world may have moved since the preview was computed. Only then does execution proceed, through a compare-and-swap consumption of the pending action, so the action can be consumed exactly once no matter how many times the confirmation arrives.
The real outcome is persisted, and only that outcome is returned to the model. The assistant’s follow-up sentence is generated from what happened, not from what was requested. If execution failed, the user is told it failed.
Important decisions
The server holds the action arguments. This is the decision the rest of the model rests on. If the confirmation turn carries arguments, then a model — or anything able to influence one — can alter the action between preview and execution, and the user’s approval no longer refers to the thing that executes. Storing the resolved action server-side and passing only an opaque reference makes the approved action and the executed action the same object by construction.
Confirmation shows computed effect, not restated intent. A preview that says “pause your campaign” is a summary. A preview that names the record, its current state and its resulting state is a fact the user can check. The second one occasionally reveals that the model misunderstood, which is the entire point.
Hash, TTL, ownership and version, checked at execution. Each of these closes a specific hole: tampering, indefinitely resurrectable approvals, one user consuming another’s pending action, and execution against facts that changed after approval.
Compare-and-swap consumption. The pending action is consumed atomically, so concurrent or retried confirmations resolve to exactly one execution. Idempotency here is not a nicety; the actions in scope include money-adjacent and capacity-adjacent changes.
The model speaks last, from the outcome. No optimistic narration. The tool returns the true result — including partial failure — and the assistant’s reply is derived from it.
Deny by default, per tool, per persona. Tools declare what they need; a new tool is unreachable until it is explicitly permitted for a persona. Adding capability is a deliberate act rather than a side effect of deployment.
Some actions are simply not tools. The most important safety decision was subtractive: sending external communications and moving money were never given to an agent at any confirmation level. That boundary is described in DCE-Ops, where it does the most work.
AI and agent usage
Both senses of “agentic” apply here, and they should be kept apart.
The product is agentic: two live agent surfaces operate over production marketplace data. The partner assistant carries roughly 35 registered tools and can perform real state mutations under the controls above. The internal operator MCP exposes roughly 120 typed tools across marketplace, financial and diagnostic domains, and is the substrate the operations layer runs on — the 88 documented operational engagements in DCE-Ops were performed through it.
The delivery was also agentic: coding agents implemented the Go tool servers, the pending-action machinery and the frontend against my requirements and acceptance criteria. That fact shaped the safety model rather than compromising it. A control I cannot state as a testable acceptance criterion is a control I cannot verify an agent implemented, so every item in the section below exists as something a test can assert.
Safety, reliability, and operational controls
Results
Both surfaces are in production. Partners use the assistant to answer their own operational questions and to make scoped changes without waiting on a human, and the internal operator tooling carries the recurring business operations described in the DCE-Ops case study — 88 documented engagements over a ten-week window, roughly nine a week.
The claim I am making here is architectural rather than statistical: this is a working design for letting a model act on production systems where the authorization, the arguments, the freshness of the underlying state and the reported outcome are all owned by the server, and where the actions with irreversible external consequences were never delegated in the first place. I am not claiming an incident-free record as a metric, because a clean interval is weak evidence for a safety property — the controls are the evidence, and they are individually testable.
What I learned
Tool count is a surface-area measure, not a capability measure. Roughly 120 typed operator tools is an honest description of coverage and a poor description of usefulness. In practice a smaller number of higher-level, task-shaped tools produces better agent behaviour than a large catalogue of primitives, because each additional near-duplicate tool is another opportunity for the model to pick the almost-right one. If I were specifying the operator catalogue again I would start from the twenty tasks people actually perform rather than from the API surface.
The confirmation step is a product feature, not a tax. I expected users to experience the second turn as friction. What it actually does is make the system legible: the preview frequently shows a user that their request meant something slightly different from what they had in mind, and they correct it before anything happens.
Refusal has to be designed, not hoped for. An agent that will not proceed without sufficient evidence is more useful than one that produces a confident answer, but that behaviour has to be specified — including what it says, what it escalates, and what a human does next. That thread continues in DCE-Ops, where an agent declining to assign a value to a roughly $21.1K line was the correct outcome.
What I would change. I would build the audit view before the tools rather than after. For several weeks the audit records existed and were only readable through database queries, which meant the safety model was verifiable in principle and inconvenient in practice — and a control that is inconvenient to inspect is one nobody inspects.