---
title: "Deterministic operations, measured"
description: "On our complex, multi-step operations benchmark, a compiled Ballet playbook matches its own answer 54 times out of 54 across 8 runs each, replies in a median 0.08s, and costs $0 in model tokens. GPT-5.6 Sol 35-54 and Claude Sonnet 5 matched 42-of-54 consistency on the identical benchmark but ran roughly 40x slower and cost $67–$112/month at just 10,000 complex tasks, with the meter climbing as volume grows."
canonical_url: "https://docs.ballet.dev/articles/deterministic-operations-measured-A0mAp58RxY"
md_url: "https://docs.ballet.dev/articles/deterministic-operations-measured-A0mAp58RxY.md"
---
# Deterministic operations, measured

**On our complex, multi-step operations benchmark, a compiled Ballet playbook matches its own answer 54 times out of 54 across 8 runs each, replies in a median 0.08s, and costs $0 in model tokens. GPT-5.6 Sol 35-54 and Claude Sonnet 5 matched 42-of-54 consistency on the identical benchmark but ran roughly 40x slower and cost $67–$112/month at just 10,000 complex tasks, with the meter climbing as volume grows.**

Most "AI automation" demos answer a question once. Operations teams need the same answer on the same input, every time, including the run nobody is watching. That gap is measurable: Sierra's [τ-bench](https://arxiv.org/abs/2406.12045) showed tool-calling agents scoring under 50% success on a single attempt can fall below 25% once you require the same task to succeed on all eight trials. One good run is a demo. Eight identical runs is a workflow.

Ballet splits those two jobs on purpose. A model helps you **author** a playbook the decision logic behind a repeatable task: the refund window, the VAT rate, which queue this goes to. The playbook then **executes as compiled code**. Ballet bills those two jobs as two separate lines for the same reason: **agent usage** is the cost of writing and revising a playbook; **compute** is the cost of running it.

This is the consolidated summary of what we measured on the compiled path determinism, speed, and cost comparing the compiled playbook directly against GPT-5.6 Sol and Claude Sonnet 5 on our own complex, multi-step benchmark.

## What we measured

We ran 54 everyday operations cases across 18 playbooks refunds, invoice aging, VAT, webhooks, discounts, seat proration, ticket priority, and the rest of a back-office Tuesday. These are complex, multi-step cases: each one chains several dependent tool calls together reading fields off a ticket, checking policy tables, applying VAT and refund logic, and picking a queue rather than a single lookup. Each case used a fixed input and a known correct output, with no live network. Each case ran 8 times through Ballet's production compiler and sandbox (432 runs total). A trial counts as a match only when its output is identical to the expected result the score that matters is whether **every** trial matched, not whether one of them did.

For the cost, speed, and determinism comparison, we ran the same 54 multi-step cases a second way: a **pinned model loop** [GPT-5.6 Sol](https://developers.openai.com/api/docs/models/gpt-5.6-sol) and [Claude Sonnet 5](https://vercel.com/ai-gateway/models/claude-sonnet-5) calling the same in-memory tools, working through the same multi-step chain, rather than Ballet's authoring chat, since that's a fairer "just put an agent on the queue" baseline (authoring prompts change). Each of the 54 cases ran 8 times on each model too, so we could check whether the model's answer held steady across the full multi-step chain, not just whether it was right once.

## Determinism: 54 of 54, on the complex benchmark, for all three

Every case produced the same answer all 8 times, for every approach we tested on the same multi-step chain, not a simplified version of it.

|                  | Compiled playbook | GPT-5.6 Sol                         | Claude Sonnet 5                     |
| ---------------------- | ----------------- | ----------------------------------- | ----------------------------------- |
| Cases (18 playbooks)   | 54                | 54                                  | 54                                  |
| Trials per case        | 8 (432 runs)      | 8 (432 runs)                        | 8 (432 runs)                        |
| Same answer every time | 54 of 54          | 35 of 54                            | 42 of 54                            |
| Retry / resume checks  | all held          | not applicable outside the playbook | not applicable outside the playbook |

That's the result on our actual benchmark: a multi-step, tool-calling ticket resolution, not a one-shot classification. Compiled code holds it by construction it isn't sampling from a distribution, so there's nothing for repetition to shake loose. 

If the answer changes when nothing else has, you don't have a workflow. You have a sample.

## Speed: 0.08s median, roughly 40x faster than asking again

|       | Compiled playbook | GPT-5.6 Sol | Claude Sonnet 5 |
| ----------- | ----------------- | ----------- | --------------- |
| Median time | 0.08s             | 4.15s       | 3.70s           |
| p95 time    | 0.10s             | –           | –               |

0.09–0.10s versus 3.70–4.15s isn't just a line item — it's real latency, and it matters most exactly when a decision has to resolve inside a live conversation or a real-time workflow, not a backend batch job nobody is waiting on.

## Cost: $0 per compiled run, a growing bill for asking again

|                         | Compiled playbook | GPT-5.6 Sol | Claude Sonnet 5 |
| ----------------------------- | ----------------- | ----------- | --------------- |
| Model cost, 8 runs (one case) | $0                | $0.0537     | $0.0900         |
| Marginal cost, one ticket     | $0                | $0.0067     | $0.0113         |

That marginal cost is a rate, not a fixed cost it scales with whatever volume you actually run:

| Tickets / month | Compiled playbook | GPT-5.6 Sol | Claude Sonnet 5 |
| --------------- | ----------------- | ----------- | --------------- |
| 500             | $0                | $3.36       | $5.63           |
| 2,500           | $0                | $16.80      | $28.13          |
| 10,000          | $0                | $67.17      | $112.51         |

List prices used: GPT-5.6 Sol at $4 / $20 per million tokens, Claude Sonnet 5 at $2 / $10. The playbook column is $0 in **model** spend compute still appears on a Ballet invoice.

![Typical time per run](https://docs.ballet.dev/api/attachments.redirect?id=8b40a903-6023-42ee-91b1-26a769555df2 " =960x540")

![Model cost at monthly volume for simple task](https://docs.ballet.dev/api/attachments.redirect?id=d63e25a2-0768-4727-a485-38d285db8413 " =960x540")

![Histogram of compiled playbook replay time across 432 runs](https://docs.ballet.dev/api/attachments.redirect?id=ecc18872-804f-4982-82d1-6addc2779b6d " =1280x720")

## A worked example

Take a single case: a customer emails asking for a refund on an order placed 34 days ago. The playbook already encodes the policy behind that: refunds within 30 days go through automatically; between 31 and 45 days, only if the item is unopened and the customer is on a paid plan; past 45 days, route to a human. It also knows the VAT treatment differs for EU customers versus everyone else, and which queue an escalation reply belongs in several dependent checks chained into one resolution, not a single lookup.

A compiled playbook reads the order date, plan tier, and region off the ticket and works through that chain directly same steps, every time, no model call, no chance of the day-30 cutoff getting fuzzy on the 200th ticket of the month. A model asked fresh each time has to re-derive that whole decision tree from the ticket text and work through the same multi-step chain itself which GPT-5.6 Sol and Claude Sonnet 5 both did correctly and consistently across all 8 runs in this test, but at a cost, and roughly 40x slower.

## Why this keeps mattering as you scale

Three things about these tables don't show up in a single snapshot.

The cost line only goes up. A compiled playbook's model spend is flat at $0 no matter how many tickets you run. A raw model loop's isn't  it's a straight line through the origin, and it keeps climbing for as long as the queue exists. Run the same rate at 50,000 or 200,000 tickets and the shape doesn't change, only the size of the bill.

Every decision is roughly 40x slower when you ask again that's real latency, not a line item, especially when a decision has to resolve inside a live conversation or a real-time workflow.

A raw model loop stays exposed to the model underneath it. GPT-5.6 Sol and Claude Sonnet 5 are today's pinned versions; when a provider reprices, deprecates, or changes model behavior on an update, a live model-in-the-loop workflow has to be re-tested and re-priced including re-checking that it still holds this same consistency, since that isn't guaranteed to carry over to a future version. A compiled playbook doesn't move the policy was already decided, and compiling it once is what stops you from re-buying that decision, and re-verifying it, every time a model or its underlying infrastructure changes.

## Write once, run the same way

[Kong's artifact-driven architecture](https://konghq.com/blog/engineering/deterministic-ai-architecture-enterprise-reliability) is the industry picture of why you compile a path and reuse it instead of regenerating a plan: use a model to capture a validated path, then reuse the artifact. In Ballet that artifact is a versioned playbook. Resume, cache keys, and uploaded files stay pinned to that version and to a stable run identity this is why authoring can stay creative while execution stays boring. Boring is the feature.

That's how you want a ticket queue to feel: expensive to design, cheap and identical to run.

## What this does not claim

These numbers describe **compiled playbook steps** with fixed inputs and no live network not a chat agent, not a step that calls a model at runtime, and not a customer SLA. Steps that call a model at runtime, or that leave the building to hit a live API, can still vary; they are not in these tables. Built-in templates that summarize a Zoom call or triage Zendesk are products, not replay proofs.

The GPT-5.6 Sol and Claude Sonnet 5 figures above are from this specific benchmark and this specific run, at temperature 0. It's a real result on a real multi-step, tool-calling test not a claim that either model is deterministic as a mathematical guarantee the way compiled code is. The compiled playbook's determinism holds by construction at any scale; the two models' determinism is empirical, measured here, and re-testing on a future model version, a different temperature setting, or a busier inference endpoint is the way to confirm it still holds rather than assume it.

[τ-bench](https://arxiv.org/abs/2406.12045) is the industry reference for agents that must work out policy themselves the harder case not a head-to-head of Ballet versus Sierra, and not a benchmark we ran GPT-5.6 Sol or Claude Sonnet 5 through ourselves. This page is not a price comparison with Zapier, n8n, or UiPath, and volume rows are extrapolated from the fixture set at today's list prices, not a quote if a provider cuts prices the slope changes, though it does not go negative.

If your queue looks like these 54 cases a multi-step decision your team has already made once and keeps re-making this is roughly what you are paying to keep asking, for an answer that, on this benchmark, was no more correct than the one you already compiled. [Book a demo](#) to run it against your own playbooks.

## Sources

* [Yao et al., τ-bench (arXiv:2406.12045)](https://arxiv.org/abs/2406.12045)
* [Sierra Research, τ-bench repository](https://github.com/sierra-research/tau-bench)
* [Kong, Deterministic AI architecture (artifact-driven execution)](https://konghq.com/blog/engineering/deterministic-ai-architecture-enterprise-reliability)
* [OpenAI, GPT-5.6 Sol model card](https://developers.openai.com/api/docs/models/gpt-5.6-sol)
* [OpenAI, GPT-5.6 Sol upgrade guide (Chat Completions + tools)](https://developers.openai.com/api/docs/guides/upgrading-to-gpt-5p6-sol)
* [Vercel AI Gateway, Claude Sonnet 5](https://vercel.com/ai-gateway/models/claude-sonnet-5)
* [Vercel, AI Gateway](https://vercel.com/docs/ai-gateway)
* [Temporal, Workflow deterministic constraints](https://docs.temporal.io/workflows#deterministic-constraints)

Measured on 2026-09-07. These are lab results on a fixed fixture set based on real world automations, not a customer SLA.