Loading…
Deterministic agents: Ballet's approach
TL;DR: Ballet runs automations as deterministic playbooks and calls AI only for the steps that genuinely need judgment, so output is reproducible, debuggable, and trustworthy. Run history and evals make agent behavior measurable rather than a black box.
Why determinism matters
Many AI agent products treat the LLM as the entire workflow — every decision, every action, every output flows through a model that can vary run to run. That works for open-ended chat. It breaks down for ops automation, where the same ticket, lead, or webhook payload should produce a consistent result your team can rely on.
Determinism is not about avoiding AI. It is about controlling where intelligence is applied so the parts that must be predictable stay predictable.
Deterministic by default
In Ballet, the core unit is a playbook — an ordered set of steps that run the same way every time. Most steps are deterministic:
- Code steps — sandboxed JavaScript, TypeScript, Python, or bash with explicit inputs and outputs.
- HTTP steps — API calls with defined request/response shapes.
- Control flow — branching and sequencing you can read in the editor.
When a playbook runs, you see step-by-step progress: which steps succeeded, what each step produced, and where a run failed. The same playbook with the same inputs should behave the same way — that is the baseline.
Intelligence only where it's required
AI agent steps exist for work that genuinely needs judgment:
- Classifying or routing ambiguous inputs (support tickets, feedback, leads).
- Extracting structure from unstructured text (emails, notes, documents).
- Synthesizing research or drafting content when the path is not fixed in code.
Everything else — fetching data, transforming fields, calling APIs, enforcing business rules — belongs in code or HTTP steps, not in open-ended LLM reasoning.
This pattern keeps the AI surface area small. Fewer model calls means fewer places for drift, hallucination, or non-reproducible behavior. Your engineering team can review deterministic steps as code; agent steps are scoped to the decisions that actually need a model.
Ask Ballet helps you plan and build this mix: it proposes playbooks where intelligence sits only on the steps that need it, and you confirm before anything is applied.
Evals: measuring the non-deterministic parts
Deterministic steps are easy to trust — run them twice, compare outputs. Agent steps are not. That is where evals matter as a discipline, even when your team is still maturing the practice:
- Define expected outcomes for agent steps (labels, extracted fields, routing decisions).
- Regression-check when you change prompts, models, or skills — did behavior improve or drift?
- Sample production runs from history and compare against baselines before promoting changes.
Ballet's run history gives you the raw material for this: every agent step leaves output you can inspect, share, and compare across runs. Evals are how you turn that visibility into ongoing confidence, not a one-time demo.
We treat evals as part of responsible agent ops — not a checkbox, but a habit: measure the steps that are allowed to be non-deterministic, and keep the rest in code.
History makes it auditable
Every playbook execution creates a run with a full timeline: step status, logs, and outputs. When something goes wrong in production, you do not guess what the agent "meant" — you open the run, see exactly which step failed or produced unexpected data, and fix the playbook or the prompt.
History also supports team accountability: share run output with stakeholders, replay failures, and document what automation did on behalf of your ops stack. Determinism plus history is how agent workflows earn the same trust as traditional integrations.