Loading…
What is an evaluation agent?
TL;DR: An evaluation agent classifies a record. It returns a choice, a score, or a probability. It does not write the reply, update the CRM, or call tools. A later playbook step does that work.
Who this is for
Builders who need a playbook to sort, route, or score the same kind of record every time: a support email, a deal, or a lead.
How is this different from a chat agent?
A chat agent has instructions, a language model, tools, and skills. It writes. An evaluation agent has questions. It decides.
| Chat agent | Evaluation agent | |
|---|---|---|
| Job | Draft, summarize, explain | Classify, route, score |
| You send | A prompt | The fields to classify, such as subject and body |
| You read | The written reply | choice, probability, or score, plus confidence |
| Tools and skills | Yes | No |
What the flow looks like
A support email goes in. The evaluation agent returns labels. The playbook branches. Routing the ticket and writing the reply are later steps.
The three question types
Choice. At least two options, written as an object of option id to description. You read answers.<id>.choice and answers.<id>.confidence.
{
"queue": {
"type": "choice",
"instructions": "Which queue should handle this email?",
"criteria": {
"billing": "Payment, invoice, or refund",
"access": "Login or permissions",
"other": "Anything else"
}
}
}
Boolean. A yes or no. Instructions are enough. You read answers.<id>.probability, a number from 0 to 1.
Score. Two to ten levels, as a list of strings. You read answers.<id>.score and answers.<id>.confidence.
The question text is the instruction. The agent instructions can be empty.
How a playbook step calls it
const r = await metaphor.agent('email-classifier', {
subject: email.subject,
body: email.body,
});
return {
queue: r.answers.queue.choice,
refund: r.answers.refund.probability,
urgency: r.answers.urgency.score,
confidence: r.answers.queue.confidence,
};
Set sandbox.allowNetwork to true. Do not pass prompt or depth. Do not turn a low confidence into a different label inside this step. Return the answers and let a later step branch.
Speed and cost
Ballet has not timed an evaluation agent against a chat model on our own playbooks. The figures below are published elsewhere. They describe a short routing decision, not a full Ballet run. The later step that writes a reply is a separate model call and is not in these numbers.
List price. Jev, the evaluation model Ballet uses (typesafe-ai/jev), costs $0.042 per million input tokens. Output tokens are not charged. TypeSafe states a latency range of 70–500ms, and that range is not a service level. Source: Learn Jev, benchmarks and cost, quoting TypeSafe's models page.
An independent routing test. AY Automate sent the same intent-routing items to Jev and four chat models and recorded wall-clock time and the bill. This is their test, not Ballet's.
| Model | Median time | Cost per 1,000 eight-way decisions |
|---|---|---|
| Jev 1.13 | 0.33s | $0.015 |
| Gemini 3.5 Flash-Lite | 0.67s | $0.087 |
| GPT-5.4 nano | 1.15s | $0.070 |
| Claude Haiku 4.5 | 1.02s | $0.357 |
| GPT-5.6 Terra | 1.17s | $0.609 |
Source: AY Automate, Jev vs GPT and Claude. Costs are rounded from their table.
The chart compares Jev with the fastest chat model in that test, Gemini 3.5 Flash-Lite. Jev's median was about half the time, and the cost per 1,000 decisions was about one sixth. Every chat model in the table cost more than Jev.
What this does not claim
These numbers are not a Ballet service level and not a measurement of a customer workload. A playbook that classifies and then drafts a reply still pays for that draft. A compiled playbook that never calls a model is a different path, measured in Deterministic operations, measured.
Recipes
- Triage a support email
- Route a deal review
- Score and route a lead
- Catch a refund request
- Check the work before a reply