Loading…
Jev vs Claude Sonnet 5: sorting 1,998 support messages
TL;DR: We sent the same 1,998 customer-support messages to Jev and Claude Sonnet 5. Each message got the same three typed questions. Jev cost about 46 times less at list price ($0.095 vs $4.34) and finished the batch in half the time (110 seconds vs 220 seconds). Sonnet 5 was slightly more accurate, by 0.7 to 1.5 points on each question.
Who this is for
Anyone deciding which model should sort support messages, leads, or tickets before a playbook acts. The job here is classification, not writing a reply.
What we measured
Both models read the same 1,998 messages from the Bitext customer-support dataset (CDLA-Sharing-1.0). The messages are synthetic and short. Each one has a labeled category (1 of 11) and a labeled intent (1 of 27). We took 74 messages for each intent.
Each message got the same three questions, with the same option text:
- Category. A choice among 11 options.
- Intent. A choice among 27 options.
- Refund. A yes or no. The label is true when the category is refund.
Jev is typesafe-ai/jev, the model Ballet's evaluation agent calls. Claude Sonnet 5 is anthropic/claude-sonnet-5. Sonnet answered in JSON only, at temperature 0, with no tools and no reasoning mode. Both models ran through the same AI gateway, 16 calls at a time, with retries on rate limits and server errors.
Cost is the gateway list price for each call. Jev billed $0 on this account during the run, so the comparison uses list price for both models. Sonnet was billed $4.54, which includes the gateway surcharge.
The run was on 23 September 2026.
Results
| Jev | Claude Sonnet 5 | |
|---|---|---|
| Category correct (11 options) | 96.4% | 97.1% |
| Intent correct (27 options) | 92.7% | 94.2% |
| Refund question correct | 98.0% | 98.7% |
| Cost, list price | $0.095 | $4.34 |
| Cost per 1,000 messages | $0.047 | $2.17 |
| Whole batch, 16 at a time | 110 s | 220 s |
| Median time per message | 0.43 s | 1.68 s |
| 95th percentile time | 2.6 s | 2.3 s |
The two models gave the same answer to all three questions on 1,771 of 1,998 messages (88.6%).
At 100,000 messages, the same list prices come to about $4.75 for Jev and $217 for Sonnet 5. That is the classification step only. A later step that drafts a reply is a separate model call and is not in these numbers.
Watch the run
The film replays the real calls at 10× speed. Each square is one message. The colour is the category the model chose. A dark centre means that category was wrong.
jev-vs-sonnet-5-sorting-1998-support-messages.mp4 640x360
What this does not claim
- List price, not the bill. Jev billed $0 on this account. Sonnet billed $4.54. The headline comparison uses list price so both models are on the same scale.
- Rate limits. Jev hit HTTP 429s. At 16 parallel calls it needed 170 retries and finished with no failures. At 32 parallel calls, 228 calls failed after retries. Those waits are in Jev's 95th-percentile time, which is why that number is slower than Sonnet's even though the median is about 4 times faster. Rate limits depend on the account.
- Synthetic messages. The dataset is short and single-intent. Real inboxes are longer and messier, so absolute accuracy will differ. The licence allows commercial use.
- JSON only. Sonnet did not explain itself and did not call tools. A Sonnet agent that reasons or uses tools costs more than this.
- Not a service level. These figures are one Ballet measurement of a classification task. They are not a latency or price guarantee.
Why this matters for automation
Sorting a support message is a known question with a known set of answers. An evaluation agent returns the choice and a confidence. A later playbook step routes the ticket or drafts the reply. Paying a general chat model to re-decide that sort, on every message, is most of the cost in the table above.
Recipes that use this pattern: