Arenas
Each arena takes one real engineering decision from
enable-ai, an AI-engineering news
product, and settles it by measurement: several options behind one interface, scored on the
same data for quality, latency and cost together.
Picks come from fixed rules and paired significance tests, not from an LLM or a hunch, so
the same report always gives the same answer. Every card says where its labels came from.
Can a voice agent's sentences be checked against its sources before the caller hears them, without slowing the call?
On 42 unsupported sentences (4 natural, 38 planted) across 1 held-out scenario, cascade-jev caught 90% while blocking 2.6% of correct sentences, adding 161 ms p95 before the first word at $0.22 per 1,000 calls. Ignoring the limits, cascade-jev-calibrated caught the most (93%), not a significant difference.
| Contender | Wrong sentences caught |
p95 added silence |
Cost / 1k |
|---|
| cascade-jev pick | 90% | 161 ms | $0.2177 |
| cascade-jev-v2 | 90% | 170 ms | $0.2550 |
| jev-1.14 | 86% | 177 ms | $0.4348 |
| jev-1.14-v2 | 86% | 188 ms | $0.5098 |
| none | 0% | 0 ms | $0 |
+ 13 more in the snapshot
- Mixed labels
- 221 items
- No product decision
Research study for production voice agents, not an enable-ai decision. The check ships as a standalone library (packages/speech-gate).
Which retrieval setup finds the existing story a new item belongs to, so enable-ai's feed doesn't show the same news twice?
Use hybrid-voyage (recall@5 96%, right story first 82%, 245ms median, $0.0118 per 1k). Production today (its logged decisions): 90.9% correct end to end; 31% of items that belong to an existing story became a duplicate story, and 2% of all items were merged into the wrong story. Production's logged candidates contained the right story for 84.5% of matching items, so 15% never reached the judge. openai-3-large reaches 97.3% (paired McNemar p=0.0005).
| Contender | Right story in top 5 |
p50 |
Cost / 1k |
|---|
| hybrid-voyage pick | 96% | 245 ms | $0.0118 |
| gemini-emb-2 | 97% | 610 ms | $0.0368 |
| openai-3-large | 97% | 396 ms | $0.0247 |
| qwen3-embed-8b | 97% | 688 ms | $0.0022 |
| voyage-4 | 96% | 245 ms | $0.0118 |
+ 3 more in the snapshot
- LLM-judge labels
- 112 items
- Decision pending
Finding - production's 0.10 cosine cutoff dropped the right story for 15% of matching items; 0.20 recovers most. Link the enable-ai change here once it ships.
What is the cheapest way to catch unsupported LLM-extracted fields without rejecting correct ones?
On 80 held-out public sources, passage-window LLM checking retained 117/122 supported natural claims and caught the sole labeled natural error, at 45% less verification cost than full-source checking. Fixed-rule pick over all 217 scored fields: value-in-evidence.
| Contender | Balanced accuracy |
p50 |
Cost / 1k |
|---|
| value-in-evidence pick | 95% | 0 ms | $0 |
| llm-passage-window | 97% | 1.0 s | $0.2311 |
| llm-full-source | 94% | 1.0 s | $0.4253 |
| hhem-passage-window | 80% | 48 ms | $0 |
| exact-quote | 50% | 0 ms | $0 |
- LLM-judge labels
- 271 items
- Decision pending
No enable-ai production behavior has been changed by this experiment yet.
For bounded classification (pick a label, yes/no, a score), should a decision model like Jev make the call, or a cheap LLM?
Use jev-1.14 (99% accurate, 185ms median, $0.0183 per 1k calls). Best decision model (jev-1.14, 99%) vs best LLM (haiku-5.5, 98%): no significant accuracy difference; the decision model is 7.7× faster and 3.3× cheaper.
| Contender | Accuracy |
p50 |
Cost / 1k |
|---|
| jev-1.14 pick | 99% | 185 ms | $0.0183 |
| clef-flash | 98% | 370 ms | $0.0056 |
| haiku-5.5 | 98% | 1.4 s | $0.0608 |
| gpt-5.4-nano | 97% | 1.1 s | $0.0856 |
| liquid-d1 | 97% | 229 ms | $0.0031 |
+ 6 more in the snapshot
- Hand-written labels
- 119 items
- Decision pending
Not yet recorded which enable-ai decisions moved because of this result.