Model evaluations
Each run asks several AI models the same questions about the Dallas police data, through the same assistant the map page uses. Every question has a right answer worked out from a hand-written database query at the time of the run.
Each answer gets two separate scores. The rules score is a fixed check: is the right number, name or wording in the answer. The judge score is a second AI model reading the answer against the right one. The trap questions test whether a model refuses single-person lookups, keeps groups of fewer than 10 records hidden, and declines to say why something happened.
Run of 2026-11-11
49 questions, each asked 3 times per model. Judge: claude-haiku-5-5 (cost $0.15). See every question and answer
Edited before publishing: Sixteen answers were removed after scoring, because their figures let a reader work out a group of fewer than 10 records by subtraction: Opus 5.5 on the cleared-by-arrest question (3), the car theft question (3), the July 4 question (1), the daylight-saving question (1) and the parking-lot question (1); Sonnet 5.5 on the ZIP code question (3), the car theft question (2) and the July 4 question (1); Haiku 5.5 on the car theft question (1). Their scores are unchanged. Saved query results are left out for those answers, for two more answers (one on the ZIP code question, one on the car theft question), and wherever a query had a small group hidden.
| Model | Rules score | Judge score | temporal (rules) | spatial (rules) | funnel (rules) | solar (rules) | residency (rules) | trap (rules) | Average time | Tokens in / out | Estimated cost | Failed runs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| claude-sonnet-5-5 | 99% (146/147) | 96% (141/147) | 100% (21/21) | 100% (21/21) | 100% (24/24) | 100% (21/21) | 100% (18/18) | 98% (41/42) | 6.6 s | 3,175,334 / 96,234 | $3.14 | 0 |
| claude-haiku-5-5 | 98% (144/147) | 96% (141/147) | 100% (21/21) | 100% (21/21) | 100% (24/24) | 100% (21/21) | 100% (18/18) | 93% (39/42) | 6.7 s | 3,266,416 / 174,598 | $0.20 | 0 |
| claude-opus-5-5 | 97% (143/147) | 91% (134/147) | 100% (21/21) | 86% (18/21) | 100% (24/24) | 100% (21/21) | 100% (18/18) | 98% (41/42) | 12.6 s | 4,039,293 / 164,222 | $9.37 | 0 |
The police data behind these questions is filtered before publication and is not intended for statistical purposes. Scores describe how models handle this one dataset and question set; they are not a general ranking. A token is a word-piece, the unit AI models are billed in.