# Vision Evals - Benchmarking My Camera Alerts
> A benchmark for the model that writes my security camera alerts
Canonical: https://devonpeet.com/projects/vision-evals
- Company: Personal project - home automation
- Role: Designer and builder - defined what a good alert is, hand-reviewed the answer key, designed the results explorer and built the grader with AI
- Duration: July to November 2026, on the side
* **44** models and settings graded on the same 20 real camera scenes
* **13.4s -> 1.9s** 90th percentile wait, the top score vs the model I picked
* **1 of ~2,000** reports where a blind reviewer overturned a confident grader answer
Every time someone walks up to our house, a vision model looks at two camera snapshots and writes a one line alert, and Home Assistant sends it to our phones. I built an eval to choose which model runs, graded 44 models and settings on 20 real scenes from our cameras, and picked the #7 over the #1, because it said nothing false in 60 alerts, answers in under 2 seconds and costs about a dollar a month.
## The Challenge
How do you know a new model is better? Evals are an essential part of AI development, and a lot of people are skipping them. I was using Gemini 2.5 Flash because it was cheap, fast and already hooked up. Then this summer Google came out with a new release every month, and I noticed one model wasn't as good as the previous one, which made me want to bench all of them against everything else I had.
The obvious first stop is public benchmarks, and there are tons of issues with them:
- They're one-dimensional. Trying to get all the facets of an LLM into a single number is crazy at best.
- A huge amount of an LLM's effectiveness depends on the context you give it and the problems you give it, and no public benchmark knows yours.
- Anything in a model's training set can be benched for. If the model already knows the answer, of course it's going to get the test right, which is part of why [OpenAI stopped reporting SWE-bench Verified](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/). Their audit also found material issues in the tests or problem descriptions of 59.4% of the 138 problems they checked, so even a major public benchmark can have a bad answer key.
- "Benchmaxxed" is literally the word people use for models that bench above their usefulness in practice.
For a designer this is the analysis side of UX: did your design actually land? It's the same way I'd use Pendo to figure out where to put more design work or whether a design choice paid off. When we deployed our ArovyAI chat at Arovy, we used these same techniques to improve the bot's responses and make sure it was actually usable, but that eval has sensitive data, so I can't show it here. This is a hobby project where I applied all of the same principles, and the two fed each other: I'd learn something at home and take it to work, then learn something at work and take it home.
## The Solution
Most of the time nobody opens the camera feed after an alert, so the model's title of eight words or fewer and description of twenty-five words or fewer are the whole message. That meant I first had to decide what a good alert does for the person reading it. I landed on three criteria: Trust is whether it told me anything false, Complete is whether it told me everything true that matters, and Present is whether it's useful and well written, which a separate LLM judge grades from 1 to 5. Trust and Complete both come down to checking whether the alert tells the reader that something is true.
I wanted a system that would let me be hands-off, and that meant a deterministic first pass. Language made that really difficult. My first grader matched keywords, with about 200 lines of negation handling trying to tell "a dog" from "no dog", and the alerts kept breaking it:
- "Pepper Patrols The Driveway" never says dog, and Pepper is our dog.
- A person heading out "with a leash in hand for a late-night walk, likely with Pepper" tells you the dog is there.
- "No dog or package" negates both nouns.
- A dog on a poster or a hoverboard in a bus ad isn't in the scene.
Then I found TypeSafe's Jev, a judgment model that returns a probability for a question instead of writing an answer. You send it a small piece of JSON and typed questions, and it returns a calibrated probability for each one. So I ask several narrow yes/no questions per claim, treat anything above 0.7 as yes and below 0.3 as no, and calculate the credit in code where I can read and change it. Grading 900 responses costs about 3.5 cents. (I have no relationship with TypeSafe, their approach just solves a really key step of evals.)
Claude Opus then checks that work as a blind reviewer. It reads every report for a scene in a shuffled order with no model names, has to settle anything Jev wasn't sure about, and can correct any other answer. After tuning, it overturned one confident Jev answer across about 2,000 reports, which gave me enough confidence to stop reading every response by hand. The reviewer mattered before that too: a model read a Safelite van as "Seattle AutoGlass", Jev treated that as Safelite at full confidence because its list had no option for an unlisted company, and only the reviewer caught it, so I added a question for unlisted companies.
To read all of this I designed a results explorer and implemented it with AI. It has a decision view, a leaderboard with a scatter plot of the trade-offs, a matrix of every model against every scene, and a view of each scene with every model's response and the exact reasons it passed or failed.
The results explorer: the leaderboard and trade-off scatter for all 44 models and settings, the model-by-scene matrix, and the written responses for five scenes. Camera frames are left out. Open it full page.
## Key Decisions & Rationale
- You should always know what's in your gold set (the list of correct answers), because it shouldn't just be generated. An AI drafted some of my answer key, including a night scene it labeled as a cat, and I accepted that. The camera's own detector said cat too, and most of the responses said cat. When I finally reviewed the gold by hand and looked at the frame, it was a rabbit. I fixed the gold before running the new models, and calling a rabbit a cat now earns partial credit, because it's wrong but it beats calling it the family dog.
- A lie should cost more than an omission. This is a message you act on without checking, so Trust is cubed in the final score, which means a model that says something false in 1 alert out of 10 keeps only about 73% of its score. Complete and Present only count on trustworthy alerts, since there's no point scoring how thorough a message is when it's already lying to you.
- Keep the answer key away from the grader. Giving Jev background facts, like who lives here, helped it read "Pepper" correctly. Giving it the scene's reference description leaked the answer: on reports with no dog in them, its probability that a dog was there rose by 0.19 on average.
- Test the model and the prompt together. Fable 5.1 was no better than Fable 5 on the stock prompt, but it jumped to #2 of 44 with a prompt tuned for its model family, and 13 of 18 models gained from tuning. Tuning a prompt for a real job is different from benchmaxxing, but I did it on the same 20 scenes, so I was explicit about trying not to overfit: one change per test, and scenes that weren't allowed to get worse. It's still an area of improvement, and more scenes would show whether the gains hold.
## Outcomes
The bench changed what I ran as early as August. Ten scenes favored Gemini 3.5 Flash, then doubling to twenty reversed that to Gemini 3.7 Flash, at less than a third of the price, so I switched that day.
With the new grader I could compare the whole lineup alongside speed and cost: 19 cloud models plus 25 local models and settings. Gemini 4 Argon scored highest at 0.839, but at the 90th percentile it takes 13.4 seconds when I want this alert in under 6 and ideally under 3, and Fable 5.1 at #2 costs about 31 times more. Our cameras now run on Gemini 3.5 Flash-Lite, #7 at 0.786, with no false statements in 60 alerts, 0.910 on Complete, 1.9 seconds at the 90th percentile and about a dollar a month at our real volume of 1,120 calls. I chose GPT-6.1 Terra at #13 as the backup. It's less complete and costs twice as much per alert, but it only gets used a handful of times a month, so I cared more about speed there.
Twenty scenes is a small sample, so I can't assume this ranking would hold across every situation our cameras see, but it gave me enough to choose a model for our house.
## Reflection
I thought my reviewer was free because it ran on my Claude subscription. It turns out that sourcing my `.env` handed the CLI an API key that takes precedence over the login, so it billed $200 of API credit until the credit ran out, on a round of testing that otherwise cost about $14. I fixed it by stripping those credentials from the reviewer's environment.
I set out to be hands-off and mostly got there, but only for the measuring. You can hand the grading to a system once you've tested it, and you should never hand over the decision of what good looks like.
To read all of this I designed a results explorer and implemented it with AI. It has a decision view, a leaderboard with a scatter plot of the trade-offs, a matrix of every model against every scene, and a view of each scene with every model's response and the exact reasons it passed or failed.