A Spec That Can Say No
A written spec is a promise. An eval is a promise that can check itself.
For most of my career, specifications were documents. Some were long and formal, some were a few bullet points in a ticket, some lived only in someone’s head. They all shared one weakness: they couldn’t tell you when they were being violated. A spec could say “the export should be fast and reliable,” and the product could quietly become slow and fragile, and the spec would sit there, unchanged and unbothered, in a folder nobody opened.
At Charsook, I’ve started writing evals before I write the feature. Not after, not “when we have time,” but first. It sounds like test-driven development, and it is a cousin of it, but it is different in one important way: most of what we build is fuzzy. There is no single correct output for “suggest a motion curve for this title card” or “clean up this edit.” There are better and worse outputs, and the difference between them is often a matter of judgment.
Unit tests handle the crisp parts. Evals handle the rest. And in an AI-native product, “the rest” is most of what users actually care about.
Why This Became Urgent
When software was fully written by humans, behavior changed only when someone changed the code. You could review the change, understand it, and reason about its effects. Quality drifted slowly, and usually someone noticed.
AI products are different in two ways.
First, their behavior depends on things that are not code in the traditional sense: prompts, model versions, retrieval settings, examples, temperature, and the training data of a model you didn’t build. Any of these can change, sometimes without you choosing to change it. A model provider updates a version, and your carefully tuned behavior shifts. Nobody wrote a line of code, yet the product is different.
Second, more and more of the changes are made by AI agents. At Charsook, agents modify prompts, refactor pipelines, and propose new approaches. They’re fast and often clever. But an agent changing a prompt has no intuition for whether the product just got better or just got different. Neither do I, honestly, unless I have a way to measure it.
So evals became the thing that holds the product steady while everything around it moves. They are the memory of what we decided “good” means.
What an Eval Actually Is
People sometimes imagine evals as a complex research apparatus. They can be, but the core is simple. For us an eval is three things: a set of inputs that represent real use, a way to score the outputs, and a threshold we agree is good enough to ship.
That’s it. The sophistication comes from doing each of these three things honestly.
The Inputs Matter Most
Early on I made the classic mistake of using clean, made-up examples. I wrote twenty tidy test cases, the kind you’d put in a demo: a well-organized project, a clear request, a standard aspect ratio. Everything scored beautifully. I felt very good about our quality.
Then real users arrived. They had messy files with inconsistent naming. Half-finished projects abandoned for weeks. Requests written in a mix of Persian and English, sometimes in the same sentence. Projects with three hundred clips when we had imagined thirty. Fonts we had never heard of. Expectations we had never imagined.
The scores meant nothing, because they measured a world that didn’t exist.
Now we collect real cases, with permission, and we keep the weird ones. Especially the weird ones. A good eval set is not a representative sample of average use. It’s a curated collection of the situations where products break: the edges, the surprises, the cases where a user did something reasonable that we didn’t anticipate.
I’ve come to think of our eval set as a kind of institutional memory. Every time something goes wrong in production, the case goes into the set. Over time, the set becomes a record of every lesson the product has learned the hard way. When a new team member or a new agent touches the system, the eval set tells them, in effect: “here is everything that has ever gone wrong; don’t let it happen again.”
Scoring the Unscorable
Some things are easy to score. Did the export succeed? Is the JSON valid? Does the timing line up with the audio? These are binary checks, and we write them the same way we’d write any test.
Some things need judgment. Is this animation smooth, or does it feel cheap? Is this layout readable at phone size? Does this suggested cut respect the rhythm of the music? Does this explanation make sense to a beginner?
For judgment, we use a mix of approaches, and I’ve learned that each has its strengths and dangers.
Model-based graders are fast and cheap. You give a model a rubric and an output, and it gives you a score. The danger is that a vague rubric produces vague scores. “Rate the quality from 1 to 10” is almost useless. “Does the primary action stand out more than any other element? Answer yes or no and point to the evidence” is much better. The more specific the question, the more trustworthy the answer.
Pairwise comparisons are, in my experience, the most reliable technique for fuzzy quality. Instead of asking “how good is this?”, you ask “which of these two is better?” Humans and models are both much better at comparison than at absolute judgment. You can feel this yourself: it’s hard to say whether a wine is a 7 or an 8, but easy to say which of two glasses you prefer. When we compare a new version of a feature to the old one, pairwise judgments across a hundred cases give us a far clearer picture than any average score.
Human review is the anchor. Every grader, whether model-based or rule-based, drifts or has blind spots. Regular human review, even of a small sample, keeps the automatic graders honest.
Never Trust a Grader You Haven’t Disagreed With
The rule I hold myself to: never trust a grader I haven’t disagreed with.
If I’ve never looked at a grader’s scores and thought “no, that’s wrong,” I haven’t looked closely enough. Every grader has failure modes. Model graders often prefer longer outputs, more confident language, or outputs that resemble their own style. Rule-based graders reward whatever the rule measures and ignore everything else.
So part of building an eval is evaluating the evaluator. We take a sample of cases, score them by hand, and compare to the grader. Where they disagree, we look closely. Sometimes the grader is wrong and we fix the rubric. Sometimes, interestingly, the grader is right and I was wrong, and I learn something about my own blind spots.
This is slow, careful work, and it is exactly the kind of work that’s easy to skip. It’s also the work that determines whether your evals are a real instrument or a comforting illusion.
Evals Change Conversations
The best side effect of serious evals is cultural.
Arguments about quality used to be about opinions. Someone would say “I think the new version feels better,” someone else would disagree, and the decision went to whoever was more senior or more stubborn. These arguments were exhausting and rarely productive.
Now they’re about cases. “This new prompt is better” becomes “this new prompt wins most of these cases and loses these few, and here’s what the losing cases have in common.” That last part is the most valuable. The losing cases tell you exactly where the new approach is weak, which often points directly to the next improvement.
It also changes how we talk about tradeoffs. Almost every change improves some cases and hurts others. Without evals, you only notice the improvements you were hoping for. With evals, you see the full picture, and you can make an honest decision about whether the trade is worth it.
Evals and AI Agents
Evals matter even more when AI agents are part of your engineering process.
When an agent changes a prompt, a model, or a pipeline, the eval suite tells us whether it made things better or just different. Without that, every change is a gamble. With it, agents can explore much more freely, because there’s a safety net that catches regressions.
There’s a subtle point here. Agents are very good at optimizing for whatever you measure. If your eval only checks that outputs are valid JSON, an agent will happily produce valid, useless JSON. If your eval only rewards brevity, it will produce terse outputs that miss the point. Evals don’t just measure quality; they define it, and agents will take that definition literally.
This is why I say evals are the new spec. In a world where agents do much of the building, the eval is the most precise statement of what you actually want. If it’s incomplete, the product will be incomplete in exactly the same places.
What Makes an Eval Good
Over time I’ve developed a few principles for evals that actually help.
Use real data. Synthetic examples have their place, especially for coverage of rare cases, but the core of your set should come from real use.
Keep the failures. Every production problem becomes a case. The set should grow with the product’s scars.
Prefer specific questions. Ask graders narrow, answerable questions rather than broad ratings.
Compare, don’t rate. Pairwise judgments beat absolute scores for anything subjective.
Check the checker. Regularly compare automated grades to human judgment.
Make it fast. If running evals takes hours, people will stop running them. We aim for a core set that runs in minutes, with a larger set for important releases.
Read the outputs. Numbers are summaries. Every few days I read raw outputs from the eval run, not just the scores. Patterns that numbers hide become obvious when you look at fifty examples in a row.
Evals as a Philosophy
Beyond the practical benefits, I’ve come to see evals as a kind of intellectual honesty.
Writing an eval forces you to answer the question “what would convince me I’m wrong?” before you start building. That’s a hard question, and most product development avoids it. We fall in love with ideas, and then we look for evidence that they’re working.
An eval written first is a commitment made before the emotional investment. It says: here’s what success looks like, and I’ll accept the result even if it disappoints me. In that sense, it’s not so different from how good science works. You state the hypothesis and the test before you run the experiment, not after.
My years coaching mathematics and computer science left me with deep respect for this kind of discipline. A proof doesn’t care how much you want the theorem to be true. An eval doesn’t care how much you love the feature. That indifference is exactly what makes them valuable.
Start Smaller Than You Think
If you’re building with models and you have no evals, don’t build a framework. Don’t evaluate tools for a week. Don’t design a dashboard.
Write twenty real examples in a file. Write down what good looks like for each, in plain language. Run them before every change and read the outputs yourself.
That’s it. The first twenty cases will teach you more about your product than a month of roadmap meetings. They’ll show you where your assumptions were wrong, where users surprise you, and where your product is weaker than you thought.
They did for me. And every week since, the set has grown, and our understanding of what we’re building has grown with it.