Evaluating Agents With Real Users, Not Just Benchmarks
Benchmarks are a starting line. Here is how to run lightweight user evals every week and turn feedback into a shipping cadence.
Public benchmarks tell you your agent is not embarrassing. They do not tell you it is useful. The only signal that matters is whether the last ten real users finished their task.
This piece is a short playbook for running lightweight user evals every week without turning your team into a research lab.
Weekly transcript review
Block thirty minutes every Friday to read ten random transcripts. Note where the agent stalled, where it hallucinated, and where the user gave up. Fix the top issue before Monday.
- Pick transcripts at random, not the ones you already know are bad.
- Read them end to end, including the user's follow-up messages.
- Write one sentence per transcript describing the failure or success.
Ratings are lagging
Users rate what they remember, not what happened. Instrument step-level signals: retries, tool errors, and time to first useful reply. Those tell you what to fix before ratings drop.
A five-star rating from a user who never came back is not a signal, it is a farewell.
Ship a small fix every week
One prompt change, one tool tweak, or one guardrail per week is more than enough. Publish the changelog to your listing page and let buyers watch the product get better in public.
Do this for a quarter and your ratings will climb without you ever chasing them directly.
