How to Evaluate an LLM Feature Before and After Launch

An LLM feature can produce fluent answers and still fail the job users need done. Evaluation turns vague impressions like “the new model feels better” into repeatable evidence about whether a change improves the product. Treat evaluation as part of feature design and release, not a one-time benchmark after the model is chosen.

Start with the user task and the cost of failure. Write down what a good answer must do, what it must never do, when it should ask a follow-up, and when it should abstain or hand off to a person. Separate requirements such as factual correctness, completeness, groundedness, format validity, tone, latency, and cost; a single overall score can hide a serious regression in one dimension.

Build a small, versioned dataset from realistic cases: common requests, edge cases, ambiguous inputs, multilingual examples, and known failure modes. Include the input, relevant context or retrieved sources, expected properties, and a reference answer only when there is a defensible one. Remove personal or confidential data, record dataset provenance, and keep a held-out set so prompt authors do not tune directly against every evaluation example.

Use deterministic checks wherever the requirement is objective: schema validation, required fields, citation presence, access-control outcomes, or exact calculations. For open-ended quality, combine expert human review with rubric-based model graders. Calibrate graders against human labels, inspect disagreements, and test whether small wording changes distort scores. A model judge is a measurement aid, not ground truth.

Run the same dataset against the current system and every proposed change: model version, system prompt, retrieval settings, tool definitions, or decoding parameters. Compare per-case results and slices by language, user group, task type, and risk. Keep difficult failures in the dataset after they are fixed. Use confidence intervals or repeated runs when model variability could change the conclusion.

Offline scores do not show the whole user experience. In production, monitor task completion, corrections, user escalations, refusal and abstention rates, latency, cost, and safety incidents, with privacy-conscious sampling of outputs. Define release thresholds and rollback conditions in advance. Review a sample with domain experts, because users and real data distributions reveal failures that synthetic prompts miss.

An effective loop is simple: define success, collect representative examples, run automated checks, review uncertain or high-impact cases, compare versions, deploy gradually, and add verified production failures to the next evaluation set. NIST’s Generative AI Profile recommends documented testing and ongoing monitoring; the OpenAI Evals guide describes reusable datasets and graders. References: https://airc.nist.gov/airmf-resources/airmf/5-sec-core/ and https://platform.openai.com/docs/guides/evals