AI spend and model operations · 1,295 words · 6 min read · Updated
AI Evaluation Before Launch
A production evaluation method for deciding whether an AI workflow is ready for a bounded launch.
Evaluate the workflow users will receive
Offline model quality is necessary but incomplete. Production behavior also depends on retrieval, prompts, tools, schemas, validation, fallbacks, integrations, review interfaces, and the quality of incoming data. Evaluate the assembled path that users will rely on.
The launch question is bounded: Is this version good enough for these users, tasks, inputs, review rules, and downstream actions? Avoid asking whether the model is accurate in general. That question has no stable operating meaning.
Critical rules come before averages
A workflow can perform well on average and still fail launch because one severe error crosses the boundary. Define critical failures such as unsupported material claims, prohibited data exposure, wrong high-impact routing, or action without required review.
Known limits are part of approval
Approval does not require solving every future case. It requires naming unsupported classes and routing them safely. A launch record should make clear what remains manual, what requires review, and what the system refuses.
Build the launch evaluation
The evaluation should be repeatable when the workflow changes.
- 01
Freeze the release candidate
Version model, prompt, retrieval, tools, schema, routing, validation, and review interface. Do not evaluate a moving target. - 02
Define the launch boundary
Name users, task classes, source types, output, human decision, downstream action, and excluded use. - 03
Assemble representative examples
Include ordinary, difficult, incomplete, conflicting, sensitive, recent, and previously failed cases. - 04
Write expected outcomes
State required content, acceptable variation, critical failures, review action, and escalation for each class. - 05
Calibrate reviewers
Have reviewers score shared examples, discuss disagreement, and refine guidance before measuring the candidate. - 06
Test the full path
Run source retrieval, model calls, tools, validation, interface, integration, logging, fallback, and rollback. - 07
Review operational behavior
Measure latency, timeouts, retries, rate limits, cost, support needs, and the effect of realistic concurrency. - 08
Make and record the decision
Approve, narrow, delay, or reject the release with evidence, known limits, owners, monitoring, and rollback conditions.
Evaluation dimensions and launch evidence
Choose measures that reflect the workflow action. Not every dimension needs one universal score.
- Dimension
- Task correctness
- Question
- Does the output satisfy the job and required format?
- Evidence
- Reviewer labels against expected outcomes.
- Possible launch condition
- Accepted output stays inside the agreed band by task class.
- Dimension
- Grounding
- Question
- Can material claims be tied to approved sources?
- Evidence
- Source checks, citation coverage, unsupported-claim labels.
- Possible launch condition
- Critical unsupported claims are blocked or routed to review.
- Dimension
- Safety and policy
- Question
- Does the workflow avoid prohibited content, action, or data use?
- Evidence
- Sensitive and adversarial cases plus control tests.
- Possible launch condition
- No critical policy failure in the approved boundary.
- Dimension
- Reviewability
- Question
- Can reviewers understand, correct, reject, and escalate?
- Evidence
- Observed review sessions and decision logs.
- Possible launch condition
- Reviewers can apply guidance consistently within operating time.
- Dimension
- Operational fit
- Question
- Does the system work under realistic latency, volume, and failure conditions?
- Evidence
- Load, timeout, dependency, retry, and fallback tests.
- Possible launch condition
- Service and fallback remain usable for the launch group.
- Dimension
- Economics
- Question
- Is expected usage supportable and attributable?
- Evidence
- Task-level calls, retries, fallbacks, output size, and review burden.
- Possible launch condition
- Owners accept the measured cost and monitoring plan.
- Dimension
- Adoption
- Question
- Can users begin and complete work without hidden burden?
- Evidence
- Workflow walkthroughs, training feedback, and pilot observation.
- Possible launch condition
- Users understand the boundary and have a correction path.
- Dimension
- Recovery
- Question
- Can the team pause and continue the work safely?
- Evidence
- Rollback exercise and prior-process test.
- Possible launch condition
- Named owners can restore the approved fallback.
Launch evidence maturity
Use the rubric to distinguish a persuasive demonstration from an approvable release.
Example coverage
- Weak
- A few successful examples selected by the build team.
- Workable
- Normal and difficult cases exist, but production failures and sensitive classes are thin.
- Strong
- Representative classes, known edge cases, prior failures, and critical scenarios are intentionally covered.
Acceptance rules
- Weak
- Reviewers use general impressions.
- Workable
- Guidance exists but severity and disagreement handling are incomplete.
- Strong
- Expected outcomes, critical failures, severity, and reviewer actions are written and calibrated.
System fidelity
- Weak
- Evaluation uses a model playground or simplified prompt.
- Workable
- Core application logic is present but integrations or fallback are simulated.
- Strong
- The release candidate runs through production-equivalent retrieval, tools, validation, review, logs, and dependencies.
Operating decision
- Weak
- Launch means the demo looked ready.
- Workable
- Measures exist but ownership and stop conditions are informal.
- Strong
- Boundary, owners, known limits, monitoring bands, support, and rollback are accepted in writing.
Make the launch decision
A no answer should narrow or delay the boundary rather than disappear into an average score.
- 01
Did the candidate pass every critical rule?
- If yes
- Continue to reviewer and operational evidence.
- If no
- Block launch for the affected class, fix the control, or route the class to human handling.
- 02
Can reviewers apply the guidance consistently?
- If yes
- Confirm the work fits the available review capacity.
- If no
- Clarify expected outcomes, calibrate reviewers, or narrow the task.
- 03
Does the full system remain usable under realistic conditions?
- If yes
- Review cost, adoption, and recovery.
- If no
- Resolve integration, latency, dependency, retry, or fallback failures before launch.
- 04
Can owners support the measured cost and operating load?
- If yes
- Approve monitoring bands and release scope.
- If no
- Redesign routing, context, volume, review, or the user boundary.
- 05
Can the workflow be paused without losing essential work?
- If yes
- Approve the bounded release and record known limits.
- If no
- Complete rollback and continuity work first.
Launch packet
Keep the packet with the release so future changes can reuse the evidence.
- ✓
Versioned workflow specification
User, task, data, model action, output, integration, review, and downstream action. - ✓
Evaluation set and labels
Examples, expected outcomes, reviewer decisions, failures, severity, and unresolved disagreement. - ✓
Known-limit register
Unsupported inputs, manual classes, mandatory review, refusals, and untested assumptions. - ✓
Operational test record
Latency, dependency failure, retries, fallback, cost, logging, and realistic-volume observations. - ✓
Owner acceptance
Business, technical, review, support, and control owners accept their responsibilities. - ✓
Release and rollback plan
Users, traffic or task boundary, monitoring band, stop condition, rollback route, and communication.
Evaluation records need precise names
These terms keep evidence and production monitoring connected.
Evaluation set
- A versioned collection of representative workflow examples used to compare releases and routes.
Critical failure
- An error that blocks release for the affected boundary regardless of average performance.
Acceptance rule
- A written condition that determines whether output can proceed, needs correction, must be rejected, or requires escalation.
Monitoring band
- The expected production range for quality, exceptions, latency, cost, or fallback that triggers investigation when crossed.
Evaluation continues after launch
Launch evidence is a starting baseline. Production introduces new inputs, user behavior, source changes, provider incidents, and model updates. Capture rejected outputs, corrections, fallbacks, and incidents in categories that can improve the evaluation set.
Re-evaluate material changes before broad release. A new model, prompt, retrieval source, tool permission, user group, action, or input class can invalidate the old boundary even if the workflow keeps the same name.
The durable asset is not one score. It is the connected evidence system: examples, rules, reviewer decisions, release versions, production observations, and a clear record of what the organization has actually approved.
Questions this article answers
How large must an evaluation set be before launch?
It must cover the decisions and failure classes that matter for the approved boundary. A smaller set with deliberate critical coverage can be more useful than a large undifferentiated sample.
Should reviewers be domain experts?
Reviewers need enough authority and task knowledge to judge the output. Some workflows require domain experts; others can use trained operators with clear guidance and escalation.
Can automated metrics replace human review?
Automated checks are valuable for schemas, exact fields, source links, policy patterns, and regression. They do not replace human judgment where correctness, usefulness, context, or consequence cannot be reduced to a deterministic rule.
What should happen when reviewers disagree?
Record the disagreement, identify the decision owner, refine guidance, and re-score shared examples. Hidden disagreement is evidence that the acceptance rule is not ready.