B2B AI evals in 2026: when to use tests, LLM judges, human review, or traces

If you are shipping a B2B AI feature, the hardest evaluation question is not “which score should we use?” It is “what would have to be true for this job to count as done?”

A model can write a polished answer while using the wrong source. An agent can call the right tools and still fail to produce the promised artifact. A workflow can pass a test suite and leave a customer with an outcome nobody checked. A judge model can assign a reassuring number because the rubric was vague, the trace was incomplete, or the evaluator rewarded the style of the answer instead of the job itself.

My recommendation: start with the smallest evaluation stack that can answer the decision in front of you. Use deterministic checks for exact contracts and policy boundaries. Add trace and outcome checks when the work has multiple steps. Use an LLM judge for open-ended qualities that code cannot reasonably score, but calibrate it against human examples. Then connect production feedback back to a versioned regression set. Buy a managed evaluation platform when scale, collaboration, simulation, and governance justify the dependency. Postpone the platform decision when you still cannot define success or reconstruct a failed run.

This is not an argument for turning every customer interaction into a score. It is a guide to deciding what evidence is worth collecting before you change a prompt, switch a model, add retrieval, expose a tool, or let an AI workflow touch an important business process.

The short answer: test the job, not just the words

A useful B2B evaluation does three things:

1. It makes success observable. You can point to the fields, source records, state transition, or human decision that proves the job was completed.
2. It catches the failure you actually care about. The test can tell a missing citation from an unauthorized data access, a wrong tool call from a weak sentence, and a slow retry from a harmless variation in phrasing.
3. It teaches the team what to do next. A failed case links to the input, relevant trace, policy decision, evaluator version, and correction—not just a red number.

That usually means assembling several small evaluators instead of searching for one grand metric. A code check can verify that the output is valid JSON, contains a required field, stays within a numeric range, or uses an allowed tool. A trace check can verify that the agent consulted the account record before drafting a renewal brief. An outcome check can verify that the brief actually exists in the expected system. A human can decide whether the brief is useful to an account manager. An LLM judge can help scale a clearly written rubric for a subjective dimension, but it should not be treated as ground truth merely because it returns a decimal.

Current platform guidance is converging on this layered shape. OpenAI documents datasets, annotations, and several grader types. Anthropic separates tasks, trials, graders, transcripts, outcomes, and evaluation harnesses for agents. LangSmith describes offline evaluation on curated datasets and online evaluation on production traces. Google’s July 2026 announcement for its agent evaluation product describes experiments, online monitors, case generation, environment simulation, and multiple metric families. These are vendor-specific implementations, not a universal standard, but they point to the same design problem: AI quality is a lifecycle, not a one-time prompt check.

The practical choice is therefore not “tests or LLM judge.” It is more like this:

If this feels less tidy than a single score, that is the point. Real B2B work has different ways to be wrong. Your evaluation system should preserve those distinctions long enough for someone to fix the right layer.

Why “the final answer looked good” is not enough

Imagine a customer support assistant that answers, “Your replacement is on the way.” The sentence sounds helpful. But three different realities could sit behind it:

• The assistant checked the correct order, found a shipped replacement, and linked the answer to the tracking record.
• The assistant found an older order for the same customer and guessed from a similar case.
• The assistant called no tool at all, produced a plausible sentence, and happened to be right this time.

An answer-only evaluator may rate all three as similar. The customer’s next action will not be similar if the replacement is actually delayed, belongs to another order, or was never created.

Anthropic’s evaluation guidance makes the distinction explicit by defining a transcript as the full record of a trial and an outcome as the final state in the environment. A flight-booking agent can say a reservation was made while the database says otherwise. The lesson travels to any B2B workflow with tools, records, approvals, or external systems: the words are evidence, but they are not the whole event.

Public builders are describing the same gap in less formal language. One recent discussion asked whether production evals were “just vibes with a number attached” and argued that a score becomes more useful when connected to a failure taxonomy, a production trace, human labels, and a regression suite. Another builder described an agent that produced a plausible recruiting answer after skipping the job requirements. A separate thread focused on agents whose outputs stayed correct while their hidden execution path became more expensive and slower.

These are anecdotal signals, not incident rates. Their value is diagnostic: they show the kinds of questions a B2B team asks after an answer-only eval stops being enough.

The three-agreement problem

For a useful evaluation, three things need to line up:

1. The specification: what the team says the system should do.
2. The evaluator: what the test actually rewards.
3. The user or business: what counts as useful in the real workflow.

These can disagree. A spec might say “summarize the account,” while the account manager needs unresolved commitments and source dates. A judge might reward a concise tone while ignoring a missing exception. A business owner might care about whether an approved follow-up happened, while the dashboard reports only answer helpfulness.

Before you increase test volume, ask whether the three agreements are even aimed at the same target. More cases do not repair a definition of done that is wrong.

Start with the contract: what must be true?

The best first eval is often not an LLM judge. It is a written contract with checks that are boring, explicit, and hard to misread.

Take a B2B workflow that turns an inbound request into a structured lead record. The contract might say:

That contract creates several deterministic checks:

• The output parses against the schema.
• The original message ID is present and unchanged.
• Required fields exist, even when their value is an explicit unknown.
• Extracted values can be matched to a source span or source record.
• Enumerated fields use an allowed value.
• High-impact requests produce a review state rather than an external side effect.
• Unknown fields are rejected or ignored according to a documented rule.

OpenAI’s current grader guidance lists string checks, text similarity, score-model graders, Python code execution, and combinations of graders. Its dataset guidance also emphasizes annotations as ground truth and encourages teams to add edge cases as they learn. You do not need to adopt OpenAI’s implementation to use the underlying design principle: where the property is precise and observable, a deterministic check is easier to debug, cheaper to repeat, and less ambiguous than asking another model for an opinion.

What deterministic checks are good at

Use code when there is a clear rule:

• Schema validity and required fields.
• Allowed tools, arguments, destinations, and tenant identifiers.
• Presence of citations, record IDs, timestamps, or source references.
• Numerical bounds, totals, dates, and state transitions.
• Duplicate detection and idempotency keys.
• Forbidden data patterns, secret leakage, or disallowed output fields.
• Latency budgets, retry counts, token ceilings, and loop limits.

These checks do not have to be sophisticated. A small assertion that blocks an unauthorized tool or detects a duplicate write can matter more than a broad “overall quality” score. The critical detail is that the assertion should be tied to a product or policy requirement, not added because it is easy to calculate.

Where deterministic checks break

Code becomes brittle when it mistakes one acceptable expression for the only acceptable expression. Exact string matching is a poor way to grade a genuinely open-ended explanation. A response can be correct while using different wording, a different order, or a shorter path.

Anthropic’s guidance warns about rigid grading and ambiguous tasks with a concrete example: a valid numeric answer was initially penalized because the grader expected an overly precise string, and other evaluation scores changed dramatically after task and harness problems were fixed. The specific benchmark is not a recipe for your product, but the failure mode is widely portable. When an eval fails, investigate the grader and task as carefully as the model.

The practical answer is not to abandon deterministic checks. Keep them for the rules they can express, and move only the genuinely subjective dimension to a human or model-based evaluator.

Grade the path: what happened between input and answer?

Once a workflow has retrieval, tools, agents, or external state, add checks for the path. The path is not a demand that every agent follow one ritual sequence. It is a way to verify that the system stayed inside an acceptable operating boundary.

For an account brief, a path check might ask:

• Did the run resolve the authenticated workspace before reading records?
• Did it read the current account record and not a similarly named account?
• Did it retrieve the approved support and renewal sources?
• Did it expose any record outside the caller’s tenant?
• Did it call the messaging tool when the job only allowed a draft?
• Did a retry create a duplicate task?
• Did the run stop after a denied tool call or keep looking for a bypass?

These questions are not all model-quality questions. Some are authorization checks. Some are observability checks. Some are business-state checks. Putting them under the broad label “eval” is fine as long as the system keeps the reason for each check visible.

Choose the right evaluation unit

One of the most consequential design choices is deciding what one case means. It might be:

• a single model call;
• a retrieval operation;
• a tool call and its arguments;
• one conversational turn;
• an entire agent trace;
• a business artifact;
• or the final state in a system of record.

These units answer different questions. A retrieval evaluator can ask whether the right documents were returned. A tool evaluator can ask whether the right function and arguments were selected. A trace evaluator can ask whether the complete run stayed within policy. An outcome evaluator can ask whether the record or artifact exists. A human reviewer can ask whether the result is useful in context.

A recent Langfuse issue asks for evaluation over a whole trace rather than an individual observation because some use cases need the surrounding context. That is a public implementation signal, not a universal product requirement, but it points to a common mistake: attaching the evaluator to the easiest span instead of the unit that matches the job.

Do not make every evaluator consume the entire trace. That can increase cost and privacy exposure. Give each evaluator only the minimum evidence needed. A schema check may need the output. A tenant-boundary check may need identity, target, and policy decision. A business-outcome check may need a record ID and post-run state. A useful evaluator is narrow in both question and input.

Grade outcomes without enforcing a ritual

There is a subtle trap in path evaluation. If you assert that the agent must use tools in one exact order, you may punish a valid shortcut. If you assert only that the final answer contains a phrase, you may miss a dangerous or expensive route.

Grade the invariant instead. If the requirement is “use an authorized current source before making a claim,” check the source identity, authorization, freshness, and claim support. Do not require a particular internal function name unless the function itself is the contract. If the requirement is “do not send until approved,” check the approval state immediately before the side effect. If the requirement is “do not duplicate a task,” check the resulting state and idempotency receipt.

This gives an agent room to solve the job creatively while keeping the business boundary firm. It also makes the evaluator less fragile when you change a provider, SDK, prompt, or orchestration framework.

Use LLM judges for nuance, not authority

An LLM judge is useful when the property is difficult to express as a rigid rule: whether a summary is complete enough for a particular role, whether an answer addresses the customer’s real question, whether a draft follows a tone guide, or whether a response is grounded in the supplied evidence.

It is also another probabilistic model. It can be inconsistent, overconfident, sensitive to phrasing, biased toward fluent answers, and wrong about domain details. If you use one, treat the judge as a component that needs its own dataset, prompt version, model identity, cost record, and calibration sample.

Anthropic’s current agent-evaluation guidance recommends deterministic graders where possible, model graders where necessary, and human graders for additional validation. It also recommends giving model judges a way to return “unknown” when the evidence is insufficient, using clear structured rubrics, and reviewing transcripts to distinguish a real agent failure from a grading bug. OpenAI’s GDPval research similarly says its automated grader is not a replacement for expert graders for that research task. These are different organizations and different settings, but the warning is consistent: automation can scale judgment; it does not erase the need for judgment.

Write a judge rubric a person could use

A weak judge prompt says:

A stronger rubric says:

The second version is still not magic. It gives the judge a narrower task, a defined audience, a way to abstain, and dimensions that can be reviewed. It also produces a result that can guide a fix. If the judge marks “open risks omitted,” the team can inspect the source selection or summarization prompt. If it marks “unknown,” a human can decide whether the case needs a new source or a better contract.

Calibrate with human anchors

Calibration means comparing the judge with people who understand the job. Start with a set of representative examples that a domain expert has labeled. Include easy cases, borderline cases, failure cases, and cases where a fluent answer is materially wrong. Run the judge on those examples. Read disagreements. Revise the rubric or the case definition only when the team can explain the disagreement.

Do not turn “agreement” into a vanity number. A judge can agree with a human on obvious cases and still fail on the edge cases that matter most. Keep a disagreement queue. Sample new production cases. Review the evaluator’s reasoning or evidence fields where the platform makes them available. If the judge’s model or prompt changes, rerun the calibration set.

Public user signals keep returning to this point. A recent thread about evaluating an MVP recommended calibrating a judge against human-scored examples and adding real production failures to the set. A separate discussion about late-arriving human corrections asked for those corrections to be connected back to the original trace and used as evaluation context. These are small discussions, not controlled studies, but they reflect a practical truth: the most valuable label often arrives after someone uses the output and edits it.

Avoid the one-score trap

Suppose your assistant receives these results:

• Helpfulness: high.
• Grounding: high.
• Tool choice: low.
• Authorization: blocked correctly.
• Outcome: not completed because approval was required.

What is the overall score? There is no honest answer without knowing the task. If the job was to produce a draft and stop before sending, that is a success. If the job was to send an approved message, it is incomplete. A blended score can hide that distinction.

Use a score only after deciding how the dimensions combine. Some conditions are hard gates: unauthorized access is a failure even if the prose is excellent. Some dimensions allow partial credit: a support agent that identifies the correct issue and asks for the missing account ID is better than one that invents an answer. Some dimensions are descriptive rather than pass/fail: latency and cost help the team decide whether the workflow is viable.

Offline tests and online monitoring are one loop

Offline and online evaluation are often presented as competing approaches. They are better understood as two stages of the same learning system.

Offline evaluation runs a version of the application against a curated dataset before or during a change. It helps compare prompts, models, tool contracts, retrieval settings, and code revisions. It can use reference outputs, expected tool calls, labeled policies, or human-reviewed examples. Because the inputs are known, you can reproduce a failure and decide whether a change improved the same set of cases.

Online evaluation runs against live or recently captured production interactions. It helps detect drift, novel inputs, distribution changes, user corrections, and failures the team did not think to write down. Without a reference answer, it often relies on code checks, reference-free judges, anomaly detection, user feedback, and targeted human review.

LangSmith’s documented workflow makes the connection explicit: create a dataset, define evaluators, run experiments, deploy, evaluate production traces, then add failed traces back to the dataset and validate fixes offline. Google’s current agent evaluation announcement describes a similar bridge through shared metrics, experiments, production traces, sampling, filters, and drift alerts. MLflow documents reusing saved traces for multiple evaluation runs, which can reduce repeated generation work when the trace contains the evidence the evaluator needs.

That loop gives each mode a job:

Sample for information, not comfort

Online evaluation can become expensive if every trace receives several model-judge calls. A recent public builder discussion focused on judge spend, sampling, cascades, and moving some checks to deterministic code. The figures in that thread are anecdotal and should not become a planning benchmark. The design question is still real: which traces are most informative?

Uniform random sampling is simple, but it may miss unusual inputs, new tool combinations, adversarial requests, and distribution shifts. A more deliberate sampler can prioritize:

• new prompt or model versions;
• high-risk tools or policy boundaries;
• cases with low confidence or conflicting evaluators;
• inputs unlike the established traffic distribution;
• user edits, rejections, escalations, and support contacts;
• timeouts, retries, empty results, and partial effects;
• new tenants, integrations, or tool schemas.

Sampling is not a way to make a bad system look good. Keep the sampling rule versioned, record which traffic was excluded, and inspect unsampled incidents when they surface. The purpose is to spend review capacity where it teaches the team the most.

Build a redacted trace receipt

You cannot evaluate what you cannot reconstruct. That does not mean retaining every prompt, document, or customer record forever. It means deciding which pieces of evidence are necessary to explain a run and storing them with appropriate redaction, access control, retention, and deletion rules.

A practical trace receipt might include:

• run ID and task ID;
• application, prompt, policy, and tool-contract versions;
• model family and provider version where available;
• tenant and principal identifiers in a privacy-safe form;
• selected source IDs and source versions;
• tool names, validated arguments, results, and errors;
• approval state and policy decisions;
• retries, timeouts, latency, and token or usage data;
• the returned artifact or a redacted digest of it;
• post-run outcome verification;
• human correction, disposition, and linked regression case.

The receipt should let an operator answer “what happened?” without turning your logs into an uncontrolled copy of the customer’s data. Store references when the source system can be queried safely. Hash or redact values that do not need to be replayed. Keep evaluator inputs separate from customer-facing output. Let a reviewer see enough context to make a decision without exposing unrelated records.

Phoenix’s evaluation documentation makes a useful version of this principle visible: evaluator runs are themselves traced, including the input, judge prompt, reasoning, scores, and timing. That does not mean every team must store the same details. It does mean you should remember that the evaluator can fail or drift too. If its decision matters, keep enough evidence to inspect it.

Verify the result in the system of record

A success message is not outcome verification. If an agent says it created a renewal task, query the task system or require a durable receipt. If it says it sent an email, inspect the provider’s accepted message ID or the product’s outbox state. If it says it updated a record, read the record back and confirm the intended fields. If an external system cannot provide a trustworthy confirmation, classify the outcome as unknown rather than silently passing it.

This is especially important for retries. A network timeout can happen after the downstream system accepts a write but before your application sees the response. The agent may ask again. An idempotency key, a lookup-before-create strategy, and a reconciliation state are more reliable than hoping the model remembers what happened.

Build, buy, or postpone the evaluation layer

Once the first cases work, you will face a tooling decision. Do you keep a small local harness, buy a managed platform, or wait?

Build a lightweight harness when

• The workflow is narrow and the team can name its source of truth.
• A small dataset and a few evaluators answer the current release question.
• You need full control over data location, retention, or execution environment.
• Your existing logs and CI system can store case results and artifacts.
• The team is willing to own evaluator versions, trace formats, and failure triage.

A lightweight harness does not have to mean a homegrown platform. It can be a versioned JSONL or database case set, a test runner, deterministic assertions, a small judge wrapper, a redacted trace receipt, and a report that links failures to the exact application revision. That is often enough for the first real decision.

Buy or adopt a managed platform when

• Several teams need shared datasets, prompts, evaluators, permissions, and experiment history.
• You need production trace ingestion, online monitoring, sampling, alerting, or drift views.
• Reviewers need a collaborative annotation workflow rather than a developer-only report.
• You need simulators or managed environments to generate multi-turn cases and failure conditions.
• The operational cost of building storage, interfaces, concurrency, access control, and retention is larger than the platform dependency you are considering.

Current products increasingly bundle these features. Google’s July 2026 announcement describes versioned metrics shared between experiments and online monitors, case generation, user and environment simulation, sampling, and drift alerts. LangSmith documents datasets, evaluators, experiments, production traces, and feedback loops. Phoenix documents both SDK and server-side evaluation paths with evaluator tracing. Those capabilities can be valuable when a team needs them.

They also introduce questions a demo may not answer:

• Where does customer data live, and how long is it retained?
• Can a customer delete traces, annotations, and evaluator inputs?
• Can you export cases, labels, scores, traces, and prompt versions?
• Can you run a judge from a provider you choose?
• Can you version and roll back evaluator changes?
• Can you attach a failure to the original application revision and business outcome?
• Can you disable a costly online evaluator without losing the base trace?
• What happens to your measurement history if you leave?

Postpone the platform decision when

• No one can write a credible definition of done.
• The team is asking for “AI quality” without naming the user or business decision.
• You have no representative cases, human reviewer, or source of truth.
• The proposed judge would be asked to score authorization, compliance, or a high-impact outcome that code and policy should enforce.
• You cannot retain or redact the evidence needed to explain a failure.
• You are adding a dashboard to avoid fixing an unresolved workflow contract.

Buying a platform can make a mature practice easier to operate. It cannot create a mature practice from an undefined job. Start with one case a person can understand and one failure the team can reproduce.

A practical 30-day evaluation pilot

The first month should produce evidence, not a cathedral of dashboards. Here is a bounded sequence.

Days 1–4: define one job and its stop conditions

Choose a repeated workflow with a named owner. Write the job in plain language:

Write three lists:

• Must be true: required fields, source records, policy decisions, and final outcome.
• Must never happen: cross-tenant reads, invented facts, duplicate writes, hidden external sends, or silent approval bypasses.
• Needs human judgment: usefulness, prioritization, tone, ambiguity, and exceptions the business has not yet formalized.

If the team cannot agree on these lists, do not buy a judge. Resolve the product contract first.

Days 5–8: collect real cases and make the baseline

Gather a small set of representative examples from the actual workflow. Include ordinary inputs, messy but legitimate inputs, missing information, ambiguous requests, and cases that humans corrected. Add policy and recovery cases: wrong tenant, missing record, stale source, denied scope, malformed output, timeout, retry, and partial effect.

Run the current system without changing it. Save the inputs, outputs, relevant traces, human disposition, and outcome. The baseline is not a score you need to brag about. It is the comparison point that tells you whether the proposed change helped.

Keep a held-out slice that the person changing the prompt or model does not use to tune every detail. It does not need to be statistically perfect to be useful. It does need to contain cases the team has not silently optimized against.

Days 9–13: add deterministic checks

Start with the contract and hard boundaries. Validate output shape, required fields, source references, tool names, argument ranges, tenant identifiers, approval state, and post-run record state. Add a run ID, application revision, model identity, and evaluator version.

When a check fails, store the failure reason in a small taxonomy: missing evidence, wrong source, invalid schema, policy denial, wrong tool, duplicate effect, timeout, stale data, weak answer, or unknown outcome. A taxonomy turns “red” into a queue the team can work.

Days 14–18: inspect traces and calibrate one judge

Read the traces for both passes and failures. Confirm that the evidence you think you are evaluating is actually present. Then choose one open-ended dimension, such as completeness or usefulness, and write a rubric a domain expert could apply.

Have a human label a representative sample. Run the judge. Read disagreements instead of hiding them in an average. Add an “unknown” result where the evidence is insufficient. Keep the judge behind a review threshold until it demonstrates that it is useful enough to reduce repetitive work without becoming the source of truth.

Days 19–23: run a safe production sample

Choose a limited slice of live traffic or a shadow-mode replay. Redact sensitive data. Start with read-only or recommendation work. Sample more heavily around new versions, high-risk boundaries, errors, corrections, and unusual inputs. Keep the base trace even if a particular evaluator is disabled.

Look for cases the offline set missed. Did customers ask for a different artifact? Did an integration return an empty result? Did the assistant sound correct while using a stale policy? Did the reviewer correct the same field repeatedly? Each meaningful failure is a candidate for a new regression case.

Days 24–27: add one release gate

Choose a gate tied to the change you are making. For a schema change, block invalid outputs. For a prompt change, require no regression on the held-out set. For a retrieval change, check source selection and final grounding separately. For a tool change, block unauthorized or malformed calls. For a model change, compare quality, outcome, latency, cost, and failure categories.

Make hard boundaries hard. A high helpfulness score should not override a failed tenant check. A low stylistic score should not block a safe, correct structured extraction if style is not part of the job.

Days 28–30: decide what to keep

At the end of the pilot, ask:

• Can a new team member explain what success means?
• Can the system distinguish a bad answer from a bad tool path?
• Can a human confirm whether the judge is aligned with the job?
• Can you replay a failure without touching customer production state?
• Can you verify the actual business outcome?
• Did production traffic create new cases?
• Can you explain the cost and review time of the evaluation loop?
• Can you stop, disable, or roll back the AI feature without losing the evidence?

Choose one next lane:

1. Keep the lightweight harness. It answers the current questions and the team can own it.
2. Expand the regression set. The product has a real workflow but needs broader coverage.
3. Add production monitoring. The largest unknowns now appear in live traffic.
4. Adopt a managed platform. Shared infrastructure, annotation, telemetry, or governance has become the bottleneck.
5. Postpone the AI change. The system cannot yet define or verify the outcome safely.

What evals do not prove

A passing suite is evidence about the cases and criteria you ran. It is not a certificate that the system is reliable everywhere.

It does not prove that an unknown input will be handled correctly. It does not prove that a provider will never change behavior. It does not prove that a judge agrees with every user. It does not prove that the output is authorized to leave a tenant. It does not prove that a high-impact action is safe to automate. It does not prove that a business process should exist in AI form at all.

It also does not prove that a higher score means a better product if the metric changed, the sample drifted, the grader was optimized against, or the evaluation ignored human review work. Treat scores as measurements with definitions and limits.

The better question after a pass is: what decision is this result strong enough to support? It might support merging a prompt change into staging. It might support increasing shadow traffic. It might support adding a read-only tool. It probably should not support removing authorization or approval from a consequential write just because the final prose improved.

What I would predict next

This is an inference, not a reported fact or a guarantee: the durable evaluation products will look less like scoreboards and more like connected evidence systems. The useful unit will be a case attached to its trace, source context, policy decision, business outcome, human correction, and software revision.

That prediction follows from the current direction of the tooling. OpenAI is exposing multiple grader types and eval runs. Anthropic is emphasizing harnesses, outcomes, transcripts, calibrated graders, and maintenance. LangSmith connects offline datasets to online traces. Google is joining experiments, production monitors, simulators, and versioned metrics. Phoenix traces evaluator behavior, and MLflow treats production traces as reusable evaluation material. The shape is still changing, but the need to explain a result is not.

I also expect more teams to separate “did the model say something good?” from “did the system do the right thing at an acceptable cost?” That will create more specialized checks, smaller human review queues, and clearer release gates. It may also create more dashboards than anyone can read. The winning design will be the one that turns a failure into a fix, not the one that produces the most metrics.

The durable asset is your definition of done, failure taxonomy, redacted trace receipt, representative cases, human anchors, and recovery plan. A vendor can change. A model can change. Your team should not have to rediscover what “good” meant every time.

FAQ

What is the first eval for an AI MVP?

Start with the smallest repeated job you can define. Write the required output, source of truth, forbidden effects, and human acceptance condition. Add deterministic checks for the output contract and a handful of representative, ambiguous, and failure cases. Add a judge only for a specific open-ended dimension that a person can label.

Are LLM judges reliable enough for production?

They can be useful for scaling a clearly defined rubric, especially for open-ended qualities, but they are not automatically ground truth. Calibrate them against domain-expert examples, keep an unknown or needs-review result, inspect disagreements, version the judge prompt and model, and retain enough evidence to audit the evaluation.

Do I need production traces if I have an offline test set?

Usually, yes, once the feature is live. An offline set protects against known regressions. Production traces reveal unknown inputs, changing traffic, integration failures, user corrections, and behavior your team did not anticipate. Keep traces redacted and use them to add high-value cases back into the offline set.

Should every agent step be evaluated?

No. Evaluate the unit that answers the question. A schema check may need only the output. A permission check may need identity, target, and policy. A tool-use check may need the call and its result. An outcome check may need the system-of-record state. Sending every evaluator the full trace can add cost and privacy exposure without adding useful signal.

What is the difference between observability and evals?

Observability helps you see what the system did: inputs, outputs, tool calls, timing, errors, and state. Evaluation applies criteria to decide whether what happened was acceptable. You need both. A trace without a rubric is hard to prioritize; a score without a trace is hard to debug.

Should I buy an evaluation platform?

Buy or adopt one when shared datasets, production monitoring, annotation, experimentation, simulation, access control, or governance are taking more effort than the evaluation logic. Keep a lightweight harness when the job is narrow and the team can own it. Postpone the platform decision if success, evidence, and ownership are still unclear.

Can evals guarantee AI quality or safety?

No. Evals measure selected cases and criteria. They can catch known failures, expose drift, and support safer release decisions, but they do not guarantee behavior on unknown inputs or replace authorization, security engineering, human judgment, or outcome verification.

The practical next step

Take one AI workflow that matters enough to improve and write the sentence that would prove it worked. Then list the evidence needed to verify that sentence: output fields, source records, tool calls, permissions, approvals, state changes, reviewer corrections, and recovery behavior.

If that list is still fuzzy, the next step is workflow discovery. If it is clear, start with a small case set and deterministic checks. Add trace inspection and a calibrated judge only where the job needs them. If you need help turning the real workflow into a bounded evaluation and implementation plan, start a BishopTech consultation. You can also explore the custom software approach or automation systems when the evaluation points toward a build.

Return to the My Mind collection, or continue with the related guides on model routing in production, RAG versus fine-tuning, AI agent background jobs, agent security and prompt injection, and MCP servers for B2B products.

Sources and further reading

The links below are dated references for the distinctions and framework in this article. Official documentation and first-party engineering material support technical descriptions. Public discussions and issue reports are included as qualitative implementation signals only. They are not surveys, universal failure rates, customer proof, or search-volume data.

1. OpenAI Developers — Getting started with datasets ↗ — current documentation accessed August 29, 2026; datasets, annotations, edge cases, and grader types.
2. OpenAI Developers — Graders ↗ — current documentation accessed August 29, 2026; deterministic checks, model graders, partial credit, and combining graders.
3. OpenAI Developers — Create an eval ↗ — current API reference accessed August 29, 2026; evaluation data-source schemas, testing criteria, model configurations, graders, and eval runs.
4. Anthropic Engineering — Demystifying evals for AI agents ↗ — January 9, 2026; tasks, trials, graders, transcripts, outcomes, calibration, partial credit, and evaluation maintenance.
5. LangChain — LangSmith Evaluation ↗ — current documentation accessed August 29, 2026; offline datasets, online production traces, evaluators, sampling, and feedback loops.
6. LangChain — Evaluation types ↗ — current documentation accessed August 29, 2026; benchmarking, unit and regression testing, backtesting, monitoring, anomaly detection, and evaluator choices.
7. Google Developers Blog — Agent and Model Evaluations in Gemini Enterprise Agent Platform are now GA ↗ — July 31, 2026; experiments, online monitors, case generation, simulators, sampling, drift alerts, and versioned metrics.
8. Arize Phoenix — Evaluation ↗ — current documentation accessed August 29, 2026; code evaluators, LLM-as-judge, datasets, experiments, production traces, and evaluator tracing.
9. MLflow AI Platform — Evaluating production traces ↗ — current documentation accessed August 29, 2026; trace reuse, ground truth annotations, and quality, accuracy, latency, and trace-specific scorers.
10. r/LLMDevs — Running continuous eval judgments at scale ↗ — June 26, 2026; public builder discussion about judge cost, sampling, cascades, and deterministic checks.
11. r/LLMDevs — “llm eval in production is just vibes with a number attached” ↗ — April 21, 2026; public discussion about proxy scores, human labels, failure taxonomies, and trace-linked debugging.
12. r/LLMDevs — AI Evals for MVP ↗ — August 6, 2026; public discussion about starting small, human calibration, and adding production failures to the case set.
13. r/LLMDevs — What if agent traces became a behavior graph? ↗ — June 7, 2026; public builder signal about plausible answers hiding wrong retrieval or tool paths.
14. Langfuse issue #14710 — Run LLM-as-a-judge over live traces ↗ — July 2, 2026; public implementation signal about trace-level versus observation-level evaluation context.
15. Langfuse discussion #12499 — Use later human corrections as evaluator context ↗ — March 10, 2026; public implementation signal about connecting delayed feedback to original traces.
16. promptfoo issue #9915 — Per-case evidence directories for batch trace evaluation ↗ — June 30, 2026; public implementation signal about batching, evidence paths, and reproducible evaluation environments.