Evaluating LLM Applications

LLM output is nondeterministic and prompt/model changes can improve some cases while silently regressing others — the same risk as any code change, except traditional unit tests (exact-match assertions) rarely apply to open-ended natural-language output. The fix isn't abandoning testing; it's building an evaluation harness suited to the problem: a fixed dataset, a way to score output against it, and a way to run that automatically on every change.

# a golden set entry
{
  "input": "What's our refund policy for items over 90 days old?",
  "expected_facts": ["90-day window", "store credit only after that window"],
  "must_not_contain": ["full refund"]   # a known-wrong claim to catch
}

For criteria too fuzzy for exact string matching (tone, faithfulness, "does this actually answer the question"), a second LLM call scores the output against a rubric. This scales far better than human review, but the judge model has its own biases — a documented one is a tendency to favor longer or more confidently-worded answers regardless of correctness — so judge scores should be periodically spot-checked against human judgment, not trusted blindly forever.

judge_prompt = (
    f"Question: {input}\n"
    f"Answer: {model_output}\n"
    f"Reference facts that must be present: {expected_facts}\n\n"
    "Score 1-5: does the answer contain all reference facts and "
    "avoid contradicting them? Respond with only the number."
)

Run the golden set (with both exact-match/rule-based checks where possible, and LLM-as-judge where not) automatically whenever the prompt, retrieval logic, or model version changes, and compare the score to the previous baseline — the same principle as a CI test suite, just with fuzzier assertions than assert x == y. This is what catches "we upgraded to a newer model and it got worse on the exact cases that mattered" before users do.