All thoughts

Generator vs. Evaluator: An AI SDLC

Separate the generator from the evaluator. Gate releases on evals. Ship quality instead of velocity theater.

How it works

Generator, evaluator, then three gatesFive blocks joined by arrows. Only the evaluator is gold.1. Generator: produces output. 2. Evaluator: a separate system judges (highlighted). 3. Local: green? open the MR. 4. CI: green? merge approved. 5. Cloud GO: separate gate before deploy. 01Generatorproduces output02Evaluatora separatesystem judges03Localgreen? open theMR04CIgreen? mergeapproved05Cloud GOseparate gatebefore deploy
Select a step to see what it does.
The generator never grades itself. The evaluator is the gold step, and every gate after it runs on its results.

The pattern

Generator: The system that produces output. Claude, GPT, Llama, whatever. It generates.

Evaluator: The system that judges quality. A separate model, a separate prompt, a separate function. It evaluates.

These must never be the same system. If your generator self-evaluates, you're measuring nothing.

How we do it

On my team, no AI-assisted code merges without local-green evals. No Cloud deploy without separate eval GO.

The eval runs before the merge request. The eval runs before the deploy. The eval is the gate.

  • Local: Developer runs evals locally. Green? Open MR. Red? Fix first.
  • CI: Evals run in CI. Green? Merge approved. Red? Back to developer.
  • Cloud: Separate Cloud GO after evals clear. Production gate is not CI gate.

The eval stack

We use:

  • Braintrust for eval orchestration
  • Mastra Evals for framework-level eval composition
  • Langfuse for observability and trace-based regression detection

No release ships without evidence. Evidence comes from evals. Evals come from separated systems.

My take

The most critical architectural decision in enterprise AI is how you separate the generator from the evaluator.

When the generator and evaluator are the same, you get:

  • No quality gate
  • No regression detection
  • No release confidence
  • Velocity theater instead of shipping quality

When they're separated, you get:

  • Eval-gated releases
  • Quality metrics that mean something
  • Confidence before deploy
  • Evidence-based merge decisions

The alternative is velocity theater: ship fast, hope for the best, fix in prod.

We don't do that. We separate the generator from the evaluator, gate releases on evals, and ship quality instead of velocity theater.

That's the AI SDLC.

Try this

  1. Before your next AI-assisted merge, run the evals locally. Green? Open the MR. Red? Fix first.
  2. Check what grades your AI's output. If it's the same system that produced it, split them.