Back to blog
• 5 min read • AI Engineering

Constraining an LLM's output space beats prompting it to be careful

The first instinct when an LLM output is wrong in a way that matters is to write a better prompt. Add a line that says “only suggest times that are actually available.” Add another that says “double-check the calendar before answering.” This works about as well as telling someone to be more careful: it lowers the error rate a little and does nothing to the failure mode itself, because the model can still, mechanically, produce any string of text you didn’t rule out.

The more durable fix is upstream of the prompt: don’t let the model generate the risky part of the output at all. Give it a small, real, closed set of options and have it pick from that set, instead of giving it a blank field and asking it to fill it in responsibly.

What “generate” versus “pick” looks like in practice

A scheduling assistant asked to suggest a new meeting time will happily produce “how about 2pm Thursday” whether or not 2pm Thursday is open. The sentence is equally fluent either way. Nothing about how confident or well-formatted the answer is tells you whether it came from checking a calendar or from pattern-matching what a scheduling suggestion sounds like.

Here’s the same task built the other way, from suggestAlternativeTimes() in a scheduling app’s AI layer (src/lib/ai/nlp-parser.ts). The available slots are computed first, by ordinary application code, not the model:

const formattedSlots = availableSlots.map((slot, index) =>
  `Option ${index + 1}: ${slot.start.toISOString()} - ${slot.end.toISOString()}`
);

The model is shown this numbered list and asked to rank the best three, but the response is never trusted as a time value:

const lines = assistantMessage.split('\n')
  .map(line => line.trim())
  .filter(line => line.length > 0)
  .filter(line => /^\d+\./.test(line));

const selectedIndices: number[] = [];
for (const line of lines) {
  const match = line.match(/^(\d+)\./);
  if (match) {
    const index = parseInt(match[1]) - 1;
    if (index >= 0 && index < availableSlots.length) {
      selectedIndices.push(index);
    }
  }
}

return selectedIndices.map(index => {
  const slot = availableSlots[index];
  return slot.start.toISOString();
});

The regex extracts a line number, not a date. That number becomes an array index into availableSlots, and the bounds check (index >= 0 && index < availableSlots.length) means a malformed or out-of-range line is silently dropped rather than treated as a real slot. Whatever time the function actually returns is availableSlots[index].start, a value that came from checking the calendar, not from the model’s text generation. The model’s entire freedom is choosing which of the pre-verified options to surface and in what order. It cannot invent a fourth one, because there is no code path that turns model-generated text into a timestamp.

Why this is a general pattern, not a scheduling-specific trick

This is the same idea behind structured decoding, output enums, and function-calling schemas with real parameter types: instead of asking the model to produce free text and then hoping a validator downstream catches the bad cases, you shrink the space of things it’s structurally able to say until every possible output is one you already know how to handle. A regex that filters bad lines is defense in depth; the actual guarantee comes from the fact that the only way a timestamp reaches the caller is by being looked up from a real array, never parsed out of the model’s prose.

The distinction matters because of what a wrong answer costs. A hallucinated fact in an ordinary chat reply is something a user can shrug off or double-check. A hallucinated time slot in a scheduling assistant is something a user is likely to act on directly, by showing up to a meeting the other party never agreed to, or accepting a reschedule that quietly conflicts with something the model never actually checked. The more consequential the action downstream of the output, the less acceptable it is to rely on the model behaving well, and the more it’s worth spending engineering effort narrowing what “behaving badly” is even capable of looking like.

The tell for when you still have this problem

Not every part of a system gets this treatment automatically, and it’s worth knowing what the weaker version looks like so you can spot it elsewhere. The same scheduling app’s chat assistant handles booking creation and cancellation differently: it asks the model to emit an [ACTION: {...}] tag inside its normal text reply, then finds and parses that tag with a regex:

const ACTION_RE = /\[ACTION:\s*\{[\s\S]*?\}\s*\]/g;
const actionMatches = assistantMessage.match(ACTION_RE);

This is structurally the same risk category as a free-text time slot, just with a JSON object standing in for a timestamp: the model is trusted to format something correctly inside prose, and downstream code has to go find it. It’s less immediately dangerous, because a malformed tag throws in JSON.parse and fails loudly instead of silently producing a wrong booking, and the action still passes through the same conflict check every other booking path uses. But the underlying shape of the risk is the same: a blank canvas where a small, pre-validated set of choices would do. The fix, when it comes, looks like suggestAlternativeTimes() already does: real tool definitions with typed parameters, called through structured function-calling, instead of a tag pattern-matched out of free text.

The general rule: if a wrong output would cost something real, don’t ask the model to be careful with a blank canvas. Take the canvas away.

Back to top