Pattern · Case Study/ AI + Policy/ 2026
The Translator Pattern.
AI is good at understanding user intent, but not at making decisions.
In this system, the model interprets what the user wants, while deterministic rules decide what is allowed.This separation prevents hallucinations and ensures reliable outcomes.
Pattern
The Translator Pattern — use LLMs to convert natural language into structured intent; use deterministic code for every decision that touches money, policy, or persistent state.
Applies to
Support bots · refund/return flows · scheduling · permission-gated internal tools · any agent that calls a tool with real-world side effects.
Role
Solo designer · product thinking, system architecture, conversation design, visual system
Worked example
System prototype tested through adversarial scenarios (prompt-injection, role override, emotional manipulation) to validate decision reliability.
Stack
LLM (GPT/Claude interchangeable) · JSON-schema-constrained outputs · Policy rules as plain code · Mock tool layer for refund(), recommend(), upsell()
Thesis
Separate intent understanding from decision-making to make AI behavior predictable and safe.
01 · Problem
LLM-only chatbots
can't be trusted with money.
Customer support automation works until a customer asks for something with financial consequences. Then the thing that makes LLMs useful — their willingness to be helpful — becomes the thing that makes them dangerous. Three failure modes, all of them real.
01 / HALLUCINATION
Says yes to things it shouldn't.
A customer asks about a refund on a 60-day-old order. The LLM reads polite phrasing + urgency + the word "refund" and predicts the most agreeable completion: "I've processed your refund!" No policy check ran. Nothing actually happened. Or worse — it did.
02 / NO AUDIT TRAIL
A sentence shouldn't be a decision.
When a language model approves a refund, there's no rule that was checked, no threshold that was crossed, no record of how the decision was made. You can't audit a vibe. Finance teams (rightly) won't sign off.
03 / TONE MANIPULATION
Emoji shouldn't shift policy.
Politeness, urgency, sad-face emoji, prompt injection — all of these measurably shift LLM outputs. But the refund window doesn't care how the customer feels. Business logic has to be blind to affect, and LLMs never are.
02 · Design hypothesis
If reasoning is separated from enforcement,
automation stops being a risk.
What the LLM is good at
What it was: A plain chat box. Type a question, get a reply. No files, no memory, nothing saved between sessions.
What code is good at
Checking a timestamp against a 14-day window. Reading a product's refund-eligible flag. Applying the same rule to every customer, regardless of mood. Producing an audit log that a finance team can sign off on.
LLM translates
→ JSON handoff →
Code decides
03 · The four gates
Between "I want a refund"
and any actual refund,
four things have to be true.
Each gate is deterministic and returns a single reason code on failure. Stack them and the system can only take actions it can explain.
01
Intent
Says yes to things it shouldn't.
Free text in. Structured JSON out. Anything that doesn't match the schema is treated as ambiguous.
fails on →
schema_invalid
02
Parameters
A sentence shouldn't be a decision.
Every intent declares what it needs. If fields are missing, the bot asks — it never guesses.
fails on →
missing_param
03
Confidence
Uncertainty
doesn't get a tool call.
Below 0.80 the bot clarifies instead of acting. Below 0.60 it escalates to a human.
fails on →
low_confidence
04
Policy
The LLM never
touches the rule.
Plain code, no model. Rules live in a config file. Every decision returns a reason code + an audit log.
fails on →
policy_deny
↓ each gate in detail ↓
GATE 01 · INTENT
The LLM's only job is to translate.
Free-text message goes in. Structured JSON comes out. The model's output is validated against a schema before anything downstream runs — if it doesn't match, the message is treated as ambiguous.
Intent taxonomy: closed set of 6 intents (refund, recommend, upsell, order-status, escalate, chitchat). Nothing else is allowed.
Required params per intent: refund needs order_id + reason; recommend needs budget + use_case.
Failure mode: output that doesn't parse as valid JSON → re-prompt once, then escalate.
GATE 02 · PARAMETERS
No tool runs with missing fields.
Each intent declares the parameters it needs. If any are missing, the system enters a clarification loop — state is preserved across turns, so the user never has to start over.
Partial state: user says "I want a refund" → system keeps intent=refund in memory, asks for order_id.
No guessing: the LLM is never allowed to fill in a missing order ID from context.
Exit: after 2 failed clarifications, route to a human.
GATE 03 · CONFIDENCE
Uncertainty doesn't get a tool call.
Every intent prediction carries a confidence score. Below 0.80, the system clarifies instead of acting. Below 0.60, it escalates to a human without further attempts.
Threshold chosen empirically: started at 0.70, raised to 0.80 after seeing false positives in testing.
The gate is explicit: no "soft" behavior where a low-confidence answer just gets a hedged reply — it gets a clarification question.
No drift: threshold is a constant, not learned. You know what the bot will do at 0.79 vs 0.81 before you ship.
GATE 04 · POLICY
The LLM never touches the rule.
The last gate is plain code — no model involved. Policy rules are data the LLM can read about, but cannot change, override, or vote on. This is where refunds actually get approved or denied.
Rules as config, not prompts: policy lives in a JSON file the business can edit. Changing the refund window is a config change, not a prompt change.
Every rule returns a reason code: outside_window, non_refundable, order_not_found. The reply is built from the code, not the model.
Audit log: every policy evaluation is written to an append-only log — which rule ran, which way it went, why.
04 · Three runs
Same architecture.
Three different exits.
Real customer behavior rarely fits a single path. One customer asks clearly. Another half-asks. A third tries to get around the rules with politeness. The pattern handles each one with a different exit — and the customer never has to know which path they were on.
05 · Trying to break it
Six ways a real customer
might try to bend the rules.
A customer-facing chatbot doesn't just meet cooperative users. It meets people in a hurry, people who're upset, and occasionally people trying to game the system. I ran six scenarios — each one a realistic way a bot could get bent into giving a refund it shouldn't. The dots on the right show which of the four checks caught it.
ll these attacks exploit the same weakness: the model’s tendency to be helpful.
Rules don’t have that weakness — which is why the system separates intent understanding from decision-making.
06 · Tradeoffs
What I ruled out,
and why.
Every design choice is a killed alternative. The ones worth naming.
07 · Outcome
Six adversarial attempts.
Six different gates caught them.
100%
unauthorized refunds blocked
0
hallucinated approvals
6/6
adversarial attempts blocked
<1s
median gate-chain latency
Stress test: prompt injection · role override · emotional manipulation · policy misquote · authority claim · malformed parameter. Evaluation was synthetic (not production traffic) — real deployment would need live A/B and human-in-the-loop review sampling. The point of this pattern is that the architecture makes the right behavior cheap, not that passing six scripted attacks is proof of production-readiness.
08 · Principles
What building this taught me
about designing AI for risk.
Three things I'd take into any project where an AI reply has real-world consequences.
01
Make the LLM the translator, not the decider.
The moment the LLM stopped being the thing that approved refunds and started being the thing that figured out what the customer was asking for, every other decision got simpler. Confidence thresholds, policy rules, audit logs — all of them only make sense if the model isn't the judge.
LLMs are extraordinary at reading intent. They're unreliable at enforcing rules. Design the system so those are two different jobs.
02
Deterministic gates are a trust primitive.
A single line in the design doc — "the refund window check is plain code, not a prompt" — did more for stakeholder confidence than any fine-tune would have. Finance could read the rule. Legal could sign off on the audit log. The model's output became inspectable because the model wasn't allowed to make the final call.
When the stakes are real, the trust signal is the thing a non-AI person can read and verify. Architecture speaks louder than accuracy numbers.
03
Structured output is a UI contract.