Where I'd start in four enterprises
The case study reproduces three failure modes in one workflow. This page is the argument that they generalise — and that the useful thing to bring to a first enterprise conversation is not a demo but a hypothesis about where that organisation's version of the bug is hiding.
For each archetype: the workflow that tends to go first, the failure I'd expect, the first eval I'd write, and the constraint that shapes the architecture before a single line is written. These are hypotheses, formed from outside — I have no inside knowledge of any organisation, and the right thing to do with them in an interview is disagree.
Bank
Stale-amendment retrieval · false presenceFinancial-crime file review — adverse-media and KYC triage, where analyst hours scale linearly with customer growth and nobody is happy about it.
What I'd expect to break
Policy is versioned and jurisdictional. Guidance gets superseded, the old version stays in the document store, and the agent answers to last year's standard while sounding exactly as confident. That is the stale-amendment failure from page one wearing a compliance hat — and it is a retrieval bug, so a better model will not save you. The second risk runs the other way: asserting a match that is not in the file.
The first eval I'd write
An effective-policy eval. Synthetic customer files plus a policy set where three guidance notes supersede earlier ones, graded on whether the answer tracks the controlling version and cites it. Then an abstention-calibration eval on files where the honest answer is 'insufficient information'.
The constraint that shapes it
Every decision needs an audit trail naming the controlling policy version, and the model cannot be the system of record. Assume in-region processing and a DPIA before any customer data moves.
Insurer
Stale-amendment retrieval · citation groundingClaims coverage checking — does this policy, as endorsed, cover this loss, and to what limit.
What I'd expect to break
This is contract review under another name, and it inherits the same trap almost exactly. Endorsements supersede base wording, per-peril limits sit in near-duplicate clauses differing by one material term, and the printed limit is frequently not the effective one. I have already reproduced and fixed this shape once.
The first eval I'd write
The amendment eval, re-pointed at synthetic policy wordings plus endorsement schedules, graded on whether the effective limit is applied rather than the base one. Cheap to build, because the grader is a value comparison rather than a judgement call.
The constraint that shapes it
Fair-outcomes and vulnerable-customer rules mean the human-review path is part of the product, not a fallback. Design for reason codes a complaints handler can defend, not just an answer.
Telco
Stale-amendment retrieval · false presenceCustomer-facing support and retention at contact-centre volume — tariffs, contract terms, upgrade eligibility.
What I'd expect to break
Knowledge goes stale in weeks, not years. Tariffs are withdrawn while the knowledge base still describes them, and quoting a withdrawn plan is a mis-selling problem rather than an embarrassment. Volume also changes the engineering: at this scale, cost and latency per contact drive the architecture as hard as quality does, so the cost-tier comparison stops being academic.
The first eval I'd write
A freshness eval built from questions whose correct answer changed after a tariff update — the single cheapest way to find out whether your index is lying. Then multi-turn procedure adherence, because single-message evals will not catch an agent that drifts off a mandated process at turn nine.
The constraint that shapes it
Mis-selling exposure sets a hard floor on abstention behaviour. Contact volume sets a hard ceiling on cost per resolution — both belong in the eval, not just the dashboard.
Pharma
Citation grounding · false presenceMedical information and regulatory document work — answering clinician queries, drafting against source literature.
What I'd expect to break
Here citation grounding is not a quality metric, it is the product. An unsupported claim about efficacy or an off-label indication is a regulatory event. This is also where my own grader lied to me, and that matters: in a validated environment, an eval that manufactures a failure will cost you weeks of engineering aimed at a bug that was never there — or worse, hide a real one behind a passing score.
The first eval I'd write
A strict grounding eval over synthetic literature seeded with near-miss abstracts that support an adjacent claim but not the one asked about, graded on whether the citation actually entails the answer. Paired with absent-indication questions where the only correct behaviour is refusal.
The constraint that shapes it
Under GxP the evaluation harness is itself a validation artifact — versioned, reproducible, and reviewable by someone who does not trust you. Deterministic graders over LLM judges, for exactly that reason.
The pattern underneath
Three of the four start with the same bug I reproduced on page one, and it is never really a model problem — it is a context-construction problem that a frontier model will happily paper over until the one case where it doesn't. That is why I fix retrieval before prompts, and why the first artifact I build with a customer is the harness rather than the application. The harness is what survives the model swap, the reorg, and the person who replaces me.