2026-04-28 · Note
Capability leak: the AI failure mode nobody's testing for
The McDonald's Python script wasn't a security breach. It's a product problem — and a lot of AI features are one prompt away from it.
A while ago, a screenshot went viral: someone asked the McDonald's support bot — “Grimace” — to write a Python function that reverses a linked list. It wrote one. Correctly. Then it asked if they wanted nuggets with that.
Everyone found it funny. I found it telling.
This isn't a security breach in the usual sense. It's what I'd call a capability leak. The engineering team dressed a general-purpose model in a McDonald's uniform, but they only restricted what it should do — not what it could do. When a user pushed against the persona, the persona was all they had. There was nothing else in between.
In regulated domains — insurance, payments, fraud — that isn't a funny meme. It's a trust problem waiting to happen. If your onboarding bot can be talked into medical advice or tax advice by a curious user, you don't have a product. You have someone else's playground with your logo on it.
Four practices keep coming up, none of them clever. Which is probably why teams skip them.
An intent gatekeeper, not a persona.
Before a user's prompt touches the reasoning model, run it through a cheap, fast classifier. If the intent doesn't belong to the domain you're serving, the conversation stops there. A persona is a costume; a gatekeeper is a door.
A human-annotated golden set, treated as a contract.
A hundred or so real conversations, annotated by hand. If a model update drops accuracy against that set, it doesn't ship. Tedious to build. The only thing that catches slow drift.
Graceful degradation, not full accuracy.
Chasing 100% is a trap; it hides the real question — what happens when the model isn't confident. Confidence thresholding plus human handoff beats overreach every time.
Observability over QA.
You can't QA an infinite prompt space. You can watch it. If the bot starts writing Python or dispensing legal advice, the alert should scream before a screenshot does.
The interesting part isn't any single practice. It's that the engineering wrapper isthe product. Without it, you're paying for someone else's model to embarrass you in public.