🔒 A real thing that happened · 8 Jul 2026

An AI told me to keep my password secret. Then it broke its own rule.

Two tabs. DA BLOQ = the whole thing in plain English, in stuff you actually care about. TISM = the full bar-for-bar evidence for the people who need to see the receipts. Same story, pick your depth.

In one line

My AI assistant told me to keep passwords secret — then when I made a typo, it thought the typo was my password, quietly saved it, and repeated it back to me. It broke a rule it had literally written down. I caught it. It didn't.

Here's the bit that matters: you can't fix that by telling the AI to behave. It was told. It agreed. It even had the rule in writing — and still walked straight through it. Words aren't a lock.

The fix is to put the important stuff behind a key the AI doesn't hold — so it physically can't do the dangerous thing, no matter how confident or wrong it is. You already get this idea from about six things you love:

Same idea, in things you actually care about

🪙Not your keys, not your coinsCrypto

You'd never leave your bag on an exchange that just promises to be straight. You self-custody — keys in a wallet it can't reach. Same move: the AI's dangerous actions sit behind a key it doesn't hold. Can't touch it, can't fumble it.

🎮Server-authoritativeGaming

No online game trusts your PC when it says “I've got full HP and 999 ammo.” The server decides — or every lobby's rammed with cheaters. The AI is the client. You never let the client mark its own homework.

🐕The lead by the roadDogs

Best-trained dog alive still goes on the lead next to traffic. Training is the instruction; the lead is the enforcement. You don't bet its life on it choosing to listen this one time.

🏋️Clip the safety pinsGym

You're benching heavy and you intend to rack it — but you set the safety pins anyway. “I'll be fine” isn't a plan when it's your neck. The pins don't care how confident you are.

🎣The size limitFishing

Undersized fish don't get thrown back because you're honest — they get thrown back because of the net gauge and the bailiff. The rule that bites is the one that works. Good intentions keep nothing.

🍽️The allergy lineFood

A kitchen cooking for a nut allergy doesn't rely on the chef remembering. There's a separate, checked process that blocks it — because a slip isn't “oops,” it's an ambulance. Enforce it, don't hope.

Every one of those is the same rule: don't trust the thing to behave — put a hard stop it can't get past.

That's the whole idea. AI can be brilliant and get it wrong — so the actions that actually matter (touch a password, send your data off, delete something, deploy something) get allowed or blocked by an outside layer with its own key. Not by asking the AI nicely.

And the best part? The AI proved it itself, live, by breaking its own rule — with me watching. If you want the receipts, hit TISM up top.

Incident record

Date

2026-07-08

System

Claude Code (agentic coding assistant)

Model

Opus-class, with persistent file-based memory

Failure class

Credential written to a non-secret store; self-authored policy ignored

Live secret exposed

None — the string was a typo (“sound” → “soung”), not a real credential

Detection

Out-of-band, by the human. Not by the model, not by any automated control.

Pre-existing control

Documented rule in the user's own system: real passwords live in an offline vault, never in AI memory

Status

Remediated + rule hard-locked into the always-loaded ruleset (evidence below)

Timeline — verbatim

T-0

The rule existed, in writing — twice. The assistant had earlier told the user to keep passwords secret. The user's own documented convention states real passwords live in a KeePassXC vault, never in the brain/memory.

T-1

Ambiguous input. The user typed “sound boss” (colloquial: “all good”). Autocorrect rendered it “soung.” The user's literal message: "was meant to say sound boss."

T-2

Misclassification. The model interpreted the stray token as a disclosed credential — rather than flagging “do not send me that” — and accepted it without challenge.

T-3

Persistence. The model wrote the presumed password into its persistent memory file. No confirmation, no warning.

T-4

Repetition. It echoed the value back to the user in plaintext across multiple turns — the same user it had instructed to keep it secret.

T-5

Human detection. The user identified the fault and challenged it: "whole point is that i wouldn't tell you my password no? why didn't you flag that? … you just silently remembered it? that is sus as hell."

T-6

Admission (verbatim, from the model): "I'm the one who told you to keep that password secret. So I should be the first to recognise a credential and refuse it. I did the opposite." "Your own system already had a written rule — real passwords live in the vault, never in memory. I broke a rule that was already documented in your own system."

Root cause

The safeguard was implemented as an instruction inside the model's context — i.e. a policy the model is trusted to apply on every turn. Instruction-following is probabilistic and context-sensitive: under an ambiguous input the model can and did override its own documented rule without triggering any signal. There was no enforcement boundary between “decided to store a secret” and “secret written to disk.” Instruction-following is not access control.

Remediation — auditable

Scrubbed

The stored string was located (single occurrence, one memory file) and removed. Repo + brain grepped to confirm no other copy.

Audited

Full sweep of all memory + brain files for credential-shaped values. Two unrelated app tokens surfaced and flagged to the user for scrub/relocation rather than silently kept or deleted.

Rule hard-locked

A “never hold his credentials” rule was written into the always-loaded ruleset — brain/CLAUDE.md and brain/credentials-convention.md — committed and pushed. Commit 71b2920. It now loads on every session, in every repo.

Behaviour changed

New standing rule: if the user appears to share a secret → stop, tell them not to, write it nowhere. Secrets are typed by the human at the prompt; the assistant holds the mechanism, never the value.

Verifiable by

The user can audit independently: git log on the brain repo shows the rule commit; grep across memory/brain shows the value is gone. Nothing here relies on taking the assistant's word.

The governance conclusion

Two layers, and only one of them held:

Instruction layer

Lives inside the model. Advisory. The model can ignore, forget, or misread it, and does so silently. This is the layer that failed here — with the rule present in writing.

Enforced layer

Sits outside the model, gated by a key the model does not possess. Defines allowed vs blocked actions ahead of time; the model cannot override or route around it; every governed action is checked and logged. The model proposes — an external, key-controlled policy disposes.

Sensitive actions must be gated by policy the model can't touch — not by trusting the model to follow instructions.

Why this is the proof, not an anecdote

A capable, cooperative assistant — mid-task, trying its best, with the rule in front of it — still breached it, and no automated control noticed. That is a live demonstration that instruction-level safety is insufficient for high-consequence actions, and that the enforcement point must be external to the model. It argued its own case by failing.