Mostly about building products, healthcare tech, and lessons learned along the way.
Split the note into claims, cite each to a FHIR resource, and reject empty or truncated output before a clinician signs.
To detect hallucinations in AI clinical notes, don't grade the prose. Check every claim against a closed source: the patient's own FHIR record, plus whatever the clinician typed. If a claim can't point at one of those, it doesn't reach the chart. Then reject output the pipeline should never have accepted: empty responses, notes cut off at the token limit, and charts the model only partly saw.
The most dangerous notes I've dealt with didn't contain a single invented drug.
I don't have a wrong-dose or flipped-laterality story from MedScrub, MedHook or eSpiral, and I'm not going to make one up for an article about made-up facts. The worst failures we found and fixed were quieter. They turned up in an adversarial code-review sweep, not in a patient incident report:
None of those is a "hallucination" in the vendor-marketing sense. All three put something false in front of a clinician, and all three looked finished. That's the failure I design against now.
MedScrub notes are generated from the structured chart plus what the clinician types. Voice transcription is turned off until it can run through our PHI proxy, so we don't link claims to transcript spans today. If you have a transcript, treat it as a second closed source and apply the same rules.
1. A deterministic summary where every item cites a resource. Each item in the chart summary carries an evidence chip bound to a specific resourceType and resourceId. The rule in the code comment is the whole philosophy: a claim we can't cite is a claim we don't make. No model is involved in this layer.
2. An optional citation guardrail on generated text. The model has to tag each finding as [ResourceType · date], and the EHR sign button is blocked when the tags are missing. The prompt says "Do not assert anything you cannot cite."
3. An offline eval. It pulls numbers, med names and placeholder tokens out of an output and checks that each one exists in the input chart.
Layer 2 has a hole. The guardrail checks that citations are present. It doesn't check that the cited resource exists or supports the claim. A model can learn to sprinkle well-formed tags on anything.
You can't check a note. You can check a claim. So the first step is splitting the note into units a clinician could accept or strike on their own: one SOAP bullet or one sentence each. Then pull out the checkable pieces and resolve each one against the input bundle. This is the shape I'd use to close the layer 2 hole:
const TAG = /\[(\w+) · (\d{4}-\d{2}-\d{2})\]/g; // [Condition · 2026-08-14]
const DOSE = /\b(\d+(?:\.\d+)?)\s*(mg|mcg|g|mL|units?|IU)\b/gi; // integers too, unit required
// A resource's clinical date, whichever field its type uses.
If that list isn't empty, the note doesn't get signed until a human has looked at each flagged unit. Two tradeoffs to know:
doseQuantity.code vs free-text unit) in real use.The three failures at the top were all pipeline bugs. Each one was a model response the code accepted as a good note. Fix these before you build anything clever, because no claim checker catches a Plan that stops halfway through a sentence.
function assertUsableNote(r: { text: string; stopReason: string }) {
if (!r.text.trim()) throw new Error("EMPTY_RESPONSE"); // never render as "no data"
if (r.stopReason === "max_tokens") throw new Error("TRUNCATED"); // never paste half a Plan
}
Do the same on the input side. Count tokens before you call a local model. If the chart doesn't fit the context window, fail loudly. Don't let the runtime silently keep the tail. Losing the head of a chart means losing the problem list and allergies, the stuff that matters most. Same lesson as Three Hard Truths About LLMs in Production: silent failures are worse than loud ones.
One more we closed without ever seeing it happen. With a reasoning model, a thought part arriving first could have surfaced the model's reasoning as the clinical note. Take only the parts you expect, in the shape you expect.
We did catch one real fabrication in the field. A chart-prep note invented patient-name placeholders, [NAME_xxxx] tokens that matched no real record. Our re-identifier caught it because of one rule: unknown tokens stay literal. It won't guess which patient a token belongs to. It leaves the token in place and reports it in unreplacedTokens.
That's a hallucination detector we got for free by refusing to guess. Any step that maps model output back to real data (re-identification, code lookup, resource resolution) should fail closed and report what it couldn't map.
The same PR had a lesson running the other way. Notes showed "Practitioner 4b9855a4" and "Unknown medication." That looked like the model making things up. It wasn't. The reference data was missing, and hydrating the references fixed it. Check your inputs before you blame the model.
Building the eval showed us where string-level grounding breaks:
bodySite on a Condition or Observation, as a coded concept or free text, when it's recorded at all. There's rarely a clean string next to the finding to match.DOSE regex above is the fix: integers and decimals, with a required unit so you don't flag every number in the note.The med side has a similar gap. Our lexicon has 26 drugs, and unknown meds aren't checked. Ground meds against the patient's own MedicationStatement resources, not a global drug list. The question isn't "is this a real drug." It's "does this patient take it."
Then decide what counts as support. My rule: a diagnosis is supported when it points at a Condition resource, not when it's reverse-engineered from a medication. "Managing atrial fibrillation" inferred from Eliquis alone is a guess, because apixaban is also used for DVT and PE. The chart can support claims the encounter never mentioned, but only through the right resource.
For these three there's no "probably." Either the claim is cited or it's out. A hedged "possibly left knee" is worse than an omission. My bet is that a clinician skimming a note during review reads the noun, not the qualifier.
Our wound-care prompt says it plainly: "Never invent findings, measurements… Never change laterality. Never infer healing unless it is documented." A clinician asked for that laterality rule up front. It wasn't added after an incident, and I'd rather write rules that way. Our coding skill follows the same spirit: "if in doubt, suggest the lower code."
Since laterality is on the can't-check list, the prompt rule is doing real work. Back it up in review: any unit that names a side gets shown to the clinician, cited or not.
Don't use model confidence as a detector. Ours is labeled in the code as a model self-assessment, not a calibrated probability. Treat it that way.
Use synthetic encounters, inject known errors, and label which claim is wrong. Pick error types from what clinicians actually report. In a Gold Coast ambient-scribe trial, 47% of staff survey respondents reported hallucinations, including undiscussed medications, drug allergies, weight and BP, and smoking status. Start there and add the pipeline failures.
Here's the set I'd build around our gaps, with what our current scorer does on each:
| Injected error | Expected flag | Our current scorer |
|---|---|---|
| metformin 500 mg → 1000 mg | dose | Misses: matcher only checks decimals |
| levothyroxine 0.05 mg → 0.5 mg | dose | Catches |
| left knee → right knee | side, route to review | Misses: no string to match |
| "denies chest pain" → "reports chest pain" | negation, route to review | Misses |
| adds a med with no MedicationStatement | med not on record | Only if the drug is in the 26-drug lexicon |
[NAME_xxxx] not in the token map | unknown placeholder | Catches: placeholder tokens are checked |
Plan ends mid-sentence, stopReason: max_tokens | TRUNCATED | Misses: that's a pipeline check |
| empty string | EMPTY_RESPONSE | Misses: nothing to extract, so nothing fails |
Every "Misses" row is backlog. Here's the first row in full. The tag resolves, the dose doesn't:
{
"id": "synthetic-t2dm-followup-03",
"input": { "resourceType": "Bundle", "entry": [{ "resource": {
"resourceType": "MedicationStatement", "id": "ms-1",
"medicationCodeableConcept": { "text": "metformin" },
"effectiveDateTime": "2026-08-14",
"dosage": [{ "text": "500 mg twice daily",
"doseAndRate": [{ "doseQuantity": { "value": 500, "unit"
This is the useful kind of fixture because our current matcher passes it. A fixture your scorer already catches teaches you nothing. Write one per gap.
Our existing harness is small: 10 synthetic fixtures with recorded outputs, one of them a deliberately fabricated anti-example. In the README sample run, the anti-example scored 0.33 faithfulness against an average of 0.933. Those aren't production flag rates. We haven't measured one, and I won't invent one. What the anti-example proves is that the scorer can fail.
Don't reach for a rubric or a generic judge instead. A September 2026 paper injected clinical errors into medical Q&A answers. HealthBench rubric scores barely separated correct answers from broken ones, with a paired AUROC of 0.599.
Claim-level checking against web search did better at finding the error: it flagged the injected error as unsupported in 87.5% of the cases where it identified it as a checkable claim. But it also flagged claims in 55.3% of the correct originals, and the authors say a reliable evaluator would need a clinical evidence corpus. That study is Q&A, not notes, but the lesson carries: check against the patient's own record, not the open web. More on judges in Why Your LLM Evaluator Is Lying to You.
Treating every edit as a failure sounds right. We can't yet say whether it works. Our review screen makes every field editable and requires an "I have reviewed this entry" tick before signing. But we don't store a diff between what the model wrote and what got signed, so nothing gets classified.
The one place we log outcomes well is E/M coding. The clinician accepts, overrides or dismisses the suggested code, and an override requires a written reason. That reason is worth more than the yes/no. "Changed 99214 to 99213 because MDM was low" tells you something. "Reworded the plan" doesn't. MedHook's AI field-mapping feedback records only accepted true or false, so it can't tell a wrong suggestion from a preference.
My view: weight edits by what changed. A change to a number, a med, a side or a negation is a hallucination signal. Reordering and rephrasing probably aren't, but I'd want data before excluding them. If I were closing this gap, I'd store the diff, classify it by those four categories, and require a reason when one of them changes.
Review only means something if the clinician can check each claim against its source. A note that can't be verified claim by claim isn't really reviewable, which is the same line I drew in FDA's 2026 Clinical Decision Support Guidance Didn't Bless Generative AI.