Most alert fatigue from LLM clinical decision support is self-inflicted. A model can write a plausible card for almost any chart, so the hook's default answer should be no card at all.
I run CDS Hooks services in production for MedHook and eSpiral. We run on ICD-10 and HCC codes and use free text sparingly. Patient-view was the heavier hook for us. My rules for an LLM behind a hook are simple. Do the expensive work before the hook fires. Never repeat what the clinician was already told. Refuse to show anything the model can't tie to a coded FHIR resource.
For the general build pain, see CDS Hooks: What Nobody Tells You Before You Build. This piece covers what changes when a model sits behind the hook.
Why LLM clinical decision support makes alert fatigue worse
Volume is cheap for a model and expensive for a clinician. That's the whole problem.
The research blames volume and repeats, not clinicians slowly tuning out. In Ancker et al. (2017), acceptance dropped 30% for each additional reminder per encounter. It dropped 10% for every 5-point rise in the share of alerts that were within-patient repeats. The same study did not support the desensitization hypothesis: acceptance of newly deployed alerts didn't fall over time.
So spend your effort inside the encounter. Fewer cards, and no repeats. Everything below serves that.
Budget 500 ms, and keep the model out of it
The spec is blunt: "CDS Services SHOULD respond quickly (on the order of 500 ms.)" Treat that as the budget. A card should not depend on how fast a model happens to answer on a given afternoon.
Patient-view fires whenever the user views a patient's information. That's why it was the heavy one for us. Every chart open is a request, and nothing in that path should wait on token generation. What kept it manageable was a precheck that filters on insurance plan and carrier before anything expensive runs.
Split the work three ways:
- Inside the hook: lookups and deterministic filters. Our plan and carrier precheck lives here. It's cheap, it's predictable, and it doesn't need a model.
- Precomputed: the model's output, generated before the chart opens. Trigger it on events you already receive: an encounter-start event from ADT, the day's scheduled-visit list, or a new Condition or claim landing for the patient.
- Per-encounter cache: the result for this patient and this encounter, keyed on a hash of the inputs. When a new Condition shows up, the hash changes and the entry is dead.
On a cache miss, return an empty cards array. Don't fall back to a live model call. A missing suggestion costs a clinician nothing. A chart that hangs while a model thinks costs you their trust in every card after it.
The spec doesn't settle staleness for you. Its STU note asks implementers for feedback on balancing performance against the patient-safety risk of stale guidance. My answer: key the cache on input data, not a timer.
Filter in prefetch before the model ever runs
The cheapest card is the one you never generate. If a suggestion can't apply to a patient's coverage, don't spend a model call finding that out. That's the job of the plan and carrier precheck.
Prefetch gets you the data without round trips. The CDS Client sends the FHIR resources with the request:
"prefetch": {
"patient": "Patient/{{context.patientId}}",
"conditions": "Condition?patient={{context.patientId}}&clinical-status=active",
"coverage": "Coverage?patient={{context.patientId}}&status=active"
}
Two spec rules will bite you:
Deduplicate per encounter, not per hook call
If I had one warning from running these hooks in production, it's this. Watch for practice advisories that surface duplicate information. Duplicate information is what upsets clinicians. If the EHR already fires an advisory on a topic and your LLM card says the same thing in different words, you've doubled the noise. Now the noise has "AI" attached, and clinicians remember that.
The obvious dedup key is wrong. hookInstance is a globally unique UUID per invocation. A clinician who prescribes two drugs in a row triggers two of them. Dedup on it and you dedup nothing.
Key on what the clinician sees as "the same card": patient, encounter, topic and the resource the card cites. Then filter against the site's own advisories. The hook request doesn't necessarily tell you which advisories the EHR fires, so get the list of active advisory topics from the site and suppress anything they already cover.
Hook calls are stateless HTTP requests, so the "already shown" set can't live in process memory. Put it in a per-encounter store that expires at encounter close:
def filter_cards(r, ctx, cards, site_advisory_topics, ttl_s):
key = f"cds:shown:{ctx['patientId']}:{ctx.get('encounterId')}"
out = []
for c in cards:
topic = c["source"]["topic"]["code"]
if topic in site_advisory_topics:
continue
if r.sadd(key, f"{topic}|c
r is a Redis client. SADD returns 0 when the member already exists, so the check and the write are one call. nx=True sets the TTL only when the key has none, so later hook calls don't keep pushing expiry out. If you know the encounter's close time, EXPIREAT on that timestamp is cleaner.
One tradeoff to accept with your eyes open: this marks a card "shown" when you return it, not when the client renders it. If the client drops it, you won't send it again this encounter. I'll take that. A missed repeat costs less than a duplicate.
source.topic is an optional Coding in the spec, and no standard topic set exists. Define your own and make it mandatory in your service. Strip internal fields like cites before you return the card.
Ask for a quiet card
Your service doesn't get to decide whether a card interrupts. card.indicator is required (info, warning, critical), but the spec doesn't require any value to render as a modal. The client decides. What you control is what you ask for.
One study shows what quiet can do. Fallon et al. (2024) replaced an interruptive COVID-precautions alert with passive CDS. Weekly alerts fell from 8,206 to 1,449, about an 80% cut. The share of symptomatic admitted patients with precautions ordered rose from 23% to 61%.
My defaults for LLM-generated cards:
indicator: info. Save warning and critical for deterministic rules, not model output.- The summary is required and under 140 characters. Write one finding, not a paragraph squeezed into a sentence.
- Put the reasoning in
detail. For non-urgent cards, the client may tuck it behind "view more details." That's fine.
- If acting on the card takes real work, use a
smart link with appContext instead of a pile of suggestions. The clinician opens it when they choose to.
- The model never proposes
systemActions. They're not shown to the user, and the client may auto-apply them. Unreviewed model output changing the chart is a hard no. It's the same reasoning behind keeping the LLM out of order-sign.
No citation, no card
At MedHook and eSpiral we run on ICD-10 and HCC codes. Free text is available for other data, but we use it sparingly. Coded input makes the output checkable. The rule I'd hold any LLM card to: every finding has to point at a resource in the prefetch data that carries a code. If it can't, drop the finding. Free text can add color in detail. It shouldn't be the only thing a card stands on.
The spec gives you a clean way to say nothing. A 200 with an empty cards array means "no guidance".
ICD10 = "http://hl7.org/fhir/sid/icd-10-cm"
def grounded(finding, prefetch_index):
res = prefetch_index.get(finding["cites"])
if not res:
return False
codes = res.get("code", {}).get("coding", [])
return any(c.get("system") == ICD10 for c in codes)
cards = [build_card(f) for f in findings if grounded(f, idx)
Run this check in code after the model, not as an instruction in the prompt. Prompts are suggestions. Code is a gate. card.source is required and should be the primary source of guidance, so the cited resource also gives you an honest source. A clinician can review the basis for the card themselves. The FDA's 2026 CDS guidance is silent on generative AI, so that part is on you.
Make overrides come back as data
The feedback API can turn an override into a signal. It has gaps.
First, no card.uuid, no feedback. The uuid is optional in the spec, but the feedback endpoint can't refer to a card without it. Always set it.
Second, return your own overrideReasons. The spec has no standard set, so pick reasons that match how your cards fail. Put "Duplicate of another alert" first. Add "already addressed" and "doesn't apply to this patient's plan." Those reasons are labeled failures. Feed them into your eval set like any hand-labeled data. Three-step product evals covers the loop, and the honest guide to LLM evals covers how to score them.
Third, know what the data hides:
- Ignored cards are never reported. The client stays silent if the user never touches a card, "in part, because it may later be acted upon." No feedback is not rejection.
- Overrides can arrive with no reason. A user who views a card and dismisses it counts as an override.
- Duplicates happen. The same card can be reported more than once. Make ingestion idempotent on uuid, outcome and timestamp.
Don't read the raw override rate as a grade. A 2020 systematic review found medication alert override rates from 46.2% to 96.2%. It also found that 29.4% to 100% of those overrides were clinically appropriate. Sometimes an override is the system working.
What to do instead: chart "duplicate of another alert" overrides weekly, broken out by source.topic. If duplicate is the top override reason for a topic, that topic goes on the suppression list that week. It's a site advisory you missed, and every day it stays live teaches clinicians to skip your cards.
