AI draft replies to patient messages are worth building only if you do three things the drafting model can't do for you:
- A router decides which messages get a draft at all.
- Every draft is grounded in the patient's FHIR record.
- You measure how often clinicians edit or throw away drafts, not how many seconds you saved.
I build clinician-facing AI in healthcare. The published studies focus mostly on the draft itself, though some, like UC San Diego's, gated which messages got drafts. They say little about what happens before the draft or after it. Before means deciding whether a message should get one. After means deciding how the draft reaches a human and how you learn from what they changed. That's where I'd spend the engineering budget.
Why time saved is the wrong metric for AI inbox drafts
The time numbers are weak, and they point in different directions.
- UC San Diego. Tai-Seale et al. found AI drafts were associated with a 21.8% increase in read time. Reply time changed -5.9%, which wasn't significant. Replies got 17.9% longer.
- Stanford. Garcia et al. saw adoption and better burden and burnout scores, but "no improvement in time."
- NYU. The best result I found is Mandal et al., npj Digit Med 2025: 331 seconds per message with a draft versus 355 without. That's 24 seconds.
I've only seen the NYU figure, and several others below, in a secondary summary of the primary papers. I name each paper so you can check the original before you quote it.
There's also a billing catch. E-visit codes CPT 99421-99423 are time-based with a 5-minute minimum. Faster drafting can push a billable message below that floor. That comes from the summary's account of Mayo Clinic's billing data (Dunlay et al., Ann Intern Med 2024).
The bigger problem is that seconds tell you nothing about whether the draft was right. A clinician who sends a wrong draft quickly looks great on a time dashboard.
The volume problem is real, though. Epic Cosmos data shows patient-authored portal messages went from 0.99 to 2.50 per patient per year between 2020 and 2025, a 153% jump (Long et al., JAMA 2026). What helps with that is triage and grounding.
Route patient messages before you draft anything
The first decision is whether a message should get an AI draft at all.
Tai-Seale's study already had a gate built in. Only four request types were eligible for drafts: refills, results, paperwork and general questions. NYU went the other way. According to the same summary, it generated drafts for every message, including the roughly 80% that never got a reply. That's compute spent on drafts nobody reads, and each one sits in a clinician's view as noise.
Here's the routing policy I'd start with:
type Route = "urgent" | "results" | "refill" | "scheduling" | "other";
const POLICY: Record<Route, { draft: "none" | "template" | "llm" }> = {
urgent: { draft: "none" },
results: { draft: "llm" },
refill: { draft: "llm" },
scheduling: { draft: "template" },
other draft
Five rules make this hold up.
Urgent symptoms never get a draft. They go to a triage queue with a flag. Stanford's error-taxonomy work (arXiv 2509.22565) shows why a draft is the wrong tool here. Errors were more frequent in drafts clinicians actually used. Most of the increase was subtle omissions, led by Missed Escalation of Care at +1.1 percentage points. A draft that sounds calm about chest tightness is worse than no draft.
Mixed messages take the most cautious label. "Can you refill my lisinopril, also my chest feels tight" is an urgent message that happens to mention a refill. The classifier can be an LLM, but it outputs labels from a fixed set, and the priority order decides.
Bias the router toward "no draft." The errors aren't symmetric. A false "urgent" costs a clinician a reply written from scratch. A false "refill" on a symptom message can cost a missed escalation. Tune your thresholds for that asymmetry.
Scheduling doesn't need a model. If the answer is an appointment time, fill a template from the Appointment resource. It can't hallucinate and it costs nothing. Per the summary, California's AB 3030 also exempts administrative messages like scheduling from its AI disclaimer rule.
Results need tone rules, not just facts. In interviews, patients wanted tone to scale with stakes: brief for routine requests, more supportive for ambiguous results (Owens et al., JAMA Netw Open 2026). So results get their own prompt, separate from refills.
Ground every draft in the patient's FHIR record
Once a message earns a draft, the model sees only what that category needs:
- Refills: active MedicationRequests.
- Results: the relevant Observations with reference ranges.
- Anything touching follow-up: upcoming Appointments.
I'd hand the model typed, trimmed context, not raw Bundles. It's the same reasoning behind giving a FHIR MCP server typed tasks instead of raw FHIR tools.
Every dose, lab value and date in the draft must trace back to a resource you sent. Check this deterministically after generation, not with another prompt:
import re
CLAIM = re.compile(
r"\b\d+(?:\.\d+)?\s?(?:mg|mcg|mmol/L|mg/dL|%|units)\b"
r"|\b\d{1,2}/\d{1,2}(?:/\d{2,4})?\b"
)
def untraced_claims(draft: str, context_values: set[str]) -> list[str]:
"""Doses, lab values and dates in the draft that aren't in the FHIR context."""
return [m for m in CLAIM.findall(draft) if m.replace(" ", "") not in context_values]
Anything untraced gets highlighted for the clinician. It doesn't get silently rewritten.
This regex only catches numbers with units, plus dates. A bare value like "potassium 5.8" slips through. To close that gap, also match unitless numbers against the Observation.valueQuantity values you sent.
Then add a second rule: no hedging on doses, results or next steps. "Your levels look a little high, you may want to follow up" isn't allowed. The draft has two options:
- State the value from the Observation and a concrete next step.
- Drop the sentence and leave a visible "next step needed" marker.
A banned-phrase list ("may want to," "probably," "should be fine") catches most of it. Vague medical language feels safe to the model. A patient reads it as a real answer, and a busy clinician can easily miss it.
The model also has to ask questions it doesn't naturally ask. In the Dartmouth study (arXiv 2601.11344), zero-shot models scored 0.17 and 0.08 theme-level edit-F1 on the symptom and medication question-asking themes. Adapting them raised those scores to 0.79 and 0.49.
One example from the press coverage shows what's missing. A 32-year-old woman on reflux medication reported nausea. The AI suggested dietary changes. The physician asked whether she could be pregnant. Each category's prompt should list the questions a clinician would ask, and your eval should check that they're there.
Measure edit rate and discard rate by message category
The number I'd put on the dashboard is how much clinicians change drafts and how often they throw them away, split by route. One aggregate edit rate hides everything. Refills might be fine while results drafts get rewritten every time.
The Dartmouth paper gives you a useful split:
- Content-level edit-F1 asks whether the draft got the specific facts, instructions and action items.
- Theme-level edit-F1 asks whether it addressed the right goals, like reassurance, triage guidance or follow-up.
Track both. A draft with the right facts and the wrong goal needs a different fix than one with the right goal and a wrong dose.
Keep your expectations honest. The authors estimate the best drafting models would cut clinician edits by 25-26%, and they call that a modest reduction. Usage can be much lower than teams expect. At UCHealth, 21,323 drafts were generated and 2,596, or 12%, were used. Net promoter score was +58 for nurses and -43 for clinicians (English et al., JAMA Netw Open 2024). So split by role too.
A low edit rate can mean rubber-stamping. In Biro et al. (npj Digit Med 2025), 20 primary care physicians reviewed drafts with errors seeded in:
- Each error was missed by 13 to 15 of them.
- 35-45% of the erroneous drafts were sent unedited.
- 90% said they trusted the tool.
Pair edit rate with periodic seeded-error audits, or you'll read silence as quality.
Then close the loop:
- Store the message, the context, the draft and what the clinician actually sent.
- Turn heavy edits and discards into regression cases.
- Run every prompt or model change against those cases before it ships.
This is the kind of small labeled dataset covered in product evals in three steps. If you grade with an LLM judge, check it against clinician labels on a recurring schedule. An uncalibrated judge drifts as silently as the model does.
The Dartmouth authors go further and conclude that drafting models need to adapt to individual clinicians' preferences. Per-clinician edit data is how you'd know whether that's worth it.
Should AI send patient portal replies? Keep drafts pended with FHIR Provenance
The model never sends anything. A draft goes back into the clinician's queue as pended content, and a human signs it.
Burdick, a co-author of the Dartmouth paper, said in the release coverage that he doesn't foresee portals replying without clinician review. Review before sending also satisfies AB 3030. The statute's disclaimer isn't required when a licensed provider reads and reviews the message.
Attach FHIR Provenance to every draft, recording which model built it and which resources it drew from:
{
"resourceType": "Provenance",
"target": [{ "reference": "Communication/draft-8812" }],
"recorded": "2026-08-14T15:02:11Z",
"agent": [{
"type": { "coding": [{ "system": "http://terminology.hl7.org/CodeSystem/provenance-participant-type", "code": "assembler" }] },
"who": { "reference": "Device/inbox-drafter-v14" }
}],
"entity": [
{ "role":
When the clinician signs, record a second Provenance with them as the verifier. Now you can answer the questions that come up after something goes wrong:
- Which model version wrote this?
- What did it see?
- Who approved it?
The source entities are the same IDs your claim check traced against, so audit and grounding come from one mechanism. US Core describes Provenance as a foundation for authenticity, trust and reproducibility.
If you track only one thing, track discard rate by category. A draft nobody uses is a cost. A draft that's always used and never edited needs a seeded-error audit.
