Stop Polling the EHR: FHIR Subscriptions and Event-Driven AI Agents

Healthcare AI agents should wake on EHR events, not on a timer. Most agent demos skip Subscriptions and FHIR webhooks entirely and poll on a schedule. Then they call it real-time. That's the mistake. A polling agent sees the 2am lab result minutes late. A CDS hook never sees it at all, because nobody has the chart open.

I've built on both sides of that gap. On one side are synchronous CDS Hooks, which only fire inside a clinician's workflow. On the other are HL7v2 pipelines: Mirth into RabbitMQ, HL7v2 into Google Cloud and Pub/Sub, and HL7v2 into Snowflake. Those posts were about moving data into storage. This one is about the layer in between, where an event in the EHR wakes an agent.

My bet today: if the hospital already runs an ADT/ORU feed through an interface engine, start there. Use topic-based FHIR Subscriptions where the EHR actually supports them. Use a vendor webhook where that's all you get. Whatever the source, normalize it into one tiny internal event, so you can swap sources later without touching the agent.

Polling the EHR costs more than it looks

Here's a back-of-envelope estimate with illustrative numbers. Take 500 inpatients and poll Encounter and Observation for each one every five minutes. That's 500 × 12 × 2 = 12,000 searches an hour, or 288,000 a day. Say 2% of polls find something new. Then about 282,000 queries a day hit the EHR and do nothing.

You still lag. On a five-minute interval the agent is 2.5 minutes behind on average and five minutes behind at worst. Cut the interval to one minute and the query count goes up five times. Polling makes you trade latency against load, and you lose on both.

Polling has one job left: reconciliation. Every event source drops things. Keep a slow background sweep, hourly or nightly, that compares current state against what you processed. That's your backstop, not your trigger.

FHIR Subscriptions vs FHIR webhooks vs HL7v2 feeds

HL7v2 ADT/ORU feeds are what I'd bet on for coverage today. HL7v2 is how most hospitals have moved clinical data for 30+ years (primer here). The feed already exists, and you own the pipe. The interface engine parses the message, stores the raw copy, and publishes a tiny event in its place. The cost is that HL7v2 has no event counter, so dedupe and gap detection are all on you. Dedupe on the sender (MSH-3/MSH-4) plus the message control ID in MSH-10. MSH-10 alone isn't unique across senders.

Topic-based FHIR Subscriptions are the portable target. R5 rebuilt Subscriptions around a SubscriptionTopic resource. A topic defines what triggers a notification, such as an Encounter created or updated to active. It also defines which filters a client may set, such as one patient or a group (FHIR build spec). For R4 servers, HL7 publishes the Subscriptions R5 Backport IG. It is still Trial-use, at 1.2.0-ballot. An Epic engineer is on its contributor list. That shows Epic helped write the spec. It doesn't show that Epic ships it. Confirm production support with your EHR vendor before you design around it.

Vendor webhooks are a bridge. Build on them, but keep a fallback. athenahealth's Event Subscription Platform "largely conforms" to the backport STU 1.0.0 and differs mainly in error handling. It supports one channel (rest-hook) and one payload type (id-only), and only 2-legged OAuth apps can use it. The Eventing Alpha Launch was 2024-11-19. The README also says these APIs are not part of athenahealth's certified US Core FHIR APIs.

So it ships, but it's an alpha, and it isn't certified. I wouldn't put a clinical workflow on it without the reconciliation sweep behind it. The design isn't the problem. Error handling is where it differs from the spec, and error handling is the part this whole design leans on.

Events also make your hooks better. athenaOne's CDS Hooks guide strongly recommends prefetching data overnight or in response to events, so hook responses stay fast. The event does the slow work, and the hook just reads the result.

The agent only ever sees the normalized event. Its shape sits at the top of the handler code below. Nothing clinical goes in it.

Id-only EHR event notifications keep PHI out of the queue

Subscriptions offer three payload levels: empty, id-only and full-resource. The R5 spec is plain about it: "To mitigate the risk of information leakage, systems SHOULD use the minimum level of detail consistent with the use case. In practice, id-only provides a good balance between security and performance for many real-world scenarios" (HL7).

For agents I'd go further: id-only, always. A full-resource notification turns your queue, your dead-letter store and your logs into copies of the chart. Every one of them then needs the same controls as the EHR. With id-only, the queue holds references, and the agent fetches what it needs at run time with its own token. If the record changed between the event and the run, the agent reads the current version, not a stale snapshot.

There are two caveats. First, an MRN is PHI too, so keep it out of the queue. Use the FHIR logical id or an opaque internal key, and keep the queue inside your compliance boundary anyway. Second, scope matters at the fetch. The backport IG has id-only-with-auth example notifications and an extension that carries authorization information, such as a token, in the notification. Whatever token the agent ends up with, scope it to the task. That's the whole argument of why system/ scopes are god mode for an agent. Also give the agent a typed fetch, not raw FHIR, as I laid out in the FHIR MCP server post.

The same rule holds for HL7v2. The interface engine keeps the raw message. The event carries the patient key, the trigger event type, the message control ID and a timestamp.

Duplicates, gaps and silence

Notifications arrive twice, late, or never. The backport IG's author was blunt about it. Errors can happen in the server while processing, in the server while sending, at the endpoint, or in your client (Canessa, DevDays). The backport IG keeps server error handling minimal on purpose. The server increments its counter, updates the subscription status and keeps answering $status. Noticing the gap is your job (backport IG errors).

You have three tools:

• Heartbeats. Set heartbeatPeriod. Leave it out and no heartbeat is sent, so a dead subscription looks exactly like a quiet night.
• Event counts. eventsSinceSubscriptionStart includes the current notification, so the first event carries 1. In R4B it's typed as a string, so parse it.
• $events. Servers MAY support GET [base]/Subscription/[id]/$events?eventsSinceNumber=... to replay missed notifications. It's Maturity Level 0, so don't assume it exists.

Use one decision rule for the counter. Trust count-based gap detection on R4B servers and on best-effort subscriptions. Under the current published spec, guaranteed delivery makes the counter optional (HL7). There, lean on heartbeats and the reconciliation sweep. Find out which case your server is in before you write the handler.

There's one more trap. Newer spec text says servers SHOULD keep numbering after an error, but older servers may restart silently. So a lower number that you haven't seen before is a signal to reconcile.

The webhook handler should do this bookkeeping and almost nothing else. Acknowledge fast, enqueue, and never run the agent inside the HTTP request. The backport IG gives an example: you last saw event 21, and event 23 arrives. Wait briefly in case 22 is still in flight, then recover.

@dataclass(frozen=True)
class AgentEvent:              # the only shape the agent ever sees
    source: str                # "fhir-sub" | "athena" | "hl7v2"
    sender: str                # subscription id, or MSH-3/MSH-4
    seq: str                   # event number, or MSH-10
    patient_key: str           # opaque key, never an MRN
    focus: str                 # e.g. "Observation/abc123"
    event_type: str            # topic, or trigger like "ADT^A08"
    ts: str
    priority: str = "normal"   # "urgent" skips debounce

def handle_notification(bundle):
    status = parse_status(bundle)  # SubscriptionStatus from the Bundle
    sub = status.subscription_id
    if status.type in ("handshake", "heartbeat"):
        mark_alive(sub)
        return
    n = int(status.events_since_start)      # typed string in R4B
    if already_seen(sub, n):
        return                              # redelivery: check this first
    last = last_seen(sub)                   # highest number seen so far
    if n < last:
        flag_possible_reset(sub, n)         # late gap-filler or reset
    elif n > last + len(status.events):
        schedule_gap_check(sub, last, n)    # wait, then $events or re-query
    for ev in status.events:
        enqueue(to_agent_event(sub, ev))
    record_seen(sub, n)                     # keeps the max for last_seen

The order of the checks matters. The seen check runs before the reset check because a redelivered old notification also has a number below last. Check for a reset first and every duplicate looks like a server reset, so you reconcile for nothing. Once duplicates are out, a new number below last is either a late arrival filling a gap or a real reset. Reconcile either way. It costs a few reads, and trusting a reset counter costs missed events.

Don't make the notification your idempotency key. Key the agent run on the resource ids and meta.versionId values it reads. Two notifications that point at the same versions add up to one run, and duplicates stop mattering.

HL7v2 has no counter, so dedupe on sender plus MSH-10, and detect silence with per-sender volume baselines. An ADT feed that normally carries traffic every hour and drops to zero at 10am on a weekday is broken. Alert on that, not just on errors.

Debounce EHR event bursts per patient

One admission is never one message. You get an ADT^A01, then a run of A08 updates, maybe an A02 transfer, then orders and ORU^R01 results trickling in. How many depends on the site's interface config, so count your own feed before you tune anything. Trigger an agent per event and you pay for many runs that each see half the picture. Worse, the clinician gets output from the half-informed ones.

Collapse events per patient with a quiet window and a hard cap:

QUIET = 120     # seconds with no new event for this patient
MAX_WAIT = 900  # never hold a patient longer than this

def on_event(e):
    w = windows.setdefault(e.patient_key, Window(first=now()))
    w.events.append(e)
    w.fire_at = min(now() + QUIET, w.first + MAX_WAIT)

# a sweeper fires each due window as ONE agent run carrying all its events

Those numbers are starting points. The cap matters most. Without it, a patient with a steady trickle of updates never gets processed.

The tradeoff is latency, and some events can't wait. A critical lab shouldn't sit behind a two-minute window. Id-only works against you here, because the router can't see the value. Route urgency upstream instead. With Subscriptions, give the events that bypass debounce their own topic or filter. With HL7v2, the interface engine sees OBX-8, the abnormal flags field, before it strips the message down. Let it set priority to urgent on panic values like HH or LL.

Dead-letter queues and an audit row for every agent run

Retry with exponential backoff, then park the event. I'd start at five attempts, doubling from 30 seconds, and tune from there.

What goes in the dead-letter record matters more than the retry math. Store the pointer: the source, the sender, the event number or MSH-10, the focus reference, the attempt count, the last error and the first-seen time. Never store the payload. If you went id-only, there's no payload to leak, and the DLQ never becomes another copy of the chart.

A DLQ nobody drains is a slower way to lose events. Give it an owner, and alert on its depth and on the age of the oldest record. To replay, put the pointer back through the same handler as live traffic, so it hits the same dedupe. The agent re-fetches current data, so a replay tomorrow reads tomorrow's chart. The audit row records exactly which versions it saw.

Every agent run gets an audit row that ties it to the events that woke it and the data it read:

create table agent_runs (
  run_id            uuid primary key,
  patient_key       text not null,   -- opaque id, never an MRN
  trigger_events    jsonb not null,  -- [{source, sender, seq, focus, ts}, ...]
  resource_versions jsonb,           -- ids + versionIds the agent read
  input_hash        text,            -- hash of resource_versions, set once read
  status            text not null,   -- queued | running | done | dead
  attempts          int not null default 0,
  started_at        timestamptz,
  finished_at       timestamptz
);

-- the same input can't run twice, but a dead run doesn't block a retry
create unique index agent_runs_live_input
  on agent_runs (input_hash) where status <> 'dead';

The input hash is the version-based idempotency from earlier, enforced by the database. Make it a partial index, not a plain unique column. With a plain unique constraint, a run that died blocks every retry on the same input forever. With the partial index, a dead run frees the slot, while a queued, running or finished run still blocks duplicates.

Someone will ask why the agent flagged a patient at 2:14am. You can answer with the events, the exact resource versions and the outcome, all in one query.

Anything the agent writes back must be idempotent too. A replayed run that files a second task or sends a second message is worse than a missed one, because now a human has to clean it up.