LLM Incidental Findings Follow-Up in Radiology: The Model Finds It, Code Closes the Loop

An LLM is good at reading a radiology report and finding the follow-up recommendation buried in the impression. It is the wrong thing to trust with whether that follow-up actually happened. Use the model to find. Use code to track. That split is the whole design.

The model reads each final report and pulls out every recommendation, along with the exact sentence it came from. Deterministic code turns each one into a FHIR Task with a due window, watches later imaging, and decides when the loop is closed. The model never decides that a follow-up happened. It never orders anything, never contacts a patient and never cancels anything.

This is the same rule I use for AI clinical notes and ICD-10 coding. The model proposes, with citations. Code checks. A human owns anything that changes care. It matters most here because the failure is silence. A recommendation sits in an impression nobody re-reads, and the scan never happens.

Why radiology follow-up recommendations get lost

The radiologist did the job. The finding is in the report, and the recommendation is in the impression. Then the report goes to an ordering clinician who was asking about something else.

Northwestern Medicine runs NLP on nearly every imaging study across its 11 hospitals. Its team says that usually only about two-thirds of follow-up recommendations get addressed, and that's the best case. In its first year, the system screened more than 460,000 studies and flagged 23,000 for lung follow-up. That's over 60 a day. Once you start looking, a navigator team inherits that volume.

Don't count on regulation to force the work. The Joint Commission goal NPSG.02.03.01 covers timely reporting of critical results. An incidental nodule with a scan recommended in six months isn't a critical result. Pennsylvania's Act 112 goes further. It requires imaging providers to notify patients directly of significant abnormalities within 20 days of sending results to the ordering practitioner. But notification isn't follow-up. Neither rule tells you whether the scan happened.

When to run follow-up extraction on radiology reports (HL7v2 ORU and FHIR)

The pipeline wakes when a radiology result arrives: an HL7v2 ORU^R01, or a FHIR DiagnosticReport that goes final. I've written up the ingest side for Mirth and RabbitMQ, Snowflake and Google Cloud. Three rules matter for this pipeline.

Run on final reports only. Preliminary reads change.

Assemble the whole report before extraction. ORU report text is often split across many OBX segments, with the impression and any addendum in their own lines. Join them in sequence before the model sees anything, and keep the segment boundaries so a cited span can be traced back to where it came from.

Re-run on corrections and supersede. Key everything on the accession number. A corrected or amended report replaces the earlier extraction. Cancel the Tasks the old version created and write new ones. Don't stack duplicates, or your navigators will chase the same nodule twice.

How to extract follow-up recommendations with an LLM

The model returns one object per recommendation. Each object has the finding, the modality, the interval, the guideline category if the radiologist stated one, a status, and the verbatim span.

{
  "recommendations": [
    {
      "finding": "6 mm solid nodule, right upper lobe",
      "modality": "CT chest",
      "interval": "6-12 months",
      "guideline": "Fleischner 2017",
      "status": "actionable",
      "span": "Recommend follow-up chest CT in 6-12 months per Fleischner guidelines."
    },
    {
      "finding": "left adrenal thickening",
      "status": "hedged",
      "span": "Consider clinical correlation."
    }
  ]
}

Every recommendation cites a span, and code checks it. Normalize whitespace and case, then require the span to appear as a substring of the report. If it doesn't, reject the item. A recommendation the model can't point to doesn't exist.

Hedged wording gets its own label. "Consider clinical correlation" is not a follow-up recommendation. If you force the model to choose only between actionable and not actionable, it will choose inconsistently, because humans do too. In a 400-report study, expert agreement on "incidentaloma requiring follow-up" was only 0.70 F1, against 0.93 for "no incidentaloma". A third label keeps that ambiguity visible instead of burying it in your false-positive rate. Hedged items don't create Tasks. They get counted, so you can see how much of your volume is hedged.

The model reports a guideline category only if the radiologist stated one. The model doesn't apply Fleischner itself. Code checks the context. Per the 2017 paper, the guideline doesn't apply to patients under 35, immunocompromised patients, patients with known cancer, or lung cancer screening. If a report cites Fleischner for one of those patients, code flags it for a human. Code already has the patient's age, so the model doesn't need it.

Give the model narrow input, and keep a cheap baseline. In 2511.11867, the best configuration narrowed the input to metadata plus the recommendation sentences and their context. On document-level follow-up detection, GPT-4o reached F1 0.832. Logistic regression reached 0.776, and human agreement was 0.846.

A second paper from the same group, 2512.05537, looked at lesion-level incidentaloma classification and found a bigger gap. The best LLM scored 0.79 incidentaloma-positive macro-F1, against 0.70 for the best supervised encoder. Adding anatomical context gained up to 0.08 macro-F1. These are different tasks, so don't read them as one comparison.

Train the boring classifier first. Then run the LLM at temperature 0 against it on your own reports. If the LLM doesn't clearly win, ship the classifier.

How to represent a radiology follow-up recommendation in FHIR

Code writes the FHIR, not the model. A radiologist's recommendation is not an order, so the ServiceRequest gets intent: proposal. The interval goes in occurrencePeriod, and reasonReference points back to the DiagnosticReport. A Task focuses on that request, holds the due window in restriction.period, and carries workflow state in businessStatus.

// ServiceRequest: from a CT read on 2025-03-14
{ "resourceType": "ServiceRequest", "status": "active", "intent": "proposal",
  "code": { "text": "CT chest" }, "bodySite": [{ "text": "chest" }],
  "occurrencePeriod": { "start": "2025-09-14", "end": "2026-03-14" },
  "reasonReference": [{ "reference": "DiagnosticReport/ct-0314" }] }

// Task: due window = recommended range + 30-day grace
{ "resourceType": "Task", "status": "requested", "intent": "plan",
  "focus": { "reference": "ServiceRequest/sr-1" },
  "restriction": { "period": { "start": "2025-09-14", "end": "2026-04-13" } },
  "businessStatus": { "text": "open" } }

The hard part is the due date. Fleischner gives ranges, such as 6–12 months to confirm a subsolid nodule persists and 3–6 months for a part-solid one, and the guideline doesn't tell you which end to track. My rule: the window opens at the early end of the range, and the item goes overdue at the late end plus a grace period. Whatever rule you pick, write it down and version it. When a navigator asks why something showed up as overdue, the answer should point to a versioned rule.

Closed-loop tracking for incidental findings: was the follow-up done?

The authors of 2511.11867 model completion by linking later reports back to an "index report" that holds the recommendation. That's the right mental model for closure, and it doesn't take a model to run.

def match(task, result, open_tasks):
    if result.status != "final":
        return None
    if task.service_request_id in result.based_on:
        return ("completed", "basedOn")
    if not (result.patient == task.patient
            and result.modality == task.modality
            and result.body_site in task.body_sites):
        return None
    # start/end come from Task.restriction.period; end already includes grace
    in_window = task.start <= result.date <= task.end
    # rivals deliberately ignore the window: any other open Task for the
    # same patient, modality and site makes the match ambiguous
    rivals = [t for t in open_tasks if t.id != task.id
              and t.patient == task.patient
              and t.modality == task.modality
              and result.body_site in t.body_sites]
    if in_window and not rivals:
        return ("completed", "patient+modality+site+window")
    return ("needs-review", "out-of-window" if not in_window else "rival-task")

The exact match is basedOn. If the follow-up was ordered from the proposal, the later DiagnosticReport.basedOn points at your ServiceRequest. That's the case you want, so make ordering from the proposal the easy path.

The fallback match is patient, modality, body site and window. Store which rule closed each Task. You'll need that audit trail.

An order is not closure. When an order is placed, the Task moves to in-progress. Only a final result moves it to completed. Orders get cancelled, and a cancelled order that already closed the loop is a missed follow-up that looks finished.

Anything ambiguous goes to a human. A matching result outside the window goes to review. So does any result when another open Task exists for the same patient, modality and site, whatever its window. A navigator decides.

Acknowledgement with a reason also closes the loop. Some follow-ups never happen because the patient or clinician decides they aren't needed, and 2511.11867 notes this explicitly. Northwestern lets the physician either accept, which orders the scan, or pick an acknowledgement reason that closes the workflow. In my design, that's a navigator or clinician action that completes the Task with a reason code. A person makes it, it gets recorded, and it is never inferred.

Surfacing overdue radiology follow-ups: navigator worklists and CDS Hooks

Overdue items go to two places.

The first is a navigator worklist sorted by days overdue. Show the report span inline, so nobody has to open the PDF to see why the item exists.

The second is a quiet CDS Hooks patient-view card. The spec lets a service return an empty cards array, and most of the time that's the right answer. When something is overdue, return one info card with a summary under the 140-character limit, such as "Chest CT recommended 6–12 mo after 3/2025 CT. None found." Getting a card to render in Epic is its own project. I covered that in CDS Hooks: What Nobody Tells You Before You Build.

A human orders, messages the patient, or closes the item. Northwestern's flow also stops at a "Place Follow-up Orders" button the physician has to click.

How to evaluate LLM follow-up detection

Track two numbers and never blend them.

Missed actionable findings are safety misses. Review every one.

False flags are navigator workload. Treat them as a budget and spend it deliberately.

A single F1 hides the tradeoff you care about. Break both numbers down by anatomy too. 2512.05537 sampled six regions (kidney, liver, lung, pancreas, adrenal and thyroid), which the authors chose because they're more likely to contain incidentalomas. A good aggregate score can hide one weak region.

Every navigator override becomes a regression case: the report, the extraction, the corrected label and the reason. Rerun the suite on every prompt change and every model swap. With expert agreement at 0.70 on what counts as actionable, the overrides are your real ground truth. The rest of the method is in my guide to LLM evals.

Evaluate the matcher too. A false closure is the worst bug in this system, because it makes a missed follow-up look done. Every week, pull a sample of auto-closed Tasks, break it down by the rule that closed each one, and have a human confirm them.