AI HEDIS Chart Abstraction: The Model Finds Evidence. It Never Decides Compliance.
NCQA is retiring HEDIS hybrid by MY 2029. Use an LLM to find chart evidence with source spans, and let CQL decide compliance.
Key takeaways
- Use the LLM to find evidence in notes and link it to its source text; let CQL or spec-based code decide numerator, denominator and exclusions.
- Check spans, negation, per-event date windows and patient-reported history in code, and send anything uncertain to a human abstractor.
- Report error per measure, separately for numerator and exclusions, and turn every abstractor override into a regression case.
An LLM should never decide whether a member meets a HEDIS measure. Use it the way a good abstractor uses a highlighter. It finds the evidence in the note and points at the exact sentence. Deterministic measure logic decides what counts.
If a tool's output is a compliance flag with no sentence behind it, I wouldn't let it near a submission.
I've built HL7v2 and FHIR pipelines on Mirth, RabbitMQ, Snowflake and Google Cloud, and I've done LLM work on clinical notes for MedScrub, MedHook and eSpiral. The rule I keep coming back to is the one behind my ICD-10 coding post and my hallucination detection post: the model picks, code decides. Quality reporting is the same problem with audits and Star Ratings attached. A compliance rate is a count, and counts belong to code.
When HEDIS hybrid measures are being retired
NCQA plans to phase out the hybrid method by MY 2029 and make HEDIS fully digital by MY 2030. Hybrid is the method that let plans pull a sample of charts and have abstractors find what claims missed. Here is that post's schedule for the eight measures that allow hybrid reporting in MY 2025:
| Measure | Moves off hybrid | Goes to |
|---|---|---|
| LSC | MY 2026 | Administrative-only (see below) |
| WCC | MY 2027 | Administrative-only |
| PPC | MY 2028 | Administrative-only |
| CBP | MY 2028 | ECDS |
| BPD | MY 2028 | ECDS |
| GSD | MY 2029 | ECDS |
| TRC | MY 2029 | ECDS |
| COA | MY 2029 | ECDS |
Don't hard-code this table into your roadmap. NCQA's newer ECDS page lists LSC-E as transitioning to ECDS in MY 2026, not going administrative-only. It already lists BPD-E and the hypertension blood pressure measure as MY 2026 ECDS measures. NCQA says some measures will offer both versions during a transition, which is the likeliest reading. Every change still goes through public comment. Re-check each measure every time NCQA publishes a spec update.
Sampling goes away. ECDS requires data for the entire eligible population, where hybrid abstracted a sample and extrapolated. Nobody can hand-abstract every member. So AI's job here is triage: it picks which charts an abstractor reviews.
Provenance is up for grabs. Starting MY 2026, NCQA no longer requires source system of record reporting for ECDS. It says it is exploring ways to "establish data provenance in alignment with interoperability standards." If you build now, put provenance on every finding from day one. Adding it later is miserable.
Medicaid needs its own caveat. The 2026 Core Sets still list Hybrid for CBP and WCC, and their ECDS specs are narrative plus value sets, not CQL. "Hybrid is dead and everything runs on CQL" is wrong for state reporting this year.
ECDS and NLP: why clinical notes still matter for digital quality measures
Digital doesn't mean structured. NCQA's own TRC transition post makes three points. The current measure "relies heavily on medical record data." Some of its components can't currently be represented in LOINC, CPT, HCPCS, SNOMED CT or the other code systems it lists. And discharge information often arrives as faxes and PDFs uploaded into the chart. The earliest the ECDS version of TRC can land is MY 2028. Because hybrid TRC feeds Medicare Star Ratings, NCQA is coordinating the change with CMS.
So the evidence plans used to chase in charts doesn't disappear. It moves out of a seasonal sample review and into a pipeline that runs across the whole population and keeps an audit trail.
Step 1: Use the LLM to extract HEDIS evidence from clinical notes
The model's job is narrow: read a note and return findings. Each finding points at its source text.
type Finding = {
measure: string; // "COL", "CBP", "TRC"...
role: "numerator" | "exclusion"; // a guess; code can overrule it
code: { system: string; code: string }; // picked from a retrieved value set
effective: string | null; // date as written in the note
source: "documented_result" | "patient_reported";
negated: boolean;
span: { documentId: string; start: number; end: number; quote: string };
confidence: number;
};
Three rules keep this safe.
The model doesn't invent codes. Retrieve the measure's value set and have the model pick from that list. My ICD-10 post covers why: GPT-4 scored 33.9% on ICD-10-CM code lookup. Picking from a retrieved list removes the invented-code failure. It doesn't remove the wrong-code failure. The model can still pick a real code that doesn't fit the note, which is why the rest of this pipeline exists.
The model doesn't decide what counts. role is a hint about where the finding might matter. If the measure logic disagrees, the hint is discarded.
Every finding carries its span. Write each accepted finding as a candidate FHIR resource, such as a Procedure, Observation or Condition. Then write a Provenance that links the candidate resource to its source DocumentReference and records the model and prompt version that produced it. Keep the span offsets and quote with it. If you've done write-back, this is the same DocumentReference plumbing from writing AI notes back to Epic, run in reverse.
Integration work teaches the underlying rule early. Parsed HL7v2 rows are only as trustworthy as your ability to get back to the message they came from. Model output is no different. When an auditor asks why a member counted, you hand them the sentence.
Step 2: Let CQL decide numerator, denominator and exclusions
Measure logic is deterministic. It runs against resources, uses the measure's value sets and the measurement period defined in the spec, and returns the same answer every time. Keep it that way.
The model sits upstream of the measure, never inside it. Candidate resources don't reach the CQL engine until they pass the checks in step 3. Where the spec is CQL, run it. Where the spec is narrative, as in the 2026 Core Sets, write the logic in code from the spec and value sets. Version it and test it like any other rules engine. Either way, no prompt ever asks "is this member compliant?"
You also get a clean way to debug failures. When a rate looks wrong, either the evidence was wrong, which points to extraction, or the logic was wrong, which points to measure code. If a model decides compliance, you can't tell which one broke. A product that can't tell you which one broke isn't an abstraction tool. It's a rate generator, and you'll be the one explaining its rates to an auditor.
Step 3: Handle negation, measurement period dates and patient-reported history
This step catches the confident wrong answers. "No colonoscopy on file." "Father had colon cancer." "Colonoscopy declined." "Patient says she had one a while back." A model can read all four as evidence of a colonoscopy. Don't trust it to sort them. Check them in code.
// sentenceAround, allowsPatientReported and threshold are per-measure config, not model output
function gate(f: Finding, note: string, win: { start: Date; end: Date }) {
// the quote must exist in the note at the offsets claimed
if (note.slice(f.span.start, f.span.end) !== f.span.quote) return "reject";
// negated is never numerator evidence; a documented refusal can matter elsewhere, so a human reads it
if (f.negated) return f.role === "numerator" ? "reject" : "abstractor";
// crude on purpose: a cue can only escalate, never accept
const cue = /\b(no|denies|declined|refused|negative for|family history)\b/i;
if (cue.test(sentenceAround(note, f.span))) return "abstractor";
// no date, no credit; the window comes from the measure spec, not the prompt
if (!f.effective) return "abstractor";
const d = new Date(f.effective);
if (d < win.start || d > win.end) return "reject";
if (f.source === "patient_reported" && !allowsPatientReported(f.measure))
return "abstractor";
return f.confidence >= threshold(f.measure) ? "accept" : "abstractor";
}
Negation isn't one rule. A negated finding should never credit a numerator. But a documented decline or refusal isn't noise. Depending on the measure, it can matter for exclusions or outreach. Don't silently drop it. Route it to a person and encode the per-measure rule once you know it. The same goes for disagreement: when the model says positive and a negation cue fires in the sentence, neither side wins. A human decides.
The window isn't always the measurement period. TRC has two rates anchored to different events. Notification of Inpatient Admission needs documentation from the day of admission through 2 days after. Receipt of Discharge Information needs documentation from the day of discharge through 2 days after. A member admitted March 3 and discharged March 10 needs the notification dated March 3 to 5 and the discharge information dated March 10 to 12. A note dated March 6 that mentions the admission falls inside the measurement year and still fails. Compute each window from the spec, per event, and keep it out of the prompt.
Patient-reported history gets its own path. The 2026 Core Set ECDS method accepts beneficiary-reported services if they are recorded, dated and kept in the legal health record. That is a different bar from a documented result. Keep source as its own field, encode the rule per measure, and never credit an undated "a few years ago."
Thresholds are per measure. For each measure, pick the lowest confidence at which numerator false positives on your labeled set stay under a rate you'd defend in an audit. Then re-pick it whenever the model or prompt changes. One shared threshold won't work, because the inputs differ: a progress note can carry several blood pressure readings, while discharge information often arrives as a faxed PDF.
The abstractor queue should show the highlighted span, the candidate resource and three buttons: accept, edit, reject. Show the sentence and enough context to judge it, not the whole chart. Minimum necessary PHI applies to your own reviewers too. It's the same principle as in my FHIR MCP server post. Log every click: the reviewer, the timestamp, the original finding exactly as the model returned it, and the decision with any edit. You need those records in step 4.
Step 4: How to measure AI chart abstraction accuracy per HEDIS measure
A single accuracy number for "the abstraction model" tells you nothing, because errors don't cost the same.
A numerator false positive credits a member who isn't compliant. It inflates your rate, and it's the error you least want an auditor to find. An exclusion false positive removes a member from the denominator. That inflates the rate too, but more quietly. A numerator false negative shows a care gap that isn't real. It costs outreach and drags the rate down, but you can see it and fix it.
So report error per measure and per role, against a hand-abstracted sample where the abstractors never saw the model's output. Keep the scoring binary, as in my evals guide: for each finding, the model either got it right or it didn't. If you've never built a labeled set, start with the three-step version. I'd start with about 100 charts per measure and grow the set from overrides. Have two abstractors label a slice of it, so you know how often humans disagree before you grade the model against them.
Don't use last year's hybrid rate as your benchmark. NCQA gives ECDS and traditional versions separate IndicatorKeys and doesn't trend them against each other. The hand-abstracted sample is your ground truth.
Every abstractor override becomes a regression case:
{"measure": "COL", "quote": "pt reports colonoscopy a few years ago",
"model": {"role": "numerator", "effective": null},
"abstractor": "reject: no date", "expected_gate": "abstractor"}
Run the full set on every prompt change, model change and value set update. An override you don't turn into a test is a mistake you've agreed to make again.
Which HEDIS measure to automate first with AI
Pick one measure whose evidence lives in notes and whose hybrid version is on the way out. CBP is a good first choice: blood pressure readings buried in progress notes, with a hybrid version that runs until MY 2028.
Build the evidence layer once: spans, Provenance, the gate, the abstractor queue and the regression set. After that, each new measure is a value set, a window rule and a block of deterministic logic.
Hybrid season is ending. The evidence in the notes isn't going anywhere. Build the pipeline that can show its work, and keep the model's hands off the count.
Questions people ask
Can AI do HEDIS chart abstraction?
Yes, for finding evidence. The LLM reads notes and returns candidate FHIR resources linked to the exact text span, with a Provenance tying each one to its source DocumentReference. Deterministic measure logic decides compliance, and uncertain findings go to a human abstractor.
When is NCQA retiring HEDIS hybrid measures?
NCQA plans to phase out the hybrid method by MY 2029 and make HEDIS fully digital by MY 2030. Under its timeline, CBP and BPD move to ECDS in MY 2028, and GSD, TRC and COA move in MY 2029. NCQA's own pages don't fully agree on some measures, and every change goes through public comment.
Should an LLM decide HEDIS numerator compliance?
No. The model should only extract evidence. CQL, or code written from the measure spec, decides numerator, denominator and exclusions, so a wrong rate can be traced to either the extraction or the logic.
Do Medicaid Core Sets still allow hybrid reporting in 2026?
Yes. The 2026 Core Sets still list Hybrid for CBP and WCC. Their ECDS specs are narrative plus value sets, not CQL.