Articles

Mostly about building products, healthcare tech, and lessons learned along the way.

Loading articles

LLM ICD-10 Coding Accuracy: Why the Model Shouldn't Type the Code — Clint Johnson

LLM ICD-10 Coding Accuracy: Why the Model Shouldn't Type the Code

GPT-4 scored 33.9% on ICD-10-CM code lookup. Don't let the model type the code: retrieve from the code set, check rules in code, cite the note.

Clint JohnsonPublished Oct 25, 202610 min read

Key takeaways

Don't let an LLM type ICD-10 codes from memory; retrieve candidates from the Index, verify against the Tabular List, and version code sets by date of service.
Check validity, Excludes1, laterality and care-setting rules in code, and send Excludes1 conflicts to coder review because a hard block can reject a correct claim.
Require every code to cite a provider statement span, and measure principal and secondary diagnoses separately using audited coder overrides as labels.

Questions people ask

How accurate are LLMs at ICD-10 medical coding?

In the Mount Sinai benchmark published in NEJM AI, GPT-4 was the best model at 33.9% exact match on ICD-10-CM. That study tested looking up a code from its description, not coding from clinical notes, and some misses were equivalent or less specific codes rather than invented ones.

Can AI do automated ICD-10 coding without a human coder?

The article argues it shouldn't. The model should extract diagnostic statements and rank candidates retrieved from the code set, deterministic checks should enforce validity and coding rules, and a coder should make the final decision.

Should an Excludes1 conflict automatically block an ICD-10 claim?

No. The FY 2026 guidelines allow both codes when the conditions are unrelated and say to query the provider when relatedness is unclear, so a conflict should go to coder review. The check also has to walk up the code hierarchy, because Excludes1 notes are inherited from parent codes.

Are probable or suspected diagnoses coded differently inpatient and outpatient?

Yes. Inpatient rules code probable or suspected conditions as if they exist, while outpatient rules say to code symptoms, signs or abnormal results instead. Encounter type has to be an input to the pipeline and to any evaluation.

Don't let an LLM type an ICD-10 code.

A model can read a long note fast. It can find the provider's diagnostic statements. It can rank a short candidate list you hand it. Everything else belongs to code or to a coder. That includes whether a code exists, whether it's billable on the date of service, whether an Excludes1 conflict is acceptable, which side to code, and whether "probable" counts in this setting.

The stakes are why I'm this strict. In the revenue cycle, a wrong code turns into a denied claim or a compliance finding. It doesn't just lower a leaderboard number. I've written that silent failures are worse than loud ones. A confident, plausible, wrong code is about as silent as a failure gets.

The pipeline I'd build has four steps. Retrieve candidates from the official code set. Check validity, Excludes1, laterality and care-setting rules in plain code. Require every code to point at a sentence in the note. Then let the coder decide.

How accurate is AI medical coding on its own?

Not very. Soroush et al. at Mount Sinai tested GPT-3.5, GPT-4, Gemini Pro and Llama2-70b Chat against 15,950 ICD-10-CM codes, plus ICD-9-CM and CPT (NEJM AI). GPT-4 did best. It hit 33.9% exact match on ICD-10-CM, 45.9% on ICD-9-CM and 49.8% on CPT. The authors concluded that every model often produced "imprecise or fabricated" codes.

That number needs context. The study tested code querying: given a description, return the code. Reading a note and producing codes is a different task. Not every miss was invented, either. Of GPT-4's wrong ICD-10-CM answers, 10.9% were equivalent codes. GPT-3.5 produced the most "generalized but correct" codes, at 18.5% on ICD-10-CM. Common codes, short codes, and codes with short descriptions all scored better.

So the model's recall is weakest in the long tail, and much of its error is under-specification. You fix both outside the model.

The paper's introduction makes one more point. Smaller fine-tuned models extract ICD-10-CM codes from clinical text better than the best LLMs do. The weakness goes deeper than memory. Give the model a narrower job, and never ask it for the code string.

Retrieve candidates from the ICD-10-CM index and tabular list

The official guidelines already describe the algorithm. Look up the term in the Alphabetic Index, then verify the code in the Tabular List. The FY 2026 guidelines call using both "essential." In pipeline terms, that's retrieve, then validate.

The model extracts the provider's diagnostic statements from the note.
You search the Index for those terms across all four parts: diseases and injuries, external causes, the Table of Neoplasms, and the Table of Drugs and Chemicals.
Each hit expands into its Tabular hierarchy.
The model picks from that candidate list and nothing else.

Retrieval works here because the corpus is closed and has a known structure. It isn't a pile of chunks (more on that). When the right code never shows up in the candidates, you have a retrieval bug, and you can see it and fix it. A hallucinated code only shows up when the denial arrives.

Version everything by date of service. CMS publishes machine-readable files for every release, including code descriptions in tabular order, the tabular and index, the addendum, and the conversion table (CMS). FY 2026 had an October 1, 2025 release and a mid-year release on April 1, 2026. The FY 2026 update alone added 487 codes, revised 38 and deleted 28 (AHCA/NCAL). The FY 2027 files went live October 1, 2026. A single "current" code list goes stale at least twice a year. Load each release as its own table, and pick the one that covers the date of service.

The rules quoted below come from the FY 2026 guidelines, so check them against FY 2027 before you hard-code any of them.

Validate billable codes in code, not in the prompt

The validity rules are mechanical:

Categories are 3 characters. Subcategories are 4 or 5. Codes run 3 to 7.
Only codes are reportable. Categories and subcategories are not.
A code is invalid if it's missing required characters, including the 7th.
Where a placeholder X exists, it has to be there.

If your allowed set holds only full, reportable codes from the right release, one lookup enforces all four. Normalize once, and carry the encounter context, because later checks need it.

@dataclass
class Encounter:
    dos: date                                    # picks the code-set release
    setting: Literal['inpatient', 'outpatient']  # uncertain-diagnosis rules
    prior_codes: list[str]                       # staged bilateral treatment

def norm(code: str) -> str:
    return code.replace('.', '').strip().upper()

def validate(code: str, enc: Encounter, codesets: CodeSets) -> listIssue

A missing 7th character or a dropped X fails the lookup, and you don't need a regex. G40.0 comes back as not billable rather than invalid, which tells the coder to go one level deeper. The model can still pick the wrong code. It can't get an invalid one onto the claim.

How to handle ICD-10 Excludes1 conflicts: flag, don't reject

The guidelines define Excludes1 as "a pure excludes note. It means NOT CODED HERE!" AAPC says Excludes notes "often hold the key to preventing claims denials" and that payers are starting to deny over them (AAPC). Two details decide whether your check works.

Notes inherit. G40.009 can't be coded with G40.1- or G40.2-, because the Excludes1 sits on the parent G40.0 (IKS Health). A check against the exact code misses that. You have to walk up the hierarchy.

The rule has an exception. The FY 2026 guidelines allow both codes when the conditions are unrelated, and they say to query the provider when relatedness is unclear. Their example is F45.8, which has an Excludes1 for sleep-related teeth grinding (G47.63). For teeth grinding you code only one of the two. But F45.8 also covers psychogenic dysmenorrhea, which can coexist with G47.63. A hard block would reject a correct claim. That makes a hard Excludes1 block a bug.

The notes come from the tabular file in the CMS code tables release. It's XML, and each diag element nests its children and carries its excludes1 notes as free text with code references inside.

CODE = re.compile(r'\b([A-Z]\d[0-9A-Z](?:\.[0-9A-Z]{1,4})?)-?')

def load_excludes1(tabular_xml: str) -> dict[str, list[str]]:
    out = {}
    for diag in ET.parse(tabular_xml).iter('diag'):
        text = ' '.join(n.text or '' for n in diag.findall('excludes1/note'))
        refs = [norm(m) for m in CODE.findall(text

Range references in notes need expanding before the prefix match. I left that step out to keep this short. Each pair gets flagged once, along with the ancestor that carries the note. The output goes to the coder, never to an auto-reject.

Laterality and specificity need more than today's note

The laterality rules (I.B.13) are a set of branches, and each one is an if-statement:

If no bilateral code exists for a bilateral condition, code left and right separately.
If the side isn't documented, use the unspecified-side code.
If documentation conflicts on the side, query the provider.

The hard case is a bilateral condition treated in separate encounters, like cataract surgery on each eye. The first encounter gets the bilateral code. The second gets the unilateral code for the side that still has the condition, unless the first treatment didn't resolve it. The right answer depends on encounter history. A pipeline that only sees today's note will get this wrong and sound sure about it. That's why Encounter carries prior_codes. If you can't fill that field, flag the case.

Specificity works the same way. The guidelines require the highest specificity the documentation supports. When the model picks an unspecified code and a more specific sibling in the candidate list has supporting evidence, surface the sibling:

def more_specific(code: str, candidates: list[str], has_evidence, cs: CodeSet) -> list[str]:
    c = norm(code)
    if not cs.is_unspecified(c):        # e.g. tabular description says 'unspecified'
        return []
    parent = cs.parent(c)
    return [s for s in map(norm, candidates)
            if s != c and cs.parent(s) == parent
            and not cs.is_unspecified(s  has_evidences

This catches the benchmark's "generalized but correct" failure before the claim goes out.

Coding uncertain diagnoses: inpatient vs outpatient rules

Inpatient (Section II.H). Code "probable," "suspected," "likely" and similar terms as if the condition exists. This applies to short-term acute, long-term care and psychiatric inpatient admissions.
Outpatient (Section IV.H). Don't code them. Code the symptoms, signs or abnormal results instead.

The same sentence has two right answers depending on setting. Read enc.setting from the encounter record. Don't let the model guess it. A suspected-diagnosis span becomes a candidate only when the setting allows it. This also breaks benchmarks, because an evaluation that mixes settings will mark correct output as wrong.

Position and setting rules fall into the same bucket. Some Z codes can only be reported as the principal or first-listed diagnosis. Nonspecific ones like Z04.9 and Z13.9 have "little justification" inpatient. Once you know position and setting, each of these is a lookup.

Linking each ICD-10 code to evidence in the clinical note

Code assignment is based on the provider's diagnostic statement, not on clinical criteria (I.A.19). So every suggested code carries character offsets for the statement that supports it. For G47.63, that might be offsets 1842-1867, "grinds teeth during sleep."

Check that the offsets exist and that the text matches the note exactly. A code without a valid span doesn't count as a suggestion. It goes to the coder flagged as unsupported, and that catches outright invention.

Then check where the span sits. A sentence from a lab table or a vitals block passes the offset check, but it isn't a provider diagnostic statement. If the model reasoned its way to a diagnosis from a potassium value, the span will point at the lab table. Restrict valid evidence to provider-authored diagnostic sections, and flag anything outside them.

Laterality is the exception. The guidelines let it come from other clinicians' documentation when the provider didn't record it. Accept other clinicians' spans for laterality and nothing else.

The span also helps the coder, because it puts the supporting sentence right next to the code.

Measure principal and secondary diagnosis accuracy separately

Principal diagnosis follows the UHDDS definition: the condition "established after study to be chiefly responsible for occasioning the admission." Chapter-specific sequencing rules for obstetrics, poisoning, HIV and newborns take precedence, and you can check those in code too.

Secondary diagnoses have to coexist at admission, develop later, or affect treatment or length of stay. Each one must require evaluation, treatment, diagnostics, a longer stay, or more nursing care. Conditions from an earlier episode with no bearing on this stay don't count.

Picking the principal means finding the reason for admission. Picking secondaries is a relevance judgment across the whole chart. They fail in different ways, so report them separately.

Score each code in three buckets:

Exact match.
Right family, less specific.
Wrong.

I usually push for binary pass/fail scoring. Coding is the exception. The middle bucket is real, the benchmark shows it exists, and your specificity check should be shrinking it. On a claim it still counts as wrong.

Don't grade any of this with an LLM judge. Judges fail hardest on domain nuance, and code specificity is nothing but domain nuance. Grade against codes.

In production, the coder's override is your label. Every accept, change, delete and add is a labeled example. Store each one with the code's position and a reason. Track the override rate split by principal and secondary. That's your main production metric.

Don't treat overrides as gospel, though. The guidelines tell coders to query the provider when documentation conflicts, so some overrides are judgment calls on unclear notes. Sample overrides for a second-coder audit. This is the labeled dataset from Product Evals in Three Steps, with coders doing the labeling.

The model drafts, something accountable signs

It's the same rule I argued for in prior auth: the model drafts, and something deterministic or accountable signs. Build it that way and the 33.9% stops being scary, because the model's raw accuracy never reaches the payer. The coder's decision does.