LLMs Can Generate Clinical Quality Language. Don't Let Them Run It.

LLMs can draft CQL. Compile it to ELM, lint it, test it on clinician-signed FHIR patients, and keep the model out of the CDS runtime.

Clint JohnsonPublished 10 min read

Key takeaways

  • Use the LLM to draft CQL at design time, then run the approved FHIR Library, pinned by url and version, deterministically behind the CDS hook.
  • Compiling to ELM is necessary but not sufficient: track compile-pass rate, clinician-labeled test pass rate and human edits per artifact separately.
  • The agent can generate synthetic FHIR test patients, but a clinician must sign the expected results, and every correction becomes a permanent regression patient.

An LLM can write Clinical Quality Language. It should never execute it. Have a coding agent draft CQL from a guideline or measure spec. Let a compiler and a terminology server reject its mistakes. Prove the logic against synthetic FHIR patients whose expected results a clinician signed. Then ship the approved library as plain deterministic code behind your CDS hook.

Two terms first, for anyone new to them. CQL is HL7's human-readable language for clinical logic, such as measure criteria and decision rules. The cqframework translator compiles it to ELM (Expression Logical Model). ELM is the machine-readable form that CQL engines execute.

I've made this same split before, for HL7v2 to FHIR mapping, for ICD-10 coding and for CMS-0057-F prior auth. Use the model at design time, where a wrong answer becomes a failing build. Keep it off the live path, where a wrong answer reaches a patient chart. CQL is the cleanest case for that split I know. It is a real language with a real compiler, so you can check the model's work with code instead of vibes.

Why CQL authoring is the bottleneck

The hook plumbing is painful, but the problems are known. I covered them in CDS Hooks: What Nobody Tells You Before You Build. The slow part is turning a guideline into logic that is actually correct. The PathWISE authors call CQL authoring "expert-driven and costly". They point to the CAREPATH project, where domain experts co-authored 296 formally defined CDS rules.

That cost is why people reach for an LLM. Fine. Just don't stop at "the model wrote CQL and it looks right." Measurement-period errors and null errors both look fine in review.

The loop below is the one teams shipping coding agents converged on for ordinary software. The agent drafts. Deterministic checks reject. A human owns what ships. Boris Cherny, who created Claude Code, calls giving the agent a way to verify its work his most important tip. He estimates that the feedback loop gives 2-3x the quality. That is his own estimate, not a measured number, but the shape is right. Clinical logic needs one change to the loop: the answer key has to come from someone other than the model.

How to stop an LLM from hallucinating value set OIDs in CQL

The first rule is that the agent never types a code, an OID or a value set URL. An OID is a long dotted string of digits. A model can emit one that looks plausible and points to the wrong value set, or to nothing at all.

Instead, the agent writes a named token, such as valueset "Diabetes": '{{vs:diabetes}}'. A build step replaces it with the canonical URL from vs-manifest.json, a file a human approved.

If the agent invents a token that isn't in the manifest, the build fails and the error goes back to the agent. An unbound token means the guideline references a concept nobody has mapped yet. PathWISE handles this the same way. Concepts with no dictionary match are flagged REQUIRES_HUMAN_MAPPING instead of getting a guessed code. Copy that. A person maps the concept, and the manifest grows.

Compile, resolve, lint

The second gate is the compiler. Run the translator on every draft. Treat any error as a rejection, and pass the translator's messages back to the agent word for word. PathWISE is built around this. It replaced an LLM critic with the deterministic Java compiler, so a hallucinated SNOMED code, malformed CQL or an undefined identifier becomes a hard compile error. The authors call that a binary pass/fail "that no stochastic critic can match". I agree. Don't ask a second model whether the first model's CQL is valid. Ask the compiler.

Then check that every bound value set actually expands against VSAC or your local terminology server. Then lint for the mistakes a compiler happily accepts. All of it fits in one small file:

import re, subprocess, requests

TOKEN = re.compile(r"\{\{vs:([a-z0-9-]+)\}\}")
COMMENTS = re.compile(r"//[^\n]*|/\*.*?\*/", re.S)
LINT = {
    "wall clock in measure logic": re.compile(r"\b(Now|Today)\(\)"),
    "placeholder false define": re.compile(r'define\s+"[^"]+"\s*:\s*false\b'),
    "bare not, write is true/is false": re.compile(r"(?<!is )\bnot\s+(?!exists\b)"),
}

def bind(cql: str, manifest: dict) -> str:
    missing = set(TOKEN.findall(cql)) - manifest.keys()
    if missing:
        raise ValueError(f"unbound value sets: {sorted(missing)}")  # back to the agent
    return TOKEN.sub(lambda m: manifest[m.group(1)], cql)

def lint(cql: str) -> list[str]:
    # Run on the token version, before bind(). Comments are blanked out;
    # string literals are not parsed, so a quoted "not" still trips the rule.
    code = COMMENTS.sub(lambda m: "\n" * m.group().count("\n"), cql)
    hits = []
    for name, rx in LINT.items():
        for m in rx.finditer(code):
            line = code.count("\n", 0, m.start()) + 1
            hits.append(f"line {line}: {name}")
    return hits

def compile_errors(path: str) -> str:
    # cqframework translator CLI; command name and flags depend on your build
    r = subprocess.run(["cql-to-elm-cli", "--input", path], capture_output=True, text=True)
    return "" if r.returncode == 0 else r.stdout + r.stderr  # verbatim to the agent

def resolves(tx_base: str, url: str, auth=None) -> bool:
    # VSAC needs UMLS credentials; a local terminology server may not
    r = requests.get(f"{tx_base}/ValueSet/$expand",
                     params={"url": url, "count": 1}, auth=auth)
    if not r.ok:
        return False
    exp = r.json().get("expansion", {})
    return bool(exp.get("total") or exp.get("contains"))  # a 200 with zero codes fails

That last line matters. A value set can return HTTP 200 and still expand to nothing, for example because it's retired or has the wrong version. A criterion bound to an empty value set compiles fine and never matches anyone.

The lint rules are crude on purpose. A false positive costs the agent one rewrite. A miss costs a patient a wrong card. Here is what each one catches:

  • Now() or Today() in a measure library. Anchor measure logic to "Measurement Period", not the wall clock. Otherwise the same patient gets a different result depending on the day you run it.
  • A bare not. CQL uses three-valued logic. If the patient has no A1c, the comparison is null, and not of null is still null. Make the author write is true or is false so the intent is explicit.
  • Placeholder false defines. PathWISE gave nodes it couldn't compute a false placeholder so the libraries still compiled. That's reasonable in a research pipeline. In a library you ship, it's a criterion that silently never fires. Fail the build on it.

Synthetic FHIR test patients for CQL

The third gate checks behavior. For every criterion, the agent generates synthetic FHIR patients:

  • one that should qualify
  • one that shouldn't
  • one at each boundary date: events on the first and last day of the measurement period, and on the days just outside each
  • one patient missing the data entirely, because that's where the null bugs live

Building a valid FHIR Bundle with an Observation dated December 31 is tedious fixture work. It suits an agent. What the agent must not do is decide the right answer for that patient.

Who should write the expected results for CQL test cases

If the same agent writes the logic and the test that defines "done," both carry the same misreading of the guideline. Addy Osmani warns about exactly this: a test can pass while the feature is broken "because the agent and your test share a misunderstanding." A commenter on that post asks the obvious follow-up: who writes the test? For clinical logic, the answer is a clinician.

Keep the fixture and the expectation in separate files, and enforce who owns each:

# expectations/a1c-gap-boundary-last-day.yaml
patient: patients/a1c-gap-boundary-last-day.json  # agent-generated
criterion: Numerator
expected: true            # clinician-owned, never agent-written
signed_off_by: <clinician-id>
rationale: "A1c resulted on the last day of the measurement period counts as tested."

Put expectations/ under CODEOWNERS so only clinical reviewers can approve changes to it. The agent can propose a patient. It can't propose the answer. If its logic disagrees with a signed expectation, the logic is wrong until a clinician says the expectation is.

The runner is short. Load each expectation, evaluate its patient against the bound library, and diff:

import json, pathlib, requests, yaml

def evaluate(cr_base: str, library_id: str, bundle: dict, expression: str):
    # Library/$evaluate on a HAPI FHIR Clinical Reasoning server;
    # check which parameter names your server version accepts.
    pid = next(e["resource"]["id"] for e in bundle["entry"]
               if e["resource"]["resourceType"] == "Patient")
    body = {"resourceType": "Parameters", "parameter": [
        {"name": "subject", "valueString": f"Patient/{pid}"},
        {"name": "expression", "valueString": expression},
        {"name": "data", "resource": bundle}]}
    out = requests.post(f"{cr_base}/Library/{library_id}/$evaluate", json=body).json()
    return next((p.get("valueBoolean") for p in out.get("parameter", [])
                 if p["name"] == expression), None)  # null comes back as None

def run_suite(cr_base: str, library_id: str) -> list[str]:
    failures = []
    for path in sorted(pathlib.Path("expectations").glob("*.yaml")):
        exp = yaml.safe_load(path.read_text())
        bundle = json.loads(pathlib.Path(exp["patient"]).read_text())
        got = evaluate(cr_base, library_id, bundle, exp["criterion"])
        if got != exp["expected"]:
            failures.append(f"{path.name}: expected {exp['expected']}, got {got}")
    return failures

A null result comes back as None, so it fails both a true and a false expectation. That's deliberate. A criterion that evaluates to null for a real patient is a bug, even when the downstream logic happens to treat it as false. The failure list goes back to the agent the same way compiler errors do.

Should the LLM run inside the CQL engine?

No. Approved CQL ships as a versioned FHIR Library. The Library carries:

  • a canonical url and a version
  • a content array holding the CQL (text/cql) and the compiled ELM (application/elm+json) as attachments
  • relatedArtifact entries for each value set it depends on

The hook service never loads "the latest diabetes logic." It loads a pinned canonical from config, such as https://example.org/fhir/Library/DiabetesA1cGap|0.3.0, and logs that pin with every card it returns.

That's the whole point. The same patient data and the same library version give the same card, every time. When someone asks why a card fired, the answer is a library version and a set of resources, not a sampled model output. FDA's 2026 CDS guidance is silent on generative AI. With a deterministic library behind the hook, you don't have to bet on how that silence gets read.

The JAMIA Open "LLM-in-the-Loop" paper takes the opposite approach. The authors modified a CQL engine so it calls an LLM whenever CQL references unstructured FHIR resources. The LLM answers a natural-language question derived from the CQL with True or False, and that answer goes back into evaluation. Structured data still goes through normal CQL. The authors say the approach reduces hallucination, but the two local models they tested reached 72% and 93% accuracy. With the weaker model, roughly one answer in four would be wrong, inside logic that clinicians read as deterministic.

PathWISE makes the argument I'd make. It says stochastic outputs in patient-facing workflows fit poorly with regulated CDS because of hallucinated codes, non-reproducible results and opaque reasoning. Keep the model in authoring. If a criterion needs unstructured text, treat it as a separate, labeled extraction problem with its own eval. Don't hide it inside an expression that looks like logic.

How to measure LLM-generated CQL

Track three numbers per artifact, and keep them separate. This is the same discipline as any real eval suite, with a clinician as the labeler.

Compile-pass rate. This is the share of drafts that translate to ELM, pass lint, and expand every value set. You need it, but on its own it tells you almost nothing. PathWISE hit 100% syntactic compilation, yet only 18 of 25 synthetic patients (72%) routed correctly, with lung and breast at 60%. The authors say the seven misses weren't system failures, because each one raised a human-review flag. I'd count them as failed test cases. A review flag is a good safety net, but the logic still got the patient wrong.

Clinician-labeled test pass rate. This is the share of signed expectations the library satisfies. Gate releases on this one. Report it per criterion, because an aggregate hides the one population rule that's broken.

Human edits per artifact. This counts the changes a reviewer made before approval. The model doesn't learn from your corrections. Your prompts, manifest, lint rules and regression patients do. If this number doesn't fall, they aren't capturing the corrections, and the loop is just expensive autocomplete.

Turn every correction into a regression patient. If a clinician catches a December 31 bug, a December 31 patient with a signed expectation goes into the suite permanently. If you fix a null slip, a no-data patient goes in, and maybe a lint rule too. The next draft has to pass all of them, whatever model or prompt produced it. Over time the suite becomes the real spec, and reviewer time shifts from reading CQL line by line to reviewing evidence.

What stays out of the library

Not every guideline sentence belongs in CQL. The PathWISE authors say terminology still needs manual curation, that ambiguous criteria can't be resolved automatically, and that live clinical evaluation is still missing. They also note that CQL's expressiveness has limits compared with the full breadth of guideline logic.

My rule: anything the loop can't prove leaves the pipeline as an open question, not as code. An ambiguous criterion goes to a clinician, with the guideline text quoted. A free-text criterion goes to a separate extraction eval. An unmapped concept goes on the manifest backlog. None of them ship as a false placeholder that compiles and never fires.

A coding agent makes the draft cheap. The compiler, the terminology server and the clinician-signed patients make it trustworthy. Skip any of those, and all you've done is generate plausible clinical logic faster.

Questions people ask

Can an LLM generate CQL (Clinical Quality Language)?

Yes, a coding agent can draft CQL from a guideline or measure spec. Trust the draft only after it compiles to ELM, passes lint, expands its value sets against a terminology server, and passes synthetic test patients whose expected results a clinician signed.

How do you stop an LLM from hallucinating value set OIDs in CQL?

Never let the model type codes, OIDs or value set URLs. Have it write named tokens that a build step replaces with canonical URLs from a human-approved manifest. Fail the build if a token is unbound or a value set expands to nothing.

Should an LLM run inside the CQL engine at runtime?

No. The JAMIA Open LLM-in-the-Loop approach reported 72% and 93% accuracy for its two local models, which is not deterministic logic. Keep the model in authoring, and run approved CQL as a versioned FHIR Library so the same data and library version always produce the same card.

Is compiling CQL to ELM enough to know the logic is correct?

No. PathWISE reached 100% syntactic compilation but routed only 18 of 25 synthetic patients correctly. Measure the clinician-labeled test pass rate separately from the compile-pass rate, and gate releases on it.