what actually works once the demo is over

Applied LLMs & AI engineering

Building with language models in production: what to build, how to test it, and what an honest audit turns up. I've shipped language models into real products at Clarity Health Project and eSpiral Health, and built MedScrub to make them safe to use on healthcare data. These are the notes I keep coming back to.

start here: 3-part series

Shipping LLMs in production

  1. Table Stakes for Pragmatic Development Using LLMs8 min read
  2. What an AI Audit Actually Finds4 min read
  3. Your Support Team Is Claude Code on a Timer7 min read

All 49 articles

  1. 8 min readFDA's 2026 Clinical Decision Support Guidance Didn't Bless Generative AIFDA's 2026 clinical decision support guidance is silent on generative AI. What changed, and how to build LLM CDS clinicians can independently review.
  2. 7 min readYour Support Team Is Claude Code on a TimerAn L2 support engineer that's just Claude Code on a /loop, with all its state living in Slack reactions. No app, no database, no deploy — and it only pings you when it's real.
  3. 4 min readWhat an AI Audit Actually FindsMost teams asking for AI don't have a model problem. They have a "which problem is even worth it" problem. Here's what a real AI audit surfaces — and why "don't build this" is often the most valuable line in the report.
  4. 24 min readInside a Production Voice Agent: How the Stack Actually ShipsProduction voice-AI has converged on a pattern: graph-based conversations, separated decision and response prompts, synthetic-call regression testing, and per-component latency budgets. Why the stack looks the way it does — and what most teams are still missing.
  5. 19 min readPrompt Engineering Didn't Die. It Got Unrolled.Everyone keeps announcing the death of prompt engineering. They are describing the symptom, not the shift. The loops you used to run by hand — refine, retry, verify, learn — moved out of your head and into infrastructure. Four of them, simultaneously.
  6. 8 min readHealthcare AI Will Be Won by Verticals. The Recipe Has Been Around for a Decade.Most healthcare AI companies are failing for the same reason. The ones winning are running the same playbook: one Palantir figured out before anyone called it AI. Forward-deployed engineer. Ontology. Integrations. Then AI tooling.
  7. 7 min readPenpal: Dispatch Tool Today, RPG Interface TomorrowI built a tool that turns GitHub issues into pull requests using a three-agent pipeline. That's the boring part. The interesting part is what happens when you stop thinking about AI agents as productivity tools and start thinking about them as a workforce, and build a world for them to live in.
  8. 8 min readProduct Evals in Three Steps (That You'll Actually Do)Most teams skip evals because the process feels overwhelming. The three steps that make eval-driven development achievable: label a small dataset, calibrate an LLM evaluator to human judgment, then iterate configs against the harness.
  9. 8 min readTable Stakes for Pragmatic Development Using LLMsUpdated for 2026: lessons from two years using Claude Code in production. Context engineering, real eval frameworks, model economics, and agent workflows. What works.
  10. 8 min readThe Open-Weight LLM Landscape in 2026: What Engineers Actually Need to KnowThe open-weight ecosystem has matured faster than most engineers realize. MoE proliferation, hybrid attention, and extended context windows are changing what's deployable on-premise. That matters more than ever for healthcare AI.
  11. 7 min readSoftware 2.0 Is Here and It Changed How I Think About ProgrammingIn 2017, Andrej Karpathy argued that neural networks would replace explicit logic as the dominant programming paradigm. Nine years later, that prediction has fully landed. The implications for how we build software are bigger than most engineers want to acknowledge.
  12. 7 min readRAG Isn't Dead. You're Just Using It Wrong.The 'RAG is dead' narrative is wrong, but it's wrong in an interesting way. What kills LLM context quality in production, and what to do about it.
  13. 8 min readFine-Tuning a 70B Model on a Consumer GPU: The Q-LoRA Practical GuideQ-LoRA + SFTTrainer + Flash Attention v2 means you can fine-tune a 70B parameter model on 24GB of VRAM. What that looks like end-to-end, what it costs in quality, and when to just use the API instead.
  14. 8 min readThe LLM Year in Review: What Actually Mattered in 2025 (And What Was Noise)The prediction was: bigger models win. The reality was: DeepSeek R1 rewrote the rules in January and nothing was the same after that. What 2025 taught us about reasoning, inference-time compute, and the economics of intelligence.
  15. 8 min readWhat the Teams Actually Shipping Coding Agents Have Figured OutCoding agents are the most economically viable AI in production today. The patterns Devin, Cline, Amp, and others converged on, and what they mean for anyone building or using agents seriously.
  16. 7 min readStop Shipping Features: Why AI Products Need an Experiment Mindset12 features shipped in a quarter. Zero meaningful metrics moved. AI products aren't software projects. The roadmap is a hypothesis board, not a delivery schedule, and treating it otherwise is expensive.
  17. 8 min readBeyond Chunks: Why Faceted Context Is the Future of RAGChunk-based RAG returns results. Faceted context gives agents peripheral vision: an understanding of the information landscape that lets them navigate rather than just consume. What that looks like in a domain where getting it wrong matters.
  18. 7 min readContext Engineering: The Skill That Replaced Prompt EngineeringPrompt engineering is a symptom, not a skill. The engineers shipping better agents aren't writing better instructions. They're designing better information spaces. Context engineering is the meta-skill no one teaches.
  19. 8 min readEvery Service Is Going to Need an MCP LayerREST APIs were designed for humans calling services through UIs. AI agents are not humans. What breaks when you expose your existing APIs to agents, and what the right architecture looks like.
  20. 7 min readWhen English Became a Programming Languagev0 just proved that English plus AI can replace traditional web development for most apps. That changes what it means to be a developer. An honest take on what shifts, what doesn't, and what to do about it.
  21. 8 min readFine-Tuning LLMs Without the RLHF Headache: The DPO ApproachRLHF is the right idea with the wrong implementation cost for most teams. DPO flips the math. How to align a healthcare AI model on clinician feedback without burning a month on reward model engineering.
  22. 7 min readYou Are No Longer a Coder: The Shift from Execution to DirectionI stopped writing most of my own code. What changed, what I delegated to AI, what I found it can't do, and why the hardest part of the transition had nothing to do with technology.
  23. 7 min readContext Rot: The Silent Performance Killer in Your LLM ApplicationYour LLM system works great in demos and degrades in production. The culprit is almost never the model. It's what you're feeding it. How to diagnose and fix context rot before it kills your product.
  24. 8 min readWhy Your AI Gets Smarter When You Let It Think LongerTest-time compute is the most underused lever in production AI right now. What chain-of-thought, best-of-N sampling, and process reward models mean for practitioners, and when to use them instead of grabbing a bigger model.
  25. 8 min readWhat the AI Tool Ecosystem Is Actually Telling YouTool selection in healthcare AI is a compliance problem, not a productivity one. Most AI tools that sounded great in 2023 are dead or deprecated. The framework I use for betting on the right ones.
  26. 7 min readWhat Autonomous Vehicles Taught Me About Multi-Agent AI DesignBAIR researchers discovered that just 5% autonomous vehicle penetration can smooth all highway traffic, with no central coordination. That finding reshapes how I think about building multi-agent AI systems.
  27. 9 min readVision + Language: How Multimodal LLMs Actually Work (And When to Use Them)Multimodal LLMs integrate vision through two fundamentally different architectures. Knowing which one you need, and why, shapes every other technical choice in your build.
  28. 8 min readThe Self-Healing Stack: What AI-Native Infrastructure Actually MeansThe AI Cloud vision, where infrastructure monitors, optimizes, and repairs itself, is compelling. Some of it exists today. Most doesn't yet. What self-healing infrastructure looks like in practice, and what engineers should be doing to prepare.
  29. 7 min readThe Tools I Dropped When AI Changed My Development WorkflowAI coding assistants forced a full rethink of every layer of the dev stack. What I dropped, what I added, and the principle behind the restructuring.
  30. 8 min readFrom GPT-2 to DeepSeek: The Architectural Changes That Actually MatteredI've been reading ML papers for 10 years. Most don't matter. These architectural choices did. RoPE, GQA, SwiGLU: each one solved a real scaling problem. What to look for when a new model claims 'better architecture.'
  31. 8 min readBuilding a GenAI Platform That Doesn't Collapse Under Its Own WeightGenAI platforms don't fail because the models are bad. They fail because teams build everything at once. A practitioner's guide to layered architecture, from the minimal production-ready core to healthcare-grade guardrails.
  32. 8 min readThe GenAI Strategy Question You're Not Asking (But Should Be)Everyone asks 'how should we use GenAI?' The honest answer requires a harder question: does AI's unique capability create new value here, or is it just a more expensive way to do something that already worked? A practitioner's framework for getting this right in healthcare.
  33. 7 min readEvery Failed AI Product Has the Same Root CauseThe same failure pattern shows up everywhere: teams shipping fast and iterating on vibes instead of building systematic evaluation. Evals aren't a nice-to-have. They're the core competency of any serious AI product team.
  34. 8 min readThe 6 Ways I've Watched GenAI Projects Fail (And How to Avoid Them)GenAI projects in healthcare go sideways in predictable ways, sometimes with real patient consequences. Six failure modes that come up over and over again, and what to do instead.
  35. 8 min readWhen to Look Beyond Standard LLMs (And When to Stop Overthinking It)Most teams should use a frontier API and move on. There are specific situations where alternative architectures matter: extreme latency, long-context scale, cost walls, privacy constraints. The decision framework.
  36. 7 min readWhen Recommendations Meet Language: The LLM-RecSys ConvergenceMost AI stacks treat the recommendation engine and the language model as two separate systems that hand off to each other. A new class of hybrid models eliminates that seam. The implications for domain-specific AI are significant.
  37. 9 min readTrading Speed for Quality: A Practical Guide to Inference-Time ScalingInference-time scaling lets you tune the latency-quality tradeoff at runtime instead of at training time. When to use Best-of-N sampling, beam search, iterative refinement, or one-shot generation, with real examples from clinical AI.
  38. 9 min readInside the Black Box: What Mechanistic Interpretability Means for BuildersHealthcare AI requires explainability. 'The model said so' is not a clinical rationale. Mechanistic interpretability is the research field trying to change that. What it offers practitioners today, where the gap is, and what to do in the meantime.
  39. 9 min readHow to Actually Test If Your AI Will Say Something DangerousMost teams treat jailbreak testing as a vibe check. StrongREJECT achieves 0.90 Spearman correlation with human judgment. Automated safety evaluation is real, and there's no excuse not to build it into your pipeline.
  40. 8 min readThe Attack Your LLM App Is Definitely Vulnerable ToPrompt injection is the #1 OWASP threat to LLM applications and most teams aren't taking it seriously. What the attack looks like, why it's hard to stop, and how to harden your system.
  41. 8 min readThe Honest Guide to LLM Evals: What Actually WorksMost teams skip real evals and wonder why their AI products degrade in production. The framework that holds up: from 30-minute manual reviews to binary scoring to knowing when your eval suite is finally doing its job.
  42. 7 min readWhy Your LLM Evaluator Is Lying to YouLLM-as-judge evaluators feel like quality assurance but behave like rubber stamps. They fail hardest on the outputs that matter most: edge cases, safety-critical errors, domain-specific nuance. What to do instead.
  43. 6 min readWhy I Stopped Using RAG for Coding Agents (And What I Do Instead)The instinct when building a coding agent is 'I need RAG to handle large codebases.' The better instinct is giving the agent tools to explore code the way a senior engineer would: reading files, following imports, tracing execution.
  44. 8 min readThe Neural Net Training Recipe That Actually WorksI spent months chasing architecture fixes when my real problem was bad debugging hygiene. The training recipe that works: start simple, visualize everything, tune last. The unglamorous discipline that separates working models from expensive experiments.
  45. 9 min readYou Don't Need GPT-4 for That: Small Models and Edge AgentsFrontier models aren't required for agentic function calling. For healthcare AI, assuming they are can also be a compliance liability. When a fine-tuned 7B model is the right architecture, and when it isn't.
  46. 8 min readMulti-Agent Orchestration in Practice: What I Learned Building Parallel Agent SystemsThe orchestrator/worker pattern is the key mental model for multi-agent systems. How to structure orchestrators, spawn and manage workers, aggregate results, and avoid the coordination failures that sink most implementations.
  47. 8 min readWhat It Actually Takes to Build a Real LLM AgentEveryone's talking about agents. Few have shipped one that works in production. The failure modes, memory tradeoffs, and tool design decisions the architecture papers skip.
  48. 8 min readThree Hard Truths About LLMs in Production Nobody Warned Me AboutTwelve years in ML and I still got burned. Stochasticity is a systems design problem. Your target model will be deprecated. Silent failures are worse than loud ones. What production healthcare AI taught me.
  49. 6 min readWhat AI Agents Actually Are (And What They Can't Do Yet)Everyone's building 'agents.' Most are just APIs with a system prompt. The precise definition, the components that matter, the failure modes I've hit, and how to pick the right pattern for your problem.

Common questions

Does FDA's 2026 clinical decision support guidance cover generative AI?

No. The guidance text is silent on generative AI, as the 2022 version was, and says nothing about clinician-facing chatbots. CDRH released a separate discussion paper on generative AI-enabled devices on August 18, 2026, with comments due October 19, 2026 under docket FDA-2026-N-7874.

Is an LLM clinical decision support tool a medical device?

FDA judges software by the role its output plays, not how the output is generated, so an LLM faces the same four criteria as a rules engine. Inputs count too. Piping a 14-day glucose trace into the context window feeds the model patterns and changes your regulatory posture.

Are risk scores allowed in non-device CDS after the 2026 guidance?

The 2026 revision removed the 2022 language saying a risk score disqualifies software from non-device CDS. Clint labels LLM confidence "model self-assessment, not a calibrated probability" and shows "unknown" when the model gives none. A real probability needs calibration against labeled data first.

Are time-critical AI alerts considered a medical device by FDA?

FDA says software for critical, time-sensitive decisions generally fails criterion 4 because clinicians lack time to review the basis. Its device examples include stroke and sepsis alarms. The guidance doesn't define "time-critical," so Clint keeps LLM output in chart prep, note review and research.