The AI Audit Checklist

Start with the outcome, not the model. Most failed AI work skips this.

• The problem is written as a business outcome (time saved, revenue, risk reduced) — not "add AI"
• There is a way to measure success that a non-engineer would agree with
• You can name the one workflow AI would change first, and who feels it
• The cost of the problem is bigger than the cost of solving it

AI is only as good as the data it can reach. This is where audits find the most surprises.

• The data the model needs actually exists and is accessible (not trapped in a PDF or someone’s head)
• You know its quality, coverage, and how often it’s wrong
• Sensitive data (PII/PHI) has a clear boundary the model will not cross
• You can produce a small, real evaluation set from this data today

The most valuable audit output is often "don’t build this."

• For each idea, there is an honest call: build, buy an existing tool, or skip
• A hosted API/model was considered before any custom training
• The simplest thing that could work was tried first
• You are not rebuilding something a $20/month tool already does

If you can’t measure whether it works, you can’t ship it responsibly.

• There is a task-specific eval that catches regressions before users do
• Failure modes are known, and the worst case is acceptable or contained
• A human is in the loop where being wrong is expensive
• You can explain, at a high level, why the system produced a given answer

A demo that works once is not a system. This is what separates the two.

• Per-request cost is known and survives real usage volume
• Latency is acceptable for where it sits in the workflow
• There is a plan for monitoring, drift, and model/vendor changes
• Someone owns it after launch