Clinical AI 101:
How to Evaluate AI Tools Before They Touch Your Patients

Clinical AI is not a future technology. It is running in your hospital right now. Sepsis prediction algorithms are flagging patients on your wards. Radiology AI is reading chest X-rays before the radiologist does. Deterioration scoring models are generating alerts in your EHR. And in most cases, the clinical staff using these systems have no formal training in how to evaluate whether they work — or for whom.

That needs to change. This guide gives you the vocabulary and the framework to evaluate any clinical AI tool with appropriate rigour.

AI is already in your hospital

The AI tools already embedded in clinical settings include: early warning scores powered by machine learning (NEWS2, Epic's deterioration index), imaging AI for diabetic retinopathy screening and fracture detection, natural language processing that extracts structured data from clinical notes, and risk stratification tools that determine which patients to follow up proactively.

Each of these tools was trained on data, validated in specific settings, and deployed with assumptions about the patient populations it would encounter. Whether those assumptions match your patients is a clinical question — not just a technical one.

The black box problem

Many clinical AI tools are "black boxes" — they produce outputs (a risk score, a flag, a recommendation) without explaining their reasoning. This creates a genuine clinical governance problem. When an alert fires and you cannot understand why, you cannot meaningfully assess whether to act on it or override it.

Explainability is increasingly a regulatory expectation. The EU AI Act (effective 2024–2026) classifies most clinical AI as "high risk" and requires transparency about how systems reach their outputs. In the US, the FDA's SaMD framework has similar expectations for higher-risk AI-enabled devices.

Key questions to ask about any clinical AI tool

Before any AI tool is used in clinical care, the following questions should have clear answers:

• Training data: What population was this model trained on? Does it match your patient population by age, ethnicity, comorbidity profile, and care setting?
• Validation: Has it been validated externally — i.e., on data from institutions other than the one that built it? Internal validation alone is insufficient.
• Performance metrics: What is the sensitivity, specificity, and AUC? What is the false-positive rate in your setting? High sensitivity with poor specificity creates alert fatigue.
• Bias assessment: Has performance been evaluated across demographic subgroups? Tools that perform well overall may perform significantly worse in specific populations.
• FDA clearance: For tools making clinical recommendations in the US, is it cleared as a Software as a Medical Device (SaMD)? What class?

The FDA SaMD framework explained

Software as a Medical Device (SaMD) is the FDA's framework for software that performs a medical function without being part of a physical device. Clinical AI tools that influence clinical management decisions fall under SaMD.

The FDA classifies SaMD by risk level: Class I (lowest risk, general controls only), Class II (moderate risk, requires 510(k) clearance), and Class III (highest risk, requires premarket approval). An AI tool that reads ECGs and flags atrial fibrillation is Class II. An AI tool that autonomously makes treatment recommendations without physician review would be Class III.

What clearance means in practice

FDA clearance does not mean a tool is effective for your patients. It means it met a regulatory threshold for safety and performance in the submission context. Clearance should be the beginning of your evaluation, not the end of it.

Algorithmic bias in healthcare

Algorithmic bias in clinical AI is documented and consequential. The most cited example is a commercial risk stratification tool used across US health systems that was found to systematically underestimate illness severity in Black patients — because it used healthcare cost as a proxy for health need, and Black patients historically received less care for equivalent illness. The bias was in the training signal, not a technical flaw.

Pulse oximetry algorithms have been shown to overestimate oxygen saturation in patients with darker skin tones — a measurement-level bias that feeds into AI tools trained on that data. When your sepsis model uses SpO2 as an input feature, this upstream bias flows downstream.

What good AI governance looks like

Health systems with mature AI governance have: a defined process for evaluating new AI tools before deployment, ongoing monitoring of model performance after deployment (models degrade over time as patient populations and documentation practices change), clear accountability structures for clinical AI decisions, and mechanisms for clinical staff to raise concerns about AI tool behaviour.

Most health systems are not there yet. Clinicians who understand these frameworks are the ones who can advocate for them.

The Health AI & Clinical Decision Support track on Medinformics covers AI evaluation frameworks, FDA SaMD classification, algorithmic bias, and AI governance — giving clinicians and health IT professionals the vocabulary and tools to engage with clinical AI as informed participants.