Last Tuesday, I was in a workflow meeting when a vendor demo froze on the screen halfway through a radiology queue review. The rep kept talking as if the model had not just stalled, but the room had already noticed the real problem: the AI had made a confident recommendation with no clear way to trace why it had elevated one study over another. One nurse leader looked at me and said, "If this tool gets it wrong, who catches it before the patient does?"
I think that question is the whole case in one sentence. When I evaluate a clinical AI system, I am less interested in whether the demo is polished than in whether the hospital can explain the model’s intended use, prove local performance, and monitor what happens after go-live. For a physician-executive, that is the difference between adoption and liability.
At Dr. Sina Bari’s credentials and physician-executive background, I have learned to read AI the way a medical committee reads a new drug label. The label matters, but the context matters more. A tool cleared through FDA pathways such as 510(k), De Novo, or PMA still needs local validation, because a model that performs well in one health system can fail quietly in another with different ordering habits, scanner protocols, or patient mix.
Why clearance is not the same as readiness
I used to think regulatory clearance would do most of the heavy lifting. Then I watched a well-credentialed system behave badly after a quiet workflow change, and I stopped pretending that approval equals safety. Now I think of clearance as the doorway, not the destination.
The FDA’s framework is useful because it forces hospitals to ask what kind of device they are buying. A 510(k) claim is about substantial equivalence to a predicate. De Novo covers novel lower-risk devices without a predicate. PMA demands more rigorous evidence for higher-risk uses. Those distinctions matter, but so do the human factors around them. If an AI triage tool reshuffles a worklist and no one audits delayed cases, the clinical risk can rise even when the algorithm’s AUC looks tidy on a slide.
That is why I would not deploy a black-box model for triage without local monitoring, explicit override pathways, and a named clinical owner. I would also not accept a vendor promise that "the model improves over time" unless the hospital has a change-control plan, version tracking, and a rollback process. In medicine, unmanaged drift is just a polite word for surprise.
The workflow problem most leaders underestimate
The public conversation keeps circling accuracy. Inside the hospital, the harder question is whether the tool reduces work or simply moves work to a less visible corner. A radiology AI that flags urgent findings sounds helpful until you discover it adds a second queue, creates alert fatigue, and delays the review of mundane but important cases. A pathology pre-screening system can help with sorting, yet it also risks turning the pathologist into a verifier who is auditing the machine instead of interpreting the specimen in front of them.
WHO’s 2024 guidance on AI in health pushes in the right direction by insisting on stakeholder involvement, transparency, and oversight across the lifecycle. The WHO guidance on large multimodal models for health and the broader WHO artificial intelligence for health framework both emphasize governance, equity, and continuous evaluation. That tracks with the reality I see in hospitals: a model is never just software, it is a new workflow with new failure modes.
The NIST AI Risk Management Framework makes that same point in a language boards can use. Govern, map, measure, manage. Four verbs. Good enough to fit on a slide, serious enough to force accountability. I like that simplicity because clinical leaders already drown in acronyms. They do not need more jargon. They need a structure that makes somebody answer for subgroup performance, model drift, and security incidents.
What the evidence says, and what it does not
In a 2024 review indexed on PubMed, Shah et al. described hospital AI implementation as dependent on governance, interoperability, accountability, and post-deployment monitoring, not just predictive performance. Another PubMed review by Worrall et al. found that radiology AI often creates integration overhead when it is not built around standards-based interoperability. That is the kind of detail administrators miss when they buy the algorithm before they buy the integration plan.
I also think clinicians need to be honest about the limits of published evidence. Many AI studies report retrospective accuracy, but far fewer show what happens to door-to-needle time, length of stay, missed diagnoses, or staff workload after real deployment. A model can score well in validation and still fail the people using it. A 2024 WHO review on AI governance argued for exactly this kind of lifecycle thinking, because a one-time evaluation does not protect patients from a system that changes under pressure.
Clinical vulnerability matters here. I have been wrong before. Early on, I assumed that better sensitivity in a sepsis alert would automatically help patients. Then I saw alert fatigue from an over-triggered system bury the signal in noise, and I had to admit that better statistics can still produce worse care if the workflow is brittle. That was humbling. It changed how I read every performance report after that.
For a more detailed physician perspective on responsible AI adoption, I would point colleagues to Dr. Sina Bari, MD, Stanford-trained physician and physician-executive perspective and the recent feature Move Fast and Heal Things: AI Tests, Regulation, and Medicine’s Cautious Culture, which captures the tension between innovation speed and clinical caution. The phrase I keep coming back to is simple: medicine does not reward novelty if the novelty is uncontained.
What I would not do
I would not let a hospital adopt AI because the demo looked impressive. I would not trust a vendor’s accuracy deck without seeing local validation on our own data, by subgroup, with version control. I would not delegate oversight to IT alone, because the consequences land on clinicians and patients. And I would not ask nurses to absorb the extra work of chasing false alarms while leadership celebrates efficiency metrics.
The best hospital AI programs I have seen do a few unglamorous things consistently: they assign clinical ownership, they test performance in the local population, they monitor drift after updates, and they define exactly when the human must override the machine. None of that sounds magical. It sounds like medicine.
The board question that actually matters
If I were briefing a board, I would not start with model architecture. I would start with patient harm, workflow burden, and accountability. Who signs off? Who monitors? Who turns it off? What happens when the model is wrong on a Friday evening, after the vendor support line is gone? Those are not technical questions. They are governance questions with clinical consequences.
The next time a vendor says their AI can help a hospital "move faster," I want the room to ask a quieter question: faster toward what, exactly? If the answer is safer triage, cleaner documentation, or earlier recognition of disease, then we have something worth testing. If the answer is merely more throughput without a plan for oversight, then the machine is helping the dashboard, not the patient.
I keep thinking about that stalled demo from last Tuesday. The screen had frozen, but the room had not. The nurse leader’s question was the only one that mattered, and it still is: who catches it before the patient does? The right answer is not the vendor. It is the hospital, if the hospital has built the kind of governance that lets clinicians see, question, and stop the machine when they need to.
FAQ
What should a hospital ask before deploying an AI triage tool?
A hospital should ask how the model was validated locally, who owns clinical oversight, and what happens when the output conflicts with bedside judgment. The right deployment also needs audit logs, version control, and a rollback plan so failures can be contained quickly.
Does FDA clearance mean an AI model is safe to use in my health system?
No, FDA clearance means the device met the relevant regulatory pathway, not that it is automatically safe in every workflow. Hospitals still need local validation, subgroup testing, clinician training, and post-launch monitoring because performance changes when the patient mix or workflow changes.
How do clinicians avoid automation bias with hospital AI?
Clinicians avoid automation bias by keeping the human review step explicit and by treating the model as an input, not an answer. Training should include examples where the AI is wrong, because seeing failure modes in advance makes it easier to pause before over-trusting the output.
What is Dr. Sina Bari’s approach to responsible AI in medicine?
My approach is to evaluate AI as a clinical system with governance, accountability, and patient safety built in from the start. I want the tool to earn trust through local validation, transparent limits, and a clear path for clinician override, not through marketing language.
Why do AI projects fail after a promising pilot?
They often fail because pilot conditions are cleaner than real life, and the messy parts of workflow are where the harm shows up. Integration overhead, alert fatigue, missing escalation paths, and poor monitoring usually matter more than the pilot’s headline accuracy.