AI Agents in Medicine Need Oversight, Not Blind Trust

AI agents looked useful until the workflow broke

Last Tuesday, I watched a vendor demo that should have been boring. A “medical agent” was supposed to draft prior authorization language, reconcile medication lists, and flag discharge follow-up gaps while the clinician stayed in control. On paper it looked elegant. In the room, it missed a contraindication buried in the chart, overconfidently summarized the wrong problem list, and needed three clarifications before it could produce a sentence I would have been comfortable signing my name to.

I used to think the main question was whether agents were accurate enough. Then I spent enough time around real workflows to see the deeper issue. Accuracy is only the first gate. The harder problem is system design: who can delegate, what the agent can touch, how the output is reviewed, and what happens when the model is confidently wrong at 4:50 p.m. on a Friday.

That is why I do not see agentic AI as a tool category. I see it as a structural change in medicine, closer to the difference between academic medicine and community medicine than to the difference between one software product and another. Academic centers absorb complexity with layers of supervision, informatics, pharmacy, risk, and specialist backup. Community settings often need simpler, clearer, more fail-safe processes. Agents will force the same split. A hospital that wants autonomy will need an architecture for autonomy. A smaller system may need constrained agents that behave more like decision support than staff.

What agents can do, and what they should never do alone

The strongest near-term value is operational. In Developing an AI-Powered Automation Framework to Streamline IT Support Tasks in Public Sector Organizations, the authors describe automation that reduces service-delivery friction by handling repetitive ticketing and access tasks. The same logic applies in hospitals, where agents can prefill referral letters, route faxes, summarize messages, and draft coding support for review. Every one of those tasks burns clinician time today. Every one of them is also a place where a small error can become a downstream safety issue if the output is copied into the chart without human review.

I have seen that failure mode up close. A patient’s discharge plan looked “complete” in a draft generated by an agent, yet the follow-up appointment was scheduled with the wrong specialty. Nobody had intended harm. The agent simply made a plausible but wrong inference. That is the danger with probabilistic systems. They do not need malicious intent to create real consequences.

Clinical use is more promising and more constrained. In radiology and pathology, for example, the literature keeps pointing toward assistive workflows rather than unsupervised replacement. In Explainability and Trust in Deep Learning for Cancer Imaging, the 2026 review argues that translational barriers remain stubborn because explainability, calibration, and clinical fit are still mismatched. That is the right lens for agents too. An agent that can draft a radiology follow-up note or organize a worklist is useful. An agent that decides independently whether a lesion is benign is a different risk class entirely.

The most honest mental model I have found is the brilliant resident model. Residents are valuable because they can synthesize, move fast, and do work that would otherwise bottleneck the team. They are also supervised because speed without judgment is a liability. I want agents in that lane. Not unsupervised. Not credentialed by marketing language. Supervised.

And I mean supervised in the operational sense, not just the ceremonial sense. The supervising clinician must be able to see the source data, review the intermediate reasoning or structured evidence, and stop the action before it reaches the patient. If an agent can place an order, message a patient, or change a coding recommendation, it needs a bounded toolset and an audit trail that a real compliance officer can inspect.

Medicine needs agent governance, not just model governance

The regulatory language is starting to catch up. The FDA’s framework for software as a medical device, especially the distinction among 510(k), De Novo, and PMA pathways, still matters because agent behavior can drift into functions that look like clinical decision making. The WHO’s guidance on responsible AI in health care and NIST’s AI Risk Management Framework both emphasize risk identification, monitoring, and human oversight. That is where hospital boards should focus. Not on whether a demo looked impressive. On whether the system has measurable guardrails.

Security is part of the same conversation. In Caging the Agents: A Zero Trust Security Architecture for Autonomous AI in Healthcare, the authors argue for least-privilege controls, segmented access, and containment for tool use. That sounds abstract until you map it to real life. An agent that can read everything, call everything, and write everywhere is a breach waiting to happen. I would not let a junior staff member with no supervision roam across every system in the hospital. I would not let an agent do it either.

That is one of my non-negotiables. What I would not do is deploy an agent with open-ended chart access and write permissions just because it can summarize well. Summary skill does not equal clinical authority. If a system cannot prove why it touched a chart element, who approved the action, and how the action was bounded, I do not want it near patient-facing workflows.

There is also a labor question hiding inside the technology question. Agents will not simply “save time.” They will redistribute work. Some tasks will vanish. Others will be compressed into fewer, more judgment-heavy moments. That is exactly why the academic-versus-community analogy matters. In high-resource settings, agents may sit inside a larger supervisory web. In lower-resource settings, the same agent may become the only layer between a patient message and a clinician response. The risk is not theoretical. The implementation context defines the harm profile.

In Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems, the authors describe human oversight as a structured function, not a vague ideal. That is the right direction. Oversight should include pre-use validation, ongoing spot checks, escalation triggers, and clear authority to override. In medicine, those are not bureaucratic extras. They are the difference between useful assistance and unsafe delegation.

What I think medicine will look like in five years

I think the best hospitals will build agent lanes the way they built ICU lanes, ward lanes, and outpatient lanes. Different risk tolerance. Different staffing. Different rules. The agent that drafts an inbox response about a prior authorization should not have the same permissions as the agent that supports sepsis triage. The agent that helps with operational ticketing should not be able to silently update the chart. Simple idea. Hard to execute. Necessary.

In From Clinical Intent to Clinical Model, autonomous coding agents are framed as clinician-driven development tools, which is exactly the direction I favor. Clinicians should define the intent. Engineers should encode the safeguards. Compliance should define the boundaries. That division of labor protects patients better than pretending the model can be trusted because it sounds fluent.

I also think the field is underestimating how much implementation culture will matter. A hospital that already tolerates sloppy handoffs will not magically become safe because it buys an agent. A hospital that documents escalation, uses checklists, and treats near-misses as data will do much better. The technology amplifies the culture already there. That is the part vendors rarely say out loud.

The right comparison is not “human versus machine.” It is “care team with or without disciplined supervision.” The agent should reduce clerical drag, surface missed signals, and extend scarce expertise. It should also be stoppable, reviewable, and replaceable. When it fails, the clinician must still be the adult in the room.

Three weeks ago, after that vendor demo, I told the team something I still believe: the future of medicine will not belong to the most autonomous system, but to the most governable one. The patient in that room never asked for autonomy. She asked for safe, timely help. That is the standard I care about. If agents can meet it, I want them in the hospital. If they cannot, they can stay in the lab.

For my broader thinking on physician-led AI governance, I keep returning to the same principle I describe at Dr. Sina Bari, MD, a Stanford-trained physician focused on responsible AI adoption, and it is simple: capability without control is just risk with better marketing.

FAQ

What happens if a hospital deploys an AI agent without clinician oversight?

It will eventually make a plausible but wrong recommendation, and that error can propagate quickly through messaging, documentation, ordering, or follow-up workflows. The safest pattern is human review for any action that touches diagnosis, treatment, or patient communication.

How is an AI agent different from ordinary clinical decision support?

An agent can take actions across multiple tools, not just display a suggestion. That increases both utility and risk, because one failure can affect several downstream systems instead of a single screen.

What is Dr. Sina Bari’s approach to agentic AI in hospitals?

I would use agents for bounded, reviewable tasks first, then expand only after auditability, access control, and escalation paths are proven. The goal is to make the agent act like a supervised resident, not a standalone clinician.

Which hospital tasks are the best early uses for AI agents?

Administrative and coordination tasks are the lowest-risk place to start: inbox triage, referral drafting, coding support, worklist cleanup, and internal ticket routing. Those tasks still need review, but they are easier to contain than autonomous clinical decisions.

Why do clinicians worry about agentic AI if the model is accurate most of the time?

Because medicine is punished by the tails, not the average. A system that is right 95% of the time can still cause serious harm if the 5% failure lands in medication reconciliation, discharge instructions, or urgent escalation.