Last Tuesday, I watched a nurse practitioner in a busy clinic scroll through a draft assessment an AI agent had assembled from the chart, the patient message, the medication list, and the last two visits. It was fast, tidy, and dangerously confident. She looked up and said, “It got the story right, but I still don’t know whether I trust the plan.”
AI agents can help medicine, but only if hospitals build them as supervised team members with narrow authority, not as unsupervised decision-makers. The right model is closer to a brilliant resident on call than a vending machine for answers, which means oversight, escalation, and accountability have to be designed into the workflow from day one.
I think that sentence should be pinned above every hospital AI committee. In my experience evaluating clinical tools, the problem is rarely whether an agent can produce a plausible answer. The problem is whether the organization can absorb its mistakes without letting them reach a patient.
That distinction matters because agents are different from the older wave of static models. A classifier can flag a chest radiograph. An agent can read the note, pull prior imaging, draft a message, place a task, and suggest next steps. That kind of chained reasoning is useful, but it also creates a longer failure surface. One bad assumption propagates. One missed contraindication becomes an order. One overconfident summary becomes clinical momentum.
For readers who want my physician profile and background, I keep that public on Dr. Sina Bari’s clinical and professional background page.
The case for agents is real, and so is the cost of pretending they are magic
I used to think the main question was whether agents would be accurate enough. Then I spent enough time around clinical workflows to see the deeper issue. Accuracy is only one layer. The harder question is operational fit. Can the tool tolerate ambiguity, hand off uncertainty, and stop at the right moment?
The FDA already signals the right regulatory posture for AI-enabled medical software. On its AI in Software as a Medical Device guidance page, the agency frames AI/ML devices through established pathways like 510(k), De Novo, and PMA, and emphasizes lifecycle management rather than one-time approval. That is the correct mindset for agents too. A model that changes behavior through data, prompts, tools, or context cannot be treated as a frozen artifact.
Hospitals should read that as a warning. Every agent needs a permission boundary. Every tool call needs logging. Every recommendation needs a human owner. If none of that exists, then the agent is not helping clinicians, it is borrowing their license by implication.
There is a useful parallel here to academic medicine and community medicine. Academic centers can absorb friction. They have layers of expertise, conferences, consults, and redundancy. Community settings often run lean, with fewer specialists and less margin for ambiguity. Agentic AI will probably widen that gap before it narrows it, because high-resource systems can supervise agents, while low-resource systems may be tempted to let them run because the staffing pressure is relentless. That is where risk compounds.
The brilliant resident model is the only honest way to think about agents
In my own practice, I think about an AI agent the way I think about a very smart resident on a difficult night. Useful. Fast. Sometimes better read than the attending on a narrow question. Also capable of being misleading in exactly the kind of plausible way that creates trouble. A resident can draft the plan, but someone senior still signs off.
The phrase “brilliant resident” is not an insult. It is the highest compliment I know that still includes supervision. Medicine already has a language for this. We do not hand the most complex decisions to the newest person in the room and hope the room is lucky. We create attending oversight, escalation rules, and limits on autonomy. AI agents need the same architecture.
That view is supported by recent work on human oversight and trustworthy agentic AI. The 2026 arXiv framework Keeping an Eye on AI: A Framework for Effective Human Oversight of AI Systems argues for explicit oversight roles and intervention pathways, while the broader survey Towards trustworthy agentic AI centers safety, robustness, privacy, and system security as core deployment requirements. Those are not abstract concerns. They are the difference between an agent that supports care and an agent that quietly amplifies error.
One study that hit me hard was When AI Gets It Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems. The paper focuses on medication-related failure modes, including missed drug interactions, wrong risk flags, and inappropriate dosage recommendations, and it reinforces a simple clinical truth: aggregate performance metrics do not protect patients when the failure mode is one bad recommendation in the wrong context. I have seen enough near-misses in medication reconciliation to know that a system can look excellent on paper and still fail in the one chart that matters.
That is why I would not deploy a general agent to make medication recommendations without a pharmacist-in-the-loop review for high-risk classes, clear provenance, and a hard stop for uncertainty. I would not let an agent draft a discharge plan without explicit clinician verification of diagnosis, medications, follow-up, and red flags. And I would not allow autonomous tool use in a live EHR environment before the hospital can prove, on paper and in drills, who gets alerted when the agent is wrong.
Clinical AI will not stall because the models are weak, but because the organizations are unready
The most interesting recent work is not about agent benchmarks. It is about organizational design. The paper From Clinical Intent to Clinical Model: Autonomous Coding-Agents for Clinician-driven AI Development points toward a future where clinician goals are translated into working systems with less engineering friction. That could be a big deal for hospitals that are drowning in backlog, prior authorizations, chart abstraction, inbox triage, referral routing, and coding complexity.
I can see the operational upside clearly. An agent can sort inboxes, pre-fill forms, reconcile documents, route tasks, draft prior auth packets, and surface missing data before a human ever opens the chart. The public-sector automation paper on IT support tasks, Developing an AI-Powered Automation Framework to Streamline IT Support Tasks in Public Sector Organizations, is not medicine, but it does show the same pattern. Automation creates access and speed when the workflow is repetitive and bounded. Clinical operations have those properties in abundance.
Still, medicine is not IT support. A missed password reset does not equal a missed anticoagulant. A failed ticket does not equal a delayed cancer workup. The tolerance for error is different by orders of magnitude, and the oversight burden rises with it.
The best hospital strategy is probably not “deploy agent everywhere.” It is to pick narrow domains where the failure modes are legible, the escalation path is clean, and the human can verify the output in seconds. Referrals. Prior auth. Chart summarization. Coding support. Internal knowledge retrieval. These are the first places I would look. Not because they are glamorous. Because they are governable.
Trust is a workflow property, not a marketing claim
One of the most clinically relevant papers in this brief is Caging the Agents: A Zero Trust Security Architecture for Autonomous AI in Healthcare. The title is blunt for a reason. Once an agent gets access to patient data, tools, and messaging systems, security stops being a sidebar and becomes a clinical safety issue. Agentic systems can leak, overreach, infer, or execute in ways that a normal dashboard never could.
That is where governance should become boring in the best possible way. Zero-trust access. Role-based permissions. Segmented tool scopes. Audit trails. Red-team testing. Human override. These are not technical ornaments. They are the seatbelts.
The imaging literature makes the same point from a different direction. In Explainability and Trust in Deep Learning for Cancer Imaging, the authors emphasize that clinical misalignment and workflow barriers are central to trust, not just model opacity. I agree. A beautiful explanation that does not match how a radiologist or oncologist actually reasons is still a poor explanation.
The physician-executive lesson is simple: do not ask whether the agent is intelligent. Ask whether the institution can supervise it, constrain it, and recover from it.
What I think changed after working with real systems
I used to think the breakthrough would come from better models. Now I think the breakthrough will come from better institutions. Medicine will need new job descriptions, new escalation ladders, new audit habits, and probably a new middle layer of clinical oversight roles dedicated to AI output review. That may feel bureaucratic. It is also how safety survives scale.
The analogy to academic and community medicine helps here because it exposes the implementation problem. Academic systems can host the first generation of agentic workflows because they already have the people to watch the machine. Community systems will need simpler guardrails, cleaner interfaces, and stronger vendor accountability. Otherwise the technology will deepen inequity by giving the most capability to the places with the most supervision.
There is one more thing I learned from being wrong early. I assumed clinicians would resist agents because of fear of replacement. In reality, most clinicians I speak with are less worried about replacement than about friction, liability, and nonsense. They want the chart cleaned up, the inbox triaged, and the right patient reached on time. They just do not want a probabilistic system pretending to be certain.
Three weeks ago, that same nurse practitioner from the opening scene showed me the final version of the agent’s note after she rewrote the plan herself. She laughed and said, “Okay, now it sounds like something I can sign.” That is the future I trust. Not autonomous medicine. Supervised medicine with faster hands.
FAQ
What happens if a hospital deploys an AI agent without clinician oversight?
The agent will eventually make a confident mistake in a context that matters, and the organization may not catch it in time. In high-stakes workflows, the problem is not only bad answers, but also silent overreach, missing escalation, and unclear accountability.
How do AI agents fit into FDA 510(k), De Novo, and PMA thinking?
They should be evaluated as lifecycle products, not one-time software demos. The FDA’s framework for AI-enabled medical devices already points toward risk-based pathways, so hospitals should expect versioning, monitoring, and post-deployment controls rather than static approval logic.
What is Dr. Sina Bari’s approach to AI agents in clinical operations?
I would use agents for bounded, reviewable work like inbox triage, chart summarization, routing, and drafting. I would not give them unsupervised authority over diagnosis, medication changes, or discharge decisions, because the clinical cost of a wrong turn is too high.
Why compare AI agents to academic medicine versus community medicine?
Because supervision capacity is uneven across health systems. Academic centers can absorb more oversight and experimentation, while community settings often need simpler, safer workflows with fewer assumptions and more guardrails.
What is the single biggest risk with agentic AI in hospitals?
False confidence. An agent can sound polished while being wrong, and that can move a team toward action faster than a human would normally allow. The cure is not fear, it is designed skepticism, logging, and human sign-off.