Building agentic AI in healthcare

The healthcare industry has reached an inflection point with respect to agentic artificial intelligence. The technology is advancing faster than most organisations can adopt it – and in healthcare that gap carries real consequences. Unlike other industries where delayed adoption means lost efficiency, in healthcare it can mean delayed care, compromised privacy, and erosion of clinical trust.

The organisations that lead aren’t necessarily the ones moving fastest. They’re the ones moving most deliberately – with a clear understanding of how agentic AI actually works and where responsible use can break down.

Understanding the layers

Agentic AI doesn't exist as a single system. It operates across five interdependent layers:

• Power requirement: like all cloud-computing platforms, a strong grid is needed to power the computing of every AI agent.
• Infrastructure/hardware: the data centres that are springing up across the United States are where the data every AI agent uses "lives".
• Network layer: sophisticated digital safeguards must be in place to ensure private data remains private.
• Data layer: digital data delivery requires a complex chain of protocols.
• Application layer: the way a user interfaces with an AI agent can take a variety of forms.

Every agentic AI user depends on all five. To troubleshoot a problem, or improve a process, you must know which layer to target. And to use these systems responsibly, you need to understand where each layer is vulnerable.

Where responsible use actually lives

Most conversations about responsible AI focus on the top three layers — network, data, and application — and for good reason. Malicious actors can infiltrate an insecure network. That’s an obvious failure. But irresponsible use doesn’t require bad intent. A well-secured organisation can still fail at the data layer through hallucinations that seem harmless until they aren’t.

Consider a concrete example: a patient rates their pain a 10 on a 10-point scale. An LLM interprets that as “severe pain” and writes it into a clinical summary — but the physician never used that word. That single interpretive leap, multiplied across thousands of cases, can create documentation that misrepresents clinical reality, exposes organisations to legal risk, and erodes the trust of the clinicians the system was built to support.

This is why responsible use requires more than secure infrastructure. It requires three things in particular:

• Bias mitigation. The history of medicine offers hard lessons about what happens when treatments and protocols embed the biases of their era. AI systems trained on that same historical data can perpetuate those patterns at scale. Healthcare organisations deploying agentic AI must actively audit for bias across age, race, gender, and other factors – not as a compliance exercise, but as a clinical imperative.
• Observability. Every person whose work is touched by an AI agent needs visibility into what that agent is designed to do and how it’s performing. Without it, there’s no way to confirm the system is operating within its guardrails – and no way to catch it when it isn’t.
• Explainability: The ability to articulate what an AI agent does, and why it does it, shouldn’t be optional. Third-party audits are a critical safeguard, and the capacity to explain a system’s purpose to stakeholders outside the immediate team is foundational to organisation accountability.

Responsible governance in this space isn’t a single membership or certification. It requires a multi-pronged approach. Industry frameworks like NIST’s Risk Management Framework and OWASP provide a critical baseline for managing AI risk and security. HIPAA compliance remains non-negotiable for any healthcare AI deployment. And organisations like CHAI, the Coalition for Health AI, are working toward codifying agentic AI principles into industry-wide best practices.

But frameworks alone aren't enough. The organisations getting this right are integrating these standards into their own internal governance frameworks — contextualising them to their specific workflows, patient populations, and risk profiles — rather than treating compliance as a ceiling. The goal isn't just to meet the minimum. It's to build the kind of trust that makes AI a durable asset, rather than a liability.

Why deliberate adoption wins

Healthcare organisations can’t afford to ignore agentic AI, but they also can’t afford to deploy it without the infrastructure to use it responsibly. The gap between these two failure modes – paralysis and recklessness – is where the real strategic work lives.

Agentic AI represents a bridge between predictive, generative, and autonomous systems. Building that bridge to last requires keeping a human meaningfully in the loop – not as a formality, but as the mechanism by which guardrails are set, monitored, and enforced over time.

The organisations that get this right won’t just avoid the risks. They’ll build the kind of clinical and organisational trust that makes agentic AI genuinely transformative – for their teams, and for their patients.

About the author

Anurag Voleti is VP of data science at Xsolis, where he leads the company's AI/ML strategy and data science team. He brings more than two decades of experience building AI and data capabilities across healthcare and enterprise technology, including CxO and senior technology leadership roles at Blue Cross Blue Shield Association, Boston Scientific, and GE Healthcare, where he spent nearly 13 years leading digital transformation and analytics initiatives. Most recently, Voleti was chief AI and product officer at Stealth AI Startups, accelerating bold ideas to impact, and he holds two patents in the AI space.