Healthcare AI’s off switch needs a budget

An AI system can be cheap to buy and expensive to stop.

For a healthcare organisation, the difficulty becomes apparent when a concern justifies suspending a tool that staff have come to depend on. Somebody must pick up the unfinished work. They need time, access to the underlying information, and a workable way to continue. The service still has obligations after the software stops.

I would want those arrangements priced before agreeing to the productivity case. If an organisation has already committed the promised savings elsewhere, it may discover that its freedom to stop exists on paper while the resources needed to exercise it have disappeared.

The business case cannot spend the same hour twice. If the fallback relies on staff released to other work, it should say what gives way when they are called back.

Current and future frameworks

The National Commission into the Regulation of AI in Healthcare’s recommendations, published on 10^th September, make this timely. They call for attention to the operational conditions required for safe deployment and for contracts to allocate responsibility for the necessary risk controls. These are proposals for a future regulatory framework. Buyers can already use the questions they raise to examine what an offer would require of their organisation.

Existing frameworks provide foundations. England’s DCB0160 clinical risk management standard covers deployment, use, maintenance, and decommissioning. The voluntary US NIST AI Risk Management Framework addresses resources, viable alternatives, and responsibility for disengaging AI. The purchasing opportunity is to make those arrangements visible alongside the price and expected benefit.

I would ask for a second demonstration: show me what the service can safely sustain when the AI function is withdrawn.

As a founder developing medicines software before its first pilot, I see this as a question worth answering whilst the design can still change. A proportionate interruption rehearsal could give buyers and suppliers something concrete to examine together.

Retaining essential skills and rehearsing downtime

Consider a fictional medicines evidence service. Software organises source documents and prepares summaries for professional review. An update introduces a mismatch between some summaries and the documents they cite. A reviewer spots the discrepancy. The affected summarisation function needs to be suspended whilst the problem is investigated.

The first test is whether the responsible person can act. They need a deputy, appropriate access, and a clear route to technical support. Staff need to identify the affected work, retrieve the original documents, and decide which tasks can continue safely, which can wait, and which need escalation. A manual fallback has practical requirements: somebody must know how to use it, have permission to access it, and have time to do the work.

Then, there are the summaries already distributed. Reverting the software leaves copies in inboxes and information in decisions already made. The response needs to establish which outputs were affected, who received them, and what review or correction is necessary.

The rehearsal should make that workload visible. Record how long it takes to recognise and contain the problem, which tasks remain unresolved, where access fails, and how much additional staff time is required. Its findings belong with the conditions tested. One successful exercise gives a buyer evidence about that scenario; clinical effectiveness and safety require their own appropriate evaluation.

Restart deserves a separate decision. The supplier may have repaired the fault whilst the provider still has an unresolved queue and staff working through an alternative process. Agreed evidence, appropriate clinical and technical judgement, and a named person authorised to resume the function should govern the return. The backlog needs an owner too.

There is a fair objection here. Healthcare teams already manage continuity plans, safety responsibilities, and scarce staff. An elaborate new exercise could absorb the capacity the investment was supposed to release.

The rehearsal should fit existing arrangements, with its scale determined by the consequences of interruption. Suppliers should provide reusable evidence. Local teams should test the dependencies specific to their service. Suspending one feature, restricting a use case, or using another supported system may preserve essential work more effectively than a complete shutdown. The risks of the alternative matter throughout.

In Digital Health, Yvette Khozam has already argued for retaining essential skills and rehearsing downtime. I would take that argument into the purchasing meeting. Whose staff will be available, what work will they be able to do, and who is paying?

That could involve trained staff, shared support, alternative technology, or a combination. Maintaining a complete manual duplicate of every automated process would be an expensive default. Buyers need an alternative proportionate to the essential work and the interruption they are planning for.

I would put a short interruption scenario beside each bid’s normal operating estimate. It should state which work continues, at what capacity, for how long, and with whose staff and systems. Include the time and cost of reconciling unfinished work after restoration. Identify the assumptions that still need testing and allocate the responsibilities between provider and supplier.

This also makes the comparison between offers more honest. A low licence price may assume that the customer supplies considerable recovery capacity. Another offer may include support that initially looks expensive. Buyers need to understand the service each price can sustain.

If a credible fallback overturns the business case, that is valuable information before purchase. It may justify a narrower deployment, a different operating arrangement or further development. The calculation should remain open to each outcome.

I want healthcare AI to earn the confidence needed for wider use. A buyer should be able to act on an inconvenient finding with the people, access, and money already accounted for.

The off switch needs authority. The people carrying on after it is used need a budget.

Editorial disclosure: The medicines workflow is fictional; no deployed case study or measured outcome is presented.

References

About the author

Varun Sharma is founder and CEO of NEUVIOR, which is developing medicines software with support from Innovate UK and Scottish Enterprise. His work focuses on medicines evidence, governance, and implementation. He writes here in a personal capacity.