Pushy NHS chatbots 'can put patients off screening'

Amid the push to digitise the UK health service, a study has sounded a note of caution about entrusting important patient appointments to chatbots.

In a paper published in the journal Lingua, researchers at the University of Surrey polled 300 patients who used a chatbot called Asa to book a cervical cancer screening test, which revealed that the manner in which an AI assistant communicates matters as much as what it does.

While the patients tended to respond positively to friendliness and choice-oriented language, they were put off by "over-messaging, pushy reminders, and blurred human-AI boundaries," according to the team, led by Dr Doris Dippold, associate professor of intercultural communication at the University of Surrey.

"Our analysis shows that anthropomorphism is not universally positive," said Dippold. "Human-like features can build rapport – but when they clash with patients' expectations for transparency in a healthcare setting, they undermine exactly the trust the chatbot is trying to build."

Understanding why that can occur is important, particularly as cervical screening uptake across the UK fell more than 5% in 2023-24, with ethnic minority groups consistently underrepresented in screening programmes – something that Asa was specifically designed to overcome. And doubly so, given the likelihood that AI assistants are likely to become an increasingly common component of NHS delivery, now that the UK government has made digital engagement a pillar of its health reforms.

Developed by SPRYT with funding from pharma company MSD, Asa was the first WhatsApp and AI-based appointment scheduling app to be approved for use by the NHS after a successful pilot at the NHS North Central London Integrated Care Board (ICB).

That chatbot is designed so patients can interact with it as they would with a human receptionist, and allows them to book, reschedule, and cancel appointments via WhatsApp at any time, without requiring a new app or website.

The Surrey study found that, on the whole, Asa users described the chatbot positively, using terms like "friendly", "kind", and "not forceful", with some reporting that the female persona presented by the AI made it easier to disclose sensitive information, such as needing to cancel an appointment due to menstruation. Asa was also seen to provide anonymity benefits in a sensitive health domain.

However, the research also uncovered what the researchers call "friction points," noting that "many patients found follow-up messages sent within 24 hours intrusive, and described imperative phrasing such as 'Let's book you in' as aggressive, rather than helpful." Those perceptions tended to be exacerbated for patients managing mental health challenges, neurodivergent conditions, or demanding caring responsibilities.

Patients also disliked a lack of dialogue opportunities with Asa, for example, when the AI did not respond to queries or did not offer opportunities to ask questions.

At the same time, many of the respondents said they had ethical concerns about the use of Asa, including data security, impersonation, and anthropomorphic features that blurred the boundaries between human and AI.

Chatbots should be designed with several key considerations, beyond simply helping patients achieve a goal. A feeling of control over decisions is critical, as is the chatbot's ability to respond appropriately, strike a respectful tone, ensure fairness, and – crucially – be transparent about the underlying technology.

"Feeling seen, appreciated, and emotionally supported is not a luxury feature in health AI - it is a condition of access," said Dippold. "If patients disengage because a chatbot feels pushy or untrustworthy, the health service loses them entirely."