Will Babylon’s app ever speak doctors’ language?

Babylon Health might be one of the fastest developing digital health innovators in recent months. It is attracting a lot of press attention and a small army of users that include the UK’s newly-appointed health secretary Matt Hancock. Recently Babylon’s CEO Ali Parsa put his artificial intelligence-based application to the test in a presentation that was live-streamed from London Royal College of Physicians.

Babylon Health’s mission

Babylon’s mission is to put an accessible and affordable health service in the hands of patients. They use a combination of cutting-edge technology and the medical expertise to deliver round-the-clock access to digital health tools to people across the world.

The company claims its AI system has demonstrated diagnostic ability that is “on-par with human doctors” after a series of robust tests. They included the relevant sections of the real-life final test each trainee general practitioner (GP) must pass to demonstrate their competence in knowledge and diagnostics.

Human versus AI in healthcare

These tests included a set of questions testing diagnostic skills from publicly available RCGP sources, as well as independently published examination preparation materials, mapped these to the current RCGP curriculum to ensure the questions resembled actual RCGP questions as closely as possible.

The average pass mark over the past five years for real-life doctors was 72%. In sitting the exam for the first time, Babylon’s app scored 81%. As it continues to learn and accumulates knowledge, Babylon expects that subsequent testing will produce significant improvements in terms of results.

In real life situation GP’s knowledge does not come from books alone, but also from face-to-face patient experience. To test these abilities the application faced seven “highly-experienced” primary care doctors using 100 independently-devised symptom sets.

The machine once more was more accurate than a human. It scored 98% against conditions seen most frequently in primary care medicine comparing to 52-99% range accuracy among experienced clinicians.

Could a computerised brain improve healthcare?

Ali Parsa thinks that the results clearly illustrate how AI-augmented health services can reduce the burden healthcare systems around the world face today.

“Our mission is to put accessible and affordable health services into the hands of every person on Earth. These landmark results take humanity a significant step closer to achieving a world where no-one is denied safe and accurate health advice,” he said.

Babylon Health’s mission becomes reality

Rwanda became the first country to make this technology available nationwide, following on the heels of the launch of a free digital NHS service in London. It was launched in Rwanda in 2016 and is already delivering accessible healthcare to 600,000 patients in partnership with the Rwandan government and the Bill & Melinda Gates Foundation.

Parsa said: “Our partnership with the Rwandan government shows what’s possible. I am proud that this partnership means Rwanda is one of the first countries to offer its entire population accessible healthcare wherever and whenever they need it. It’s a model that I hope can be used to make good healthcare more accessible around the world.”

A watchful eye on the Babylon Health app

As much as the new technology is taking that one African country by storm and contributing to improving its healthcare system, it does not look all so promising in the UK. There the GP at Hand application was rolled out across the Hammersmith and Fulham Clinical Commissioning Group last year and is currently delivering an online doctor services to over 26,000 people.

But back in March this year the NHS issued a tender for an independent review of Babylon’s app, and at the same time objected to plans for a significant expansion of this service beyond London following issues raised by the national independent healthcare standards watchdog Care Quality Commission (CQC).

The CQC found two cases of medicines being prescribed outside their licensed use without the required explanation for this use, and no records showing that patients were informed of this exceptional use.

The inspectors also found instances of GPs relying on a patient’s word, rather than clinical evidence to support a diagnosis – for instance, there was no record of blood tests carried out for three patients with thyroid disease. The CQC also found information was not always shared with a patient’s primary doctor to ensure prescribing was safe or appropriate. The standards watchdog also found there was no system to give assurance that patients’ conditions were being appropriately monitored.

The report only became public after Babylon had tried, and failed, to gain a legal injunction to block it.

RCGP’s cautious optimism

Helen Stokes-Lampard, chair of the Royal College of General Practitioners, said: “We need to be cautious about the claims being made at the moment. It’s really exciting to see AI evolving and there will be great applications as its gets better, but right now we don’t have independently-verified, peer-reviewed research – the scientific standard for accepting innovation and moving forward.

“What we have is some in-house research from Ali and his team. I am delighted that they are using partners and academic institutions to increase the robustness of their research, but it’s still very early days.

“I’m concerned about the hype surrounding this right now. We need to be cautious, we need to demand evidence because we have to be sure that anything we introduce is safe.”

Despite this uncertainty in the delivery of Babylon services, they seem to attract a lot interest from investors and technology giants. The company recently partnered with Samsung to incorporate their Ask an Expert video doctor appointments to Samsung’s Health app. It also signed a collaboration deal with Tencent, Chinese e-commerce and social networks leader, to deliver personal health assessments, treatment advice and individual record across mainland China.

Publishing its latest study and test results on the company website, Babylon Health’s authors say the application may hold the promise of reduced costs and improve access to healthcare worldwide, but add that this would entail acquiring "greater levels of confidence from the medical community and the wider public". The researchers also conclude that "studies using larger, real-world cohorts will be required to demonstrate the relative performance of these systems to human doctors".