This is part 1 of our blog series on why we are concerned about the proposed use of AI in the 10 year plan for the health service recently released by the government . Part 2 can be found here
When an NHS patient received a letter inviting him to a screening for diabetes earlier this year, he was perplexed.
He had never requested the appointment nor shown any signs of the condition.
At first, he dismissed the letter as an admin error, until a nurse asked him about his diabetes at a subsequent check-up.
He explained he didn’t have the condition, but the nurse was adamant that his medical records showed he had been diagnosed with it.
On reading his medical notes, the patient realised they had been generated by AI.
And as well as misdiagnosing him with diabetes, the AI transcription software had also falsely recorded that he was at risk of heart disease, Fortune reported.
The errors, which the NHS has claimed were a one-off, illustrate the potential risks the health service faces as ministers push to rapidly adopt AI.
Last month, the government pledged to make the NHS “the most AI-enabled care system in the world” as part of its 10-year plan.
The promise, framed as a “digital revolution”, aims to speed up access to care, reduce in-person appointments, and cut costs.
What might those plans look like in practice?
For starters, the government is proposing something called “ambient AI” during patient consultations which “can listen to what is being said, identify the key points, and record them into the care record.”
In other words, exactly the technology that led to a patient being erroneously invited to a diabetes screening.
To what extent AI holds the promise of unburdening staff from paperwork remains unclear.
Trials already underway in several London NHS organisations had mixed results: while 21% of staff said AI freed up time, 28% said that AI either had a negative impact, none at all, or weren’t sure, according to a survey carried out by UCLPartners.
Reliability was one issue flagged by staff in the trial. Although one AI tool reportedly helped staff clear a backlog of bone fractures x-rays, another, used to analyse for lung issues, was dropped due to a high rate of false positives.
Studies have shown that AI in healthcare has benefits, but researchers caution that it should complement rather than replace clinical judgement.
Yet, in the proposals set out by the government, the clinicians seem to be out of the picture.
An NHS GP tool, we are told, will “use AI-algorithms to take a patient’s descriptions of their worries or symptoms, ask the right follow-up questions and provide personalised guidance.” It will even advise on self-care and direct them to appropriate products, all seemingly without direct human intervention.
As the plan acknowledges, the NHS will have to “work in partnership with those creating exciting new technologies” to adopt AI.
In other words, it could represent a significant transfer of power and money from the public to the private sector.
That’s what we talk about next in part 2 of this blog series: The Price of AI in Healthcare: Will US Tech Giants Control the NHS?