The US Food and Drug Administration (FDA) has cleared an artificial intelligence (AI) algorithm from GE Healthcare that analyzes chest x-rays for pneumothorax and helps flag suspected cases for radiologists to prioritize reading, the company announced today.
The algorithm, part of a set of other quality-assurance algorithms named the Critical Care Suite, was developed to run on a GE Healthcare mobile x-ray device. The software is not yet for sale, and an outside expert expressed concern about its false positive rate.
The idea for the application came from bedside clinician experience of waiting for radiologists to read chest x-rays, said Rachael Callcut, MD, MSPH, a surgeon and director of data science for the Center for Digital Health Innovation at the University of California, San Francisco. UCSF proposed developing the feature as part of a development partnership with GE Healthcare.
Since thousands of chest x-rays are taken in a hospital every day, looking for pneumothorax in critical patients is “kind of like trying to find a needle in a haystack,” Callcut told Medscape Medical News. The algorithms themselves are not making diagnoses, she said, but “draw everyone’s attention to finding something potentially life-threatening for patients.”
In addition to data from UCSF, the pneumothorax-flagging algorithm was developed using more than 12,000 images from six total data sources in three different countries, said GE Healthcare’s Katelyn Nye, global product manager of Artificial Intelligence & Analytics, X-ray.
In an interview with Medscape Medical News, Nye said it was a “first for the world” for the algorithm to run on a device for the purpose of alerting a radiologist for triage.
An FDA spokesperson said the agency does not specifically track software with AI functionality, because it tracks devices by their intended use, not platform. The agency has previously cleared AI software, including Zebra Medical Vision’s similar algorithm to analyze chest x-rays for pneumothorax, which runs on the company’s server rather on the x-ray device itself.
In data submitted for FDA clearance, the Critical Care Suite algorithm detected large pneumothoraxes on chest x-rays with 96% sensitivity, small ones with 75% sensitivity, and 94% specificity for both sizes. Its overall area under the curve (AUC) for detecting a pneumothorax was 0.96.
Critical Care Suite’s positive predictive value ranged from 35%-70% depending on the prevalence of pneumothorax: For 4% prevalence the algorithm had 1 true notification per 2 false notifications, and for 15% prevalence it had 7 true notifications per 3 false notifications. Negative predictive value ranged from 97%-99% based on the same prevalence range.
The decisions of two independent readers and an arbitrator, all radiologists from St. Luke’s University Health Network in Pennsylvania, determined “ground truth” to compare the algorithm’s performance, said Nye.
The sensitivity and specificity of the algorithm suggest that it “performs as well as an attending radiologist reading chest x-rays for pneumothorax,” James Tsung, MD, a professor of emergency medicine and pediatrics at Icahn School of Medicine at Mount Sinai in New York City, wrote to Medscape Medical News in an email.
Tsung was not involved in developing the Critical Care Suite but served as an educational consultant for GE Healthcare’s ultrasound division in 2017. He agreed that it would be helpful to get a result from a chest x-ray as soon as possible, especially if a patient has unstable vital signs, but called the algorithm’s rate of 2 false positives per 1 true positive at a low prevalence of pneumothorax “concerning.”
Keeping the methods of development opaque “is a problem for all proprietary AI algorithms because they should be independently validated to examine if they do what they are supposed to do,” he said.
Callcut said the false positives tend to be in complex cases that even a radiologist would have to look at carefully, not on normal x-rays. “With conditions that are life threatening, as a clinician, I would rather have more false notifications that there might be a problem than have those notifications not occur,” she said.
Despite the FDA’s clearance, the Critical Care Suite is not yet for sale. GE Healthcare will conduct clinical pilots at US and global healthcare institutions to gather feedback on users’ experience with the software, such as the interface design, and in a second phase will aim to measure any clinical and operational benefits, Nye said.
At UCSF, Callcut said trials are being planned to research questions including whether clinicians actually use the feature, if it affects their decisions about patient care, and if the software reduces the time to diagnose pneumothorax.
Tsung reports serving as an educational consultant for GE Healthcare’s ultrasound division in 2017. As a UCSF faculty member, Callcut oversees and works on the partnership between the university and GE Healthcare, but she does not have a separate relationship with the company.