Algorithms subject to an AI challenge organized by radiological Society of North America (RSNA) have shown excellent performance to detect breast cancers on mammography images, increasing the sensitivity to screening while maintaining low recall rates, according to a study published today in RadiologyThe first newspaper of the RSNA.
The challenge of detection of the detection of the RSNA Mammography breast cancer was a crowdsource competition which took place in 2023, with more than 1,500 participating teams. THE Radiology Details of the article An analysis of the performance of algorithms, led by Yan Chen, Ph.D., Cancer screening professor at the University of Nottingham in the United Kingdom.
We were overwhelmed by the volume of candidates and the number of AI algorithms that were submitted as part of the challenge. This is one of the challenges most participated in the AI RSNA. We were also impressed by the performance of algorithms given the relatively short window authorized for the development of algorithms and the requirement to offer training data from open source locations. “”
Yan Chen, Ph.D., Cancer screening professor, University of Nottingham
The objective of the challenge was to obtain AI models which improve the automation of cancer detection in mammograms screening, to help radiologists to work more efficiently, to improve the quality and safety of care for patients, and to reduce unnecessary costs and medical procedures.
RSNA invited team participation from around the world. Emory University in Atlanta, Georgia and BreastScreen Victoria, Australia, has provided a set of training data of around 11,000 breast screening images, and the challenge participants could also obtain training data accessible to the public to their algorithms.
Professor Chen’s research team evaluated 1,537 work algorithms subject to the challenge, testing them on a set of 10,830 rescue exams only accompanied by the set of training data – which were confirmed by pathology results as positive or negative for cancer.
Overall, algorithms have given median specificity of 98.7% to confirm that no cancer was present in mammography images, 27.6% sensitivity to positively identify cancer and a recall rate – the percentage of cases that AI deemed positive of 1.7%. When the researchers combined the top 3 and top 10 effect algorithms, it increased sensitivity to 60.7% and 67.8%, respectively.
“During all the most efficient entries, we were surprised that different AI algorithms are so complementary, identifying different cancers,” said Professor Chen. “Algorithms had optimized thresholds for a positive predictive value and high specificity, therefore different cancer characteristics on different images triggered high scores differently for different algorithms. »»
According to the researchers, the creation of a set of the 10 most efficient algorithms produced performances that are close to those of an average screening radiologist in Europe or Australia.
Individual algorithms have shown significant differences in performance according to factors such as the type of cancer, the manufacturer of imaging equipment and the clinical site where the images have been acquired. Overall, algorithms had greater sensitivity to detect invasive cancers than for non -invasive cancers.
Given that many Participants’ AI models are open source, the challenges of the challenge can contribute to the additional improvement of the experimental and commercial AI tools for mammography, in order to improve the results of breast cancer in the world, said Professor Chen.
“By releasing algorithms and a set of complete imaging data to the public, participants provide valuable resources that can generate additional research and allow benchmarking that is required for the effective and safe integration of AI in clinical practice,” she said.
The research team plans to conduct monitoring studies to compare the performance of the main challenge algorithms compared to commercially available products using a larger and more diverse data set.
“In addition, we will study the efficiency of smaller and more difficult test sets with robust human references, such as those developed by the performance scheme, a program based on the United Kingdom to assess and ensure the quality of radiologist performance as an AI approaches, and compare its usefulness to that of large-scale data sets,” said Professor Chen.
RSNA is organizing an AI challenge each year, with this year’s competition that is looking for bids for models that help detect and locate intracranial aneurysms.
