American College of Radiology Launches Landmark Artificial Intelligence Quality Registry

Today, the American College of Radiology® (ACR®) launched Assess-AI, the world’s first artificial intelligence (AI) quality registry designed to monitor ongoing performance of an array of imaging AI algorithms in real world clinical settings.

Assess-AI is the newest ACR National Radiology Data Registry, capable of monitoring AI results and collecting an array of contextual information such as patient demographics, modality/exam meta data, and output from radiology reports. Its purpose is to provide radiology facilities with analytics on how clinical AI is operating in their own practices over time and compare their results against aggregated national performance benchmarks from other sites using identical or similar products.

“Users of AI technology in radiological care need to ensure that algorithms perform sufficiently in their local environment,” said Christoph Wald, MD, PhD, MBA, FACR, vice chair of the ACR Board of Chancellors and chair of the ACR Commission on Informatics. “Most legacy radiology systems were not built to support activities to ensure algorithms used in clinical settings are operating as expected. Assess-AI assists facilities with insights into the performance of deployed AI products. We look forward to growing this program with ACR Data Science Institute (DSI) facilitated oversight, offering a tangible real-world approach to address a challenge radiologists are increasingly faced with today.”

Assess-AI is the latest AI-related resource offered by ACR. ACR’s portfolio of AI products includes AI Central – a tool that helps facilities make informed algorithm purchasing decisions to care for their patients – and the ACR Recognized Center for Healthcare-AI (ARCH-AI), the first national AI facility quality assurance program.

Utilizing Assess-AI, clinical sites and AI developers will be able to obtain performance reports of deployed AI derived from local practices. This data will also help developers make improvements to future versions of algorithms.

Radiology facilities that participate in Assess-AI will have access to services including:

• Monitoring the stability of algorithm inputs over time, including imaging equipment, protocols and software version.
• Continuous oversight of AI result concordance/discordance with radiology reports.
• Aggregation of observations into reports and dashboards to share with a facility’s AI governance team, empowering further action as needed.

“Assess-AI will play a critical role in safely and effectively accelerating the clinical adoption of AI in radiology by ensuring AI products perform optimally in clinical settings so that radiologists can focus on what’s most important – providing high-quality care to their patients,” said Woojin Kim, MD, Chief Medical Officer of the ACR DSI. “Expanding on its commitment to the radiology profession, the launch of Assess-AI is ACR’s latest step in helping empower radiologists to implement AI safely, effectively, and transparently.”

With the rising demand for imaging outpacing the supply of radiologists, AI is seen as an essential tool to help bridge the gap and enable radiologists to maintain high standards of care while meeting increasing demands. ACR, working with leadership at the U.S. Food and Drug Administration and Congress, is positioned to guide the future of AI in radiology.