Key Takeaways from the 2024 ACR-SIIM Data Science Summit

The ACR Data Science Institute® (DSI) annually hosts a summit to update members and industry professionals on its projects and initiatives. This year’s event, held on June 26, 2024, at the Gaylord National Resort in National Harbor, Maryland, brought together thought leaders, industry professionals, and radiologists to discuss the latest advancements and challenges in radiology artificial intelligence (AI). This year’s conference emphasized safe, testable, and trustworthy AI in medical imaging, highlighting the necessity for bias detection, validation, transparency, and reliability, approaches to build clinical and operational AI tools that are not only for everyone but also for everyone else.

Welcome and Keynote Address

Co-chairs Drs. Po-Hao “Howard” Chen and Bibb Allen Jr. welcomed attendees, setting the stage for a day packed with insights and discussions. Dr. Paul Yi’s keynote address, titled "The Road to Biased AI is Paved With Good Intentions," emphasized the importance of trust and safety in AI applications within radiology. He outlined essential components for building trustworthy AI, including algorithmic fairness and transparency, and shared lessons from his own experiences. Dr. Yi's key takeaways were:

• You don't know what you don't know
• Question your assumptions
• Look for silver linings and be open to new ideas

Session 1: Understandable, Auditable, & Transparent AI

Moderated by Dr. Allen, the first session focused on making AI in radiology transparent and understandable. 

Dr. Allen discussed principles to ensure AI's trustworthiness, including minimizing harm, ensuring transparency, and curtailing bias. He emphasized the importance of ethical standards and the radiologist’s responsibility as the ultimate decision-maker, even with AI involvement. Laura Brink, an ACR data scientist, highlighted the necessity of explainability in AI models, stressing that AI systems must provide understandable, accurate, and meaningful explanations tailored to their audience. Dr. Bernardo Bizzo from Mass General Brigham shared practical insights on implementing AI in clinical practice, advocating for a phased approach to AI validation and deployment to ensure reliable performance in diverse clinical settings. This session underscored the collaborative effort required among developers, clinicians, and regulators to integrate AI responsibly into radiology, enhancing patient care while maintaining high standards of safety and efficacy.

Session 2: Testing AI for Robustness & Reliability

Moderated by Dr. Andrew Missert, the second session addressed testing AI throughout its lifecycle to ensure robustness and reliability. 

Dr. Missert emphasized that AI models require thorough testing to understand their limitations and potential failure points. Dr. Tessa Cook discussed Penn Medicine's structured approach to evaluating AI solutions, emphasizing a three-phased testing process: retrospective evaluation, small-scale prospective evaluation, and larger production-scale evaluation. Dr. Panagiotis Korfiatis highlighted the importance of exploratory data analysis and model testing at the Mayo Clinic, where a program bridges the gap between basic research and clinical applications. Dr. Zhongnan Fang stressed the need for a quality management system, informed by FDA guidelines, to ensure rigorous verification, validation, and post-production monitoring. The session concluded that robust testing protocols are essential for safely integrating AI into clinical practice and ultimately enhancing patient care.

Session 3: Making AI Safe

The third session was moderated by Dr. Chen and explored strategies to ensure AI safety in clinical settings. 

Dr. Judy Gichoya from Emory University presented a 2024 update on detecting AI bias, emphasizing the challenges in defining fairness and the necessity for continuous evaluation to ensure proper patient care. Dr. Marla Summer from Texas Children’s Hospital highlighted the need for pediatric-specific AI tools, advocating for increased transparency and accuracy to ensure young patients' safety. She proposed standardized labels and warnings on AI tools to prevent misuse of adult-centric models in pediatric care. Dr. Matt Lungren, Chief Data Science Officer for Microsoft Health & Life Sciences, discussed advancements in large language models and the challenges they present, such as hallucinations and automation biases. He highlighted the importance of transparency and explored techniques like prompt engineering and feature control to improve AI reliability and safety in healthcare.

Session 4: ACR's Efforts in Advancing AI

The final session, moderated by Dr. Christoph Wald, Chair of the ACR Commission on Informatics, showcased ACR's efforts to facilitate AI integration within radiology. 

Introduced by the incoming ACR CEO Dr. Dana Smetherman, Dr. Laura Coombs unveiled the ACR Recognized Center for Healthcare-AI (ARCH-AI) program, aimed at helping radiologists incorporate AI safely and effectively. Dr. Tarik Alkasab from Mass General Brigham discussed the unification of data models for next-generation imaging tools. ACR Data Scientist Chris Treml provided an overview of the AI market landscape through ACR AI Central, highlighting the exponential growth of AI technologies in radiology. Finally, ACR Chief Information Officer Mike Tilkin emphasized the importance of ongoing monitoring and real-world validation of AI tools. This confluence of thought leaders and innovators from ACR provided a fitting finale to the summit, leaving attendees with actionable insights and a clear vision for the future of AI in radiology.

Concluding Remarks

The ACR Data Science Summit 2024 underscored the importance of ethical considerations in the development and deployment of AI in radiology. The discussions highlighted the need for transparency, robust testing, and continuous monitoring to ensure AI tools enhance patient care without compromising safety or quality. As radiology evolves with AI, it is crucial for practitioners to stay informed and engaged with these advancements. The summit reinforced that while AI offers tremendous potential, its integration must be approached with caution, keeping the patient at the center of AI innovation.

Po-Hao "Howard" Chen, MD, MBA | Vice Chair of AI in Diagnostics Institute, IT Medical Director for Enterprise Radiology, and Staff Radiologist | Cleveland Clinic, Cleveland, OH