Many people hear “FDA-cleared algorithms” and think of all of the work that has been done. The extensive research that goes on behind the scenes to vet medical devices at the FDA is no secret and concluding that FDA clearance comes after a rigorous review is simple enough to do. When it comes to FDA-cleared AI algorithms related to medical imaging, however — here’s why it pays to do your homework.

Let’s start with the basics: Not everyone understands the difference between the terms “FDA approved” and “FDA cleared” when it comes to medical devices. Do they mean the same thing? Are the terms interchangeable? Not at all.

AI algorithms for medical imaging are not FDA approved, despite various articles you may have read that misclassify them this way. According to the FDA, devices (including software as a medical device) are classified according to risk. For a device to be FDA approved, it must be approved via a premarket application as a Class III device, “demonstrating with sufficient, valid scientific evidence that the devices are safe and effective for their intended uses.” FDA-cleared algorithms, on the other hand, need only to demonstrate “substantial equivalence” to a predicate device.

So, there is far more than semantics involved in the two terms. The FDA processes are very different. FDA approval is much harder to achieve than FDA clearance.

Learning About the Catalog

In 2020, the ACR Data Science Institute® (DSI) created a catalog of AI algorithms “cleared” for use in radiology to help simplify research by radiologists and developers. As of May 2021, the regularly updated catalog had models, ranging from detecting pneumothorax in chest X-rays to highlighting segments of the brain on MRI.

Each model in the catalog includes a summary with the model manufacturer, FDA product code, body area, modality, predicate device, product testing and evaluation related to product performance, and clinical validation. Many of the models match the ACR DSI’s Define-AI use cases and are linked under related use cases. Clicking on individual models takes users directly to the FDA summary letter for more details. All of these features make it easier to vet the algorithms — an important step for any practice considering incorporating a new algorithm.

Here are some interesting factoids about the current ACR DSI catalog of FDA-cleared algorithms:

• The clearances include 94 products, meaning that so far only 17 in the current catalog have undergone updates or revisions that required reevaluation and clearance. This might be surprising given that many people think of AI and the concept of continuous learning as synonymous. But according to current FDA guidelines, algorithms are locked and cannot be modified in any significant manner without necessitating a (likely costly and time-consuming) repeat evaluation by the FDA. So, except for these 17, all of the algorithms remain exactly as they were initially approved — despite any improvements or learnings the vendors might have had since release.
• Ninety-four products represent 65 companies operating in this space. So contrary to what you may have thought, the major tech companies are not dominating this space yet. The vast majority of the companies are startups with only one cleared product.
• As of May 2021, none of the products in the catalog underwent the more rigorous premarket approval process (Class III); 108 underwent the 510K process and three underwent the De Novo process — which provides a pathway to classify novel medical devices for which general controls alone, or general and special controls, provide reasonable assurance of safety and effectiveness for the intended use, but for which there is no legally marketed predicate device.

Not everyone understands the difference between the terms “FDA approved” and “FDA cleared” when it comes to medical devices. Do they mean the same thing? Are the terms interchangeable? Not at all.

Caring About the Catalog

For starters, if you’re thinking about purchasing an AI product, this catalog is a great place to see what’s available. If you know the type of algorithm you’re looking for (for example, which organ system, subspecialty, or specific use case is most applicable), you can see how many companies offer products in that space. It’s also a great way to understand more about the types of algorithms that are available — which ones are triage/notification only, post-processing algorithms, detection algorithms, or actual diagnostic algorithms.

A recent ACR DSI survey found that 95% of current AI users find FDA-cleared algorithms to be inconsistently accurate when tested on their own data (see page 14 for more information from the survey). What does this mean for you? Unfortunately, there’s no way of knowing if an algorithm will work at your institution just by looking through the FDA clearance details — but it may give you some hints. For example, studies have shown that algorithms work best on the images acquired using the same manufacturer of the scanner that the algorithm was trained on. Typically, this information is included in the FDA summary, and you can compare it to the type of scanners you use. More generally, you can also get a sense for the types of studies that were conducted to gain clearance, including how many images were included in their trials or whether clinical validation studies were performed.

Looking to the Future

We look forward to a time when algorithm manufacturers provide more robust information about their products’ validation process. A better understanding of the training and validation parameters will help users understand potential biases and pitfalls that can arise in clinical use, leading to a better user experience overall. The ACR DSI hopes to be able to provide validation and training information through the catalog, when available, to assist the medical imaging community in better understanding which algorithms provide the greatest benefit to their patients.