ACR Appropriateness Criteria® Overview

Prologue

In creating the ACR Appropriateness Criteria® (ACR AC), the ACR Task Force on Appropriateness Criteria incorporated attributes for developing acceptable medical practice guidelines used by the Agency for Healthcare Research and Quality (AHRQ) as designed by the Institute of Medicine. These attributes are:

  • Validity: Guidelines are valid if they lead to better outcomes. Validity assessment should be based on the quality of the scientific evidence and the method of evidence evaluation.
  • Reliability/Reproducibility: Another set of experts should be able to produce similar guidelines when using the same methodology to evaluate the same scientific evidence
  • Clinical Applicability: Guidelines should include an explicit description of the applicable patient population
  • Clinical Flexibility: Guidelines must specify known or expected exceptions
  • Clarity: Guidelines must be unambiguous with clearly defined terms. They should be presented in a logical manner and be easy to follow.
  • Multidisciplinary Process: Affected provider groups should have representation in the guideline development process
  • Scheduled Review: All guidelines should undergo scheduled review to determine whether revision is indicated based on current scientific evidence
  • Documentation: The development procedure, the participants, the evidence, and the methods of analysis should be documented
The AHRQ has been explicit in stating its intent that scientific evidence should be used as much as possible but that judgment and group consensus will be necessary in the development of medical guidelines. The National Guidelines Clearinghouse (NGC), one of the initiatives of AHRQ, is a public resource for evidence-based clinical practice guidelines. The ACR AC topics are posted on the NGC site, as well as on the ACR website (www.acr.org/ac).

Disclosure of Funding Source and Committee Independence

Funding Source 

“The funding for the process is assumed entirely by the American College of Radiology. ACR staff support the expert panels through the conduct of literature searches, acquisition of scientific articles, drafting of evidence tables, dissemination of materials for the Delphi process, collation of results, conference calls, document processing, and general assistance to the panelists.”

AC Participants’ Independence in Guideline Development

The AC panel members have editorial independence from the ACR when developing the guidance documents. There are inputs within the committee governance structure to encourage diversity and representation for the guidance development panels members and to support adequate review by primary care providers, patient representatives, and technical experts. The recommendations on the final documents are the result of the objective evidence-based processes and the process to determine group consensus. The panels review the recommendations and modify the text to ensure the text represents the final recommendations. The final AC documents are not reviewed, modified, or approved by an ACR committee external to the process, such as the ACR Board of Chancellors (BOC).

The ACR BOC has delegated authority for independently creating the committees and documents through ACR’s governing structure and process. The chair of the ACR AC Committee, who is ultimately responsible for the AC documents, is recommended for appointment to the BOC Chair by the Chair of the Commission on Quality and Safety.

 

Disclosing Potential Conflicts of Interest and Management of Conflicts of Interest

An important aspect of committee operations is the disclosure and management of potential conflicts of interest. In 2016, ACR began an organization-wide review of its conflict of interest policies. The current ACR COI policy is available on its website. The AC program’s COI process varies from the organization’s current policy to accommodate the requirements for qualified provider-led entities as designated by the Centers for Medicare and Medicaid Services’ AUC program.

When physicians become participants in the AC program, welcome letters are sent to inform them of their panel roles and responsibilities, including a link to complete the COI form. The COI form requires disclosure of all potential conflicts of interest. ACR staff oversees the COI evaluation process, coordinating with review panels consisting of ACR staff and members, who determine when there is a conflict of interest and what action, if any, is appropriate. In addition to making the information publicly available, management may include exclusion from some topic processes, exclusion from a topic, or exclusion from the panel.

Besides potential COI disclosure, AC staff begins every committee call with the conflict of interest disclosure statement listed below reminding members to update their COI forms. If any updates to their COI information have not been submitted, they are instructed not to participate in discussion where an undisclosed conflict may exist.

Finally, all ACR AC are published as part of the Journal of the American College of Radiology (JACR) electronic supplement. Those who participated on the document and are listed as authors must complete the JACR process that includes completing the International Committee of Medical Journal Editors (ICMJE) COI form which is reviewed by the journal’s staff/publisher.


 

The following COI statement is read to the Panel at the beginning of each call:

“By participating on this call, you confirm that you do not have a conflict of interest that would exclude you from working on this AC document. If you have a potential conflict of interest that has not been previously disclosed and reviewed by staff, we ask that you recuse yourself from the relevant discussion. Please discuss your possible conflict of interest with ACR staff at your convenience any time after the conference call.”

Committee Structure

In 2000, the ACR Task Force on Appropriateness Criteria became the Committee on Appropriateness Criteria under the ACR’s Commission on Quality and Safety. In 2012, two separate AC Committees were created – the Committee on Diagnostic Imaging (DI)/Interventional Radiology (IR) Appropriateness Criteria and the Committee on Radiation Oncology (RO) Appropriateness Criteria. (Currently, the ACR RO AC is undergoing a major transition. While the RO AC panels are not actively developing new content, active RO topics are still available on the ACR web site.)

The DI/IR AC committee is comprised of the panel chairs from the DI and IR panels. The DI/IR Committee oversees the activities of their expert panels. The diagnostic panels are mainly organized along body systems (breast, cardiac, gastrointestinal, musculoskeletal, neuroradiology, thoracic, urologic, vascular, pediatric and Gyn and Ob imaging). Each expert panel is chaired by an individual with leadership capabilities and national recognition of expertise in the area of focus.

The Subcommittee on Radiation Exposure assigns and regularly reviews the relative radiation levels for the procedures included in the topics and updates the “Radiation Dose Assessment Introduction” document as needed.

The Subcommittee on Appropriateness Criteria Methodology provides methodological oversight to all the panels. Its purpose is to ensure that consistent, sound methods are used in developing and revising AC topics across all the panels. It includes members from the various AC panels as well as individuals with methodological and research expertise.

The current AC organizational chart can be found here.

Committee Composition

Over 450 volunteer physicians are involved in the ACR Appropriateness Criteria development process.

Each panel chair is responsible for recruiting and selecting the radiology members of their expert panel. Physicians representing diverse geographic regions, practice settings (e.g., freestanding imaging centers, hospitals, private practice), demographic characteristics (e.g., sex, age, years of practice), imaging modalities (e.g., CT, MR, nuclear medicine, ultrasound), or clinical settings (e.g., ambulatory, emergency). ACR staff works with the staff from numerous major medical societies to recruit a multidisciplinary team, including nonradiology clinical experts for topic areas relevant to their specialty to participate in the development of the criteria. More than 80 representatives from 20 medical specialty organizations participate on the ACR AC expert panels.

In addition to the experts who practice in imaging studies and relevant clinical topics provided by the ACR and collaborating medical specialty organizations, the expert panels have primary care providers (for example but not limited to internal medicine, pediatrics, obstetrics/gynecology and general medicine providers). The expert panels also have PhD and masters degree physicians with expertise in clinical trial design, methodology and/or statistical analysis. The panels have about 12 to 15 members on average.

The expert panels are support by the AC Methodology Subcommittee that reviews and disseminates updates to the methodology through ACR staff. All staff are trained in the ACR AC methodology in weekly meetings and guide the panels through the daily implementation of the methodology and administrative processes. They are non-voting methodologists to their assigned panels.

Process of Criteria Development

The ACR AC are based on a systematic review of evidence as demonstrated by literature search, evidence table development, and topic development documents. Topic selection may be based on the prevalence of the condition, the variability of practice, the relative cost, the potential for morbidity or mortality, and the potential for improved care. Each question is clarified and refined to be as specific as possible and frequently the clinical conditions are broken down into a number of variants.

Once a clinical condition has been defined, a literature search of peer-reviewed medical journals is conducted and the relevant articles are identified and collected. The topic author assesses the literature then drafts or revises the narrative summarizing the evidence found in the literature. ACR staff drafts an evidence table based on the analysis of the selected literature. These tables rate the study quality for each article included in the narrative.

The expert panel reviews the narrative, evidence table and the supporting literature for each of the topic-variant combinations and assigns an appropriateness rating for each procedure listed in the variant table(s). Each individual panel member assigns a rating based on his/her interpretation of the available evidence.

Defining Appropriateness

The ACR adopted the definition of appropriateness mentioned in the RAND/UCLA Appropriateness Method User’s Manual (Fitch 2001) where “the expected health benefit (e.g., increased life expectancy, relief of pain, reduction in anxiety, improved functional capacity) exceeds the expected negative consequences (e.g., mortality, morbidity, anxiety, pain, time lost from work) by a sufficiently wide margin that the procedure is worth doing, exclusive of cost" (Brook et al., 1986; Park et al., 1986).

An assumption when assessing appropriateness is that the ordering health care provider has not yet determined whether a radiological procedure is clinically useful for the specific situation. The expert panel may recommend no radiological procedure as being appropriate for a specific clinical scenario. In those instances where more than one radiological procedure may be appropriate, the expert panel will provide additional guidance or clarification of the issues.

Rating Appropriateness

The ACR AC methodology is based on the RAND/UCLA Appropriateness Method. The appropriateness ratings for each of procedure or treatment included in the AC topics are assessed using a modified Delphi method. An initial survey is conducted to elicit each panelist’s expert interpretation of the evidence based on the available data regarding the appropriateness of an imaging or therapeutic procedure for a specific clinical scenario. The expert panel members review the evidence presented and assess the risks or harms of doing the procedure balanced with the benefits of performing the procedure. The direct or indirect costs of a procedure are not considered when determining appropriateness (additional assumptions regarding rating appropriateness can be found in the document Rating Round Information). When the evidence for a specific topic and variant is uncertain or incomplete, expert opinion may supplement the available evidence or may be the only means for assessing appropriateness.

The appropriateness is represented on an ordinal scale that uses integers from 1 to 9 grouped into three categories: 1, 2, or 3 are in the “usually not appropriate” category, where the harms of doing the procedure outweigh the benefits; and 7, 8, or 9 are in the “usually appropriate” category, where the benefits of doing a procedure outweigh the harms or risks. The middle category, designated “may be appropriate”, is represented by 4, 5, or 6 on the scale. The middle category is where the risks and benefits are equivocal or unclear, the dispersion of the individual ratings from the group median rating is too large (i.e., disagreement), the evidence is contradictory or unclear, or there are special circumstances or subpopulations which could influence the risks or benefits embedded in the variant.

The ratings assigned by each panel member are presented in a table displaying the frequency distribution of the ratings without identifying which members provided any particular rating. To determine the panel’s recommendation, the rating category that contains the median group rating without disagreement is selected. This may be determined after either the first or second rating round. If there is disagreement after the first rating round, a conference call is scheduled to discuss the evidence and, if needed, clarify the variant or procedure description. If there is still disagreement after the second rating round, the appropriateness rating is “may be appropriate.”

This method enables each panelist to articulate his or her individual interpretations of the evidence or expert opinion without excessive influence from fellow panelists in a simple, standardized and economical process. For additional information on the ratings process, see the Rating Round Information document.

Patient and Public Perspectives

ACR AC is creating online public synopses to help patients understand the recommendations better. The patient committee drafting the synopses has access to provide their perspectives to the panel during the development process to provide their unique perspectives.

ACR’s online feedback form is available to any registered AC user. The individual can put general comments about the AC such as suggested topics, new evidence or questions about methodology. The form also allows for detailed comments which can be linked to a panel, topic, variant or a specific procedure for a variant. This linking will allow ACR to better respond to specific issues.

ACR is working towards a more formal external review process to better manage solicitations for public comments.

Document Review and Update

The ACR AC addresses 211 clinical conditions with more than 1,050 variants. New topics are added to reflect changes in technology and clinical practice. The ACR AC are reviewed annually and updated by the panels every 3 years or sooner, depending on the introduction of new and important scientific evidence.

Additional Information

Additional methodology documents and all ACR AC topics can be found on the ACR website at www.acr.org/ac. For more information on the ACR Appropriateness Criteria, please contact the ACR at acr_ac@acr.org.

Disclaimer

The ACR Committee on Appropriateness Criteria and its expert panels have developed criteria for determining appropriate imaging examinations for diagnosis and treatment of specified medical condition(s). These criteria are intended to guide radiologists, radiation oncologists and referring physicians in making decisions regarding radiologic imaging and treatment. Generally, the complexity and severity of a patient’s clinical condition should dictate the selection of appropriate imaging procedures or treatments. Only those examinations generally used for evaluation of the patient’s condition are ranked. Other imaging studies necessary to evaluate other co-existent diseases or other medical consequences of this condition are not considered in this document. The availability of equipment or personnel may influence the selection of appropriate imaging procedures or treatments. Imaging techniques classified as investigational by the FDA have not been considered in developing these criteria; however, study of new equipment and applications should be encouraged. The ultimate decision regarding the appropriateness of any specific radiologic examination or treatment must be made by the referring physician and radiologist in light of all the circumstances presented in an individual examination.