Report of the ACR Task Force on Certification in Radiology: History, Challenges and Opportunities

Certification in radiology administered by the American Board of Radiology (ABR) has a long, respected history under the auspices of the American Board of Medical Specialties (ABMS). However, changes in the format and content of certifying examinations have led to concerns by ACR members. Responding to such concerns early in 2019, around the same time the ABMS Vision Commission issued its recommendations, ACR leadership established the Task Force on Certification in Radiology, which produced this report. This report primarily reviews maintenance of certification (MOC) but also addresses:

1. Some aspects of initial certification.
2. The history of medical certification.
3. Specific specialty certifications.
4. A review of how other specialties and professions address certification.
5. The legal and regulatory environment.
6. How certification and recertification processes can be improved.

Opportunities for collaboration and improvement that align with Vision Commission recommendations include:

1. Increasing transparency of ABR activities and finances.
2. Clarifying and prioritizing communication.
3. Offering diplomates more questions relevant to their practices.
4. Sponsoring independent research to assess the validity of examinations in measuring competence and improving patient care.
5. Minimizing costs of initial and continuing certification.
6. Increasing integration of stakeholders into governance.
7. Re-evaluating specialty-specific pathways for certification.
8. Diversifying the gender, racial and ethnic composition of the ABR executive boards
9. Ensuring well-balanced processes for exchange between individual radiologists and the certifying board.

The ACR, in representing its members, is eager to collaborate with the ABR to help effect reform.

Since 1934, the American Board of Radiology (ABR) has conducted certification examinations for radiologists, with the profession universally accepting this certification as the culmination of radiologists’ training. The ABR operates under the auspices of the American Board of Medical Specialties (ABMS), which supervises 24 medical boards covering 40 recognized specialties and 87 subspecialties. The ABMS is responsible for overseeing and enforcing the standards, policies and programs for continuing certification programs of the Member Boards. Under the supervision of its Committee on Continuing Certification (3C), the ABMS allows member boards to design unique Maintenance of Certification (MOC^®) programs and determine participation requirements [1,2].

ACR members have questioned whether the ABR and its programs satisfy their mission “to certify that [our] diplomates demonstrate the requisite knowledge, skill and understanding of their disciplines to the benefit of patients” [3]. Many have challenged evidence that these programs enhance patient care and improve outcomes. In addition, some believe that requirements are unnecessarily burdensome, contribute to burnout [4], are not uniformly relevant [5–9], and are costly [10].

In response to such comments by ACR members, early in 2019 ACR leadership established the Task Force on Certification in Radiology with the charge of researching the history of certification; examining best practices for certification; developing membership surveys; and formulating recommendations on how certification might further advance continuous learning, support the profession and improve patient care.

This report primarily focuses on Continuing Certification, heretofore known as MOC, but will also discuss:

1. Initial certification
2. The history of medical certification
3. Specialty certifications within radiology
4. How other medical specialties and professions address certification and recertification
5. The legal and regulatory environment
6. Beginning steps to certification reform

In recent decades, health systems, practice groups, medical corporations and other medical organizations have incorporated mandatory certification into their bylaws and regulations to ensure clinical privileges, employment and insurance payments. (See Appendix 1 and Table 1 with the appendices for a detailed description of medical and radiologic certification.)

Radiologists certified by the ABR prior to the implementation of time-limited certificates are not required to comply with ABR MOC requirements. Over time, rules requiring physician certification were issued by organizations such as the American Hospital Association, the Accreditation Council for Graduate Medical Education (ACGME) and malpractice insurance companies [11]. Notably, board certification is not a requirement of the Centers for Medicare and Medicaid Services [12,13]. In addition, a status designated on the ABR website as “not meeting requirements of MOC” could lead to institutions denying or revoking privileges to practice as well as terminating payments, even in the absence of misconduct or incompetence [14,15].

Continuous Certification and Online Longitudinal Assessment (OLA)

The current radiology Continuous Certification formula consists of four components: Part 1: Professionalism and Professional Standing, Part 2: Lifelong Learning and Self-Assessment, Part 3: Assessment of Knowledge, Judgment, and Skills and Part 4: Improvement in Medical Practice; although these divisions may change with implementation of the Vision Commission’s recommendations.

Previously, for MOC Part 3, the ABR required diplomates to take high-stakes, high-security examinations in designated testing centers every 10 years. “In 2012, the ABR’s MOC program changed from a 10-year cycle to Continuous Certification. Under Continuous Certification, new ABR certificates no longer have "valid-through" dates; instead, the date of initial certification is noted, accompanied by the statement that "ongoing validity of this certificate is contingent upon meeting the requirements of Maintenance of Certification." Progress is evaluated annually in March, using a rolling calendar-year “look-back” of the past three years” [16].

In May 2016, the ABR announced a change from the 10-year examination to OLA, which was first introduced in 2019 [17]. All diplomates with time-limited certificates must participate in OLA as available for their specialty [17]. With OLA, diplomates create an online practice profile of their specialty/subspecialty area(s). They receive two question opportunities each week and, in most cases, must answer 52 questions per year. Diplomates are allowed a limited time to answer each question. They receive immediate feedback with rationale for the correct answer and a reference. An incorrect answer prompts a follow-up question in subsequent weeks. After answering 200 questions, OLA performance will be evaluated at the MOC annual review. The passing standard for OLA is criterion-referenced. The aggregate rating for a diplomate’s 200 questions determines their passing score. If below the passing standard, MOC status will change to “not meeting requirements,” and the diplomate will have one year to remediate.

Other Certifying Boards

The American Osteopathic Association administers radiology board certification for doctors of osteopathy under the American Osteopathic Board of Radiology [18]. The American Board of Physician Specialists, as the certifying body of the American Association of Physician Specialists, also offers ongoing certification [19]. Physicians’ concerns regarding accountability of ABMS member boards led to the 2015 formation of a competitive board, the National Board of Physicians and Surgeons, which provides an MOC program [20]. (See Appendix 2 for more on alternative boards.)

ABMS Vision Commission

Appreciating physicians’ concerns about MOC, the ABMS constituted the independent multistakeholder ABMS Vision Commission in 2018. The commission provided an open public comment period on the preliminary report, leading to an opinion letter from the ACR [21]. Following the release of the final report in February 2019, the ABMS and its member boards formed five task forces to address the commission’s recommendations [22].

Online appendices include discussions of certification and recertification programs in the specialties of interventional radiology (Appendix 3), nuclear medicine (Appendix 4), physics (Appendix 5) and radiation oncology (Appendix 6).

We also compared the certification processes of other ABMS member boards as well as those of other professions. We calculated overall costs for initial certification and over a 30-year career. Costs ranged from $440 (family nurse practitioner) to $27,720 (prosthodontist) [23,24]. Of note, the 30-year career cost for radiologists for the ABR is $14,680, one of the highest among medical specialties, compared to $7,100 for pathology and $11,165 for orthopedic surgery.

See Appendix 7 for a comprehensive description of the approaches of other medical specialties and nonmedical professions. See Table 2 for a listing of features of multiple specialties’ and professions’ certification processes.

The ABMS Vision Commission report notes that CME alone is “insufficient to ensure [certified physicians] remain up to date” [22]. Representatives of the ABR assert that the exponential growth of knowledge in medicine requires more effective methods to ensure a physician’s skills, knowledge, and judgement remain current [25]. MOC proponents also emphasize that patients expect their physicians to continue to show competence. In 2003, the American Board of Internal Medicine contracted the Gallup Organization to perform a telephone survey of the public. Results showed that “most adults (79%), once informed of what certification involves, feel that recertification of physicians is very important” [26]. In 2010, the ABMS commissioned another such survey by the Opinion Research Corporation. Respondents were informed that MOC is “a process by which doctors who are Board Certified continue to participate in a continuous process of lifelong learning and self-assessment in their specialties” [27]. Of respondents, 91% stated that this was important, but only 23% correctly defined MOC prior to being informed. Informing survey respondents about board certification in advance represents unblinding and introduces a potential for unexplained variance. Although board certification is one factor in selecting a physician, our focus groups and an informal in-person survey suggest that the public prioritizes other factors as well.

The efficacy of MOC is controversial. A cross-specialty survey of physicians in 2016 found that 81% felt that MOC was a burden, and only 9% believed that patients care about their MOC status [28]. The 2018 Survey of America’s Physicians: Practice Patterns & Perspectives, conducted by Merritt Hawkins on behalf of The Physicians Foundation, found that more than two-thirds disagreed or strongly disagreed that MOC accurately assessed their clinical abilities. Also, 95% indicated that MOC contributed significantly or very significantly to burnout, and over 68% of physicians “do not believe that the criteria on which they are judged to maintain board certification, which is a critical component of sustaining their livelihoods and reputations, is valid” [4].

Time-limited certificate-holders have also expressed dissatisfaction that older radiologists are exempted from MOC, demonstrating discrepancies in board certification standards [29]. A study of more than 20,000 radiologists found that nearly all radiologists with time-limited certificates participated in MOC, but only 14% of “grandfathered” diplomates in diagnostic radiology chose to enroll [30,31]. It is unknown whether grandfathered diplomates believe MOC can help maintain skills and knowledge, whereas others are mandated to do so.

ABR “Agreement”

The ABR recently asked all candidates for initial certification and diplomates participating in MOC to sign an “Agreement for Candidates and Diplomates” (see Appendix 8). Not signing could have damaging consequences, possibly revoking the right to continue OLA or to complete the initial certification process; however, the ramifications of refusing to sign were not clear. The ABR acknowledged concerns and withdrew and revised the agreement.

In response to this controversy, at the 2020 ACR Annual Meeting, the ACR Council unanimously adopted Resolution No. 50, establishing the official ACR position that certifying boards should minimize power imbalance by committing to representative, inclusive and transparent decision making and should never require radiology professionals to waive their fundamental due process rights. It also stated that certifying boards “should seek … input from the ACR Council Steering Committee as a representative body of candidates for certification and diplomates prior to … implementing new and/or existing policies …” [32].

ABR Examination Delays Due to COVID-19

As the effects of COVID-19 became apparent in early March 2020, the ABR appropriately announced plans to delay the normal in-person administration of its written and oral examinations to ensure the safety of candidates and staff. Many individuals and organizations expressed concerns that continued delays, uncertainty, and lack of remote testing options were disruptive to both candidates and training programs, and a letter was sent to the ABR from the Multispecialty Early Radiologic Career Coalition, signed by 17 constituent organizations and followed by an ACR letter of support [33].

The ABR was responsive to these concerns and, after further consultation with stakeholders including the Association of Program Directors in Radiology (APDR), announced on June 22, 2020, that it will be “moving all currently unscheduled and future oral and computer-based exams to virtual platforms beginning in the first half of 2021” [34]. This is a positive outcome that came from communication and collaboration with stakeholders.

Testing Accommodations for Pregnant and Lactating Candidates

There are concerns that the ABR does not have sufficient accommodations for pregnant and lactating candidates [35]. Magudia et al (2020) write: “We hope the ABR’s openness to embrace virtual examination will further drive the development of a fair and equitable examination process, including the consideration of multiple testing dates throughout the year to accommodate family planning or other major life events” [35]. Testing accommodations for lactation are based on a 2013 ABR policy, which allows lactating women an additional 30 minutes and states that private rooms with electrical outlets are not guaranteed [36]. Following the editorial by Magudia et al, the ABR responded “it hopes future plans to offer online test-taking will begin to allay some of these challenges,” eliminating “the need to travel, making the process safer during and after the pandemic.” They pledged to "actively" seek input from stakeholders and focus on similar concerns in its analyses,” also stating “we remain committed to fully supporting nursing mothers in the radiology community” [37]. With the transition to virtual testing, this is an opportunity for the ABR to review and revise their lactation accommodation policy to further support nursing mothers.

Growing Use of Certification Requirements for Practice

The requirements for board certification have accompanied an increasingly complex healthcare system. Because patients are covered by a variety of carriers, if even a small number of patients in a health system are insured by a company that requires certification, likely they will require all of their physicians to be certified. Because licensing, employment, participation in provider panels and clinical privileging all must align, it is rarely practical for a physician with a time-limited certificate to decline to comply with MOC [7,38,39].

The National Committee on Quality Assurance (NCQA) is a private, nonprofit organization that accredits most major insurance plans in the US. NCQA’s Healthcare Effectiveness Data and Information Set, which includes board certification, partially determines insurers’ accreditation [40–43]. Presumably, health systems require MOC partly because NCQA evaluates insurers based on their physicians’ compliance with MOC.

The change in terminology of certification to include MOC has created ambiguities in regulations for many organizations because innumerable legacy bylaws and regulations that refer to “certification” may now unintentionally refer to initial certification plus MOC. Potential consequences of noncompliance include termination, losing credentials as an insurance provider and being denied as an expert witness. Interestingly, these publicized consequences have largely resulted from physicians refusing to pay their MOC fees, not because they paid and failed to meet requirements [11,14,15].

AMA Policy Principles on MOC

Given the controversies surrounding continuing board certification, the American Medical Association (AMA) adopted AMA Policy H-275.924, which states that “The CBC [Continuous Board Certification] program should not be a mandated requirement for licensure, credentialing, recredentialing, privileging, reimbursement, network participation, employment, or insurance panel participation.” Also, “Our AMA will continue to work with the national medical specialty societies to advocate for the physicians of America to receive value in the services they purchase for CBC from their specialty boards. Value in CBC should include cost effectiveness with full financial transparency, respect for physicians’ time and their patient care commitments, alignment of CBC requirements with other regulator and payer requirements, and adherence to an evidence basis for both CBC content and processes” [44].

Activities Related to State and Federal Anti-MOC Laws

In recent years, multiple physician groups have initiated efforts to pass state laws limiting or outlawing the use of MOC for various purposes. Such groups claim that MOC requirements 1) overlap with existing state-imposed continuing medical education (CME) requirements, 2) are costly, and 3) have become a mandatory prerequisite to medical practice [45,46]. Multiple states have passed laws restricting the use of MOC with varying levels of strength and often with substantial exceptions.

While debating an anti-MOC bill being considered in the Maryland state legislature [47], the U.S. Department of Justice Antitrust Division was asked to provide an opinion. The response letter [48] included a recommendation to consider facilitating competition among board-certifying bodies, and to redefine board certification, so physicians still qualify as certified if they do not participate in MOC.

Federal lawsuits, mostly class-action, have been brought against ABMS and several member boards, including the ABR. Arguments include 1) antitrust violation, 2) restraint of trade, 3) unjust enrichment, 4) fraud, 5) racketeering under the Racketeer Influenced and Corrupt Organizations (RICO) Act and 6) illegal tying of initial certification to a “Continuous Professional Development” product (MOC). (See Appendix 9 for details on legislative actions and federal lawsuits.)

Cost

A key complaint of radiologists is high fees for ABR services [49]. Residents and fellows, who are often burdened by large student debts [50], pay $640 per year, which on average represents about 1% of their annual income. If they do not pass the original examination, they are charged an additional re-examination fee. Historically, residents also paid for travel and lodging expenses.

The detailed budgets of the ABR are not accessible. However, according to the ABR’s IRS Form 990 for the fiscal year ending December 2018, the total assets of the ABR were $42.6M. Revenue in 2018 was about $14M with $13.4M from certification fees. Diplomates currently pay an annual fee of $340, accounting for a majority of the ABR’s revenue, which predictably increases because all newly certified radiologists must enroll in MOC [50,51]. Fees have remained unchanged for initial certification since 2016 and MOC since 2015. We acknowledge there is a cost of doing business and reserves are necessary but increased transparency and cost effectiveness are encouraged.

Evidence of Value

Radiologists are also concerned that there is absence of scientific evidence of value. Most studies of MOC or even initial certification are nonblinded and retrospective, often conducted by representatives of the ABMS and other member boards. One review suggested that “physicians who are board certified provide better patient care, albeit the results have modest effect sizes and are not unequivocal” [52]. Literature reviews for nonradiology specialties did not uncover any study that highly associated MOC with improved patient outcomes or better physician processes. Even studies that suggested equivocal improvement usually found poor performance by both physicians who did and did not comply with MOC [53–56]. No available studies address radiologists’ performance associated with MOC. One of the ABMS-constituted Vision Commission’s recommendations was that the ABMS and member boards undertake research to evaluate the effectiveness of MOC.

The 2019 ACR Annual Meeting open-mic session was designated to discuss MOC, and participants shared numerous complaints. Additionally, the ACR Task Force on Certification convened two focus groups (14 ACR members) in January 2020 by video conference to explore perceptions of MOC (Appendix 10). Positive perceptions of ABR certification included the need for well-respected certification, and that OLA is an improvement over the 10-year examination. However, participants also felt that 1) some questions are too general or not relevant, 2) OLA has little effect on improving their quality of work and 3) there is little scientific evidence that MOC is beneficial. Although such responses are useful, a broadly distributed survey is in development.

The primary purpose of this report is to address opportunities for improvement in radiology certification. As medical professionals, we are committed to lifelong learning, continuous improvement and demonstrating value for our patients.

Assessment Methods and Learning

We congratulate the ABR for modernizing its testing platform for MOC Part 3. The move to OLA is a responsive change from feedback. However, we are not aware of any theory or research that supports how the annual completion of 52 online multiple-choice questions (MCQ) demonstrates professional competence. Migration to the remote online format from in-person testing has received mostly positive responses. However, technical problems with the new format have been perceived as avoidable, and ABR’s messaging and formal responses have not always been satisfactory.

When the indivisible relationship of testing to learning is appreciated and integrated into planning, organizations can optimize reforms in certification testing and processes. The priority placed on teaching for the certification tests strongly influences a resident’s ability to master certain skills while subordinating other skills and knowledge. The literature shows that studying for the core examination during residency emphasizes the MCQ format [57], sacrificing Socratic case-review (“hot-seat”) sessions in some programs, which simulate clinical practice more closely than MCQ-based conferences. Instructing residents how to prepare for the core examination, the APDR has stated “although the examination is image-rich, it is computer-based, requiring no need to prepare for the "boardsmanship" aspect of the traditional oral examination” [58]. Furthermore, a correlation exists between the use of case-based, hot-seat conferences and the likelihood of passing the core examination: “Although the traditional oral format of case-based conferences was used frequently for teaching all residents, residents who failed reported receiving these types of conferences more frequently than passing residents” [59]. Scores on standardized tests have been shown to correlate better with each other than with professional performance [60]. Practical radiology is difficult to assess by MCQs, requiring a much greater skillset of inquiry and judgement.

Research and Innovation in Learning and Assessment

Successful certification programs undertake early and independent research of assessment tools, prior to implementation. This is a vital step to ensure the accurate assessment of both learner competence and patient outcomes.

Medicine and radiology should not be limited by legacy methodology. Recognizing that learning and assessment are inseparable, the ABR has the opportunity to lead other radiology organizations, integrating emerging techniques such as peer-learning and simulation into residency programs [61]. Assessment techniques are most effective when they create authentic simulations of learners’ actual jobs, although such techniques can be time-consuming and resource-intensive to develop. For example, the ABR could collaborate with radiology organizations to promote integrated diagnostics, emphasizing principles of data science in its assessment tools, a move that would help position radiology at the forefront of artificial intelligence. The ABR has funds available through the ABR Foundation, not financed by candidate fees, that could support grants directed toward these efforts. In addition, many such organizations would welcome the opportunity to work more closely with the ABR, including the ACR as a representative organization.

Diversity

Incorporating principles of diversity and inclusion should be a top priority for improving radiology certification programs, as radiology has long failed to successfully recruit women and underrepresented minorities. Improving diversity within the radiology workforce can help combat health disparities and is believed to be crucial to improving equal access to healthcare in the U.S. [62]. Significant differences and challenges exist in assessing populations with differing demographics [63], even within the small cohort of radiology trainees, requiring a sensitive and responsive approach to the target population undergoing assessment, being free from bias, and carefully constructing examinations for balance and fairness.

Competition

The expense and expertise required to establish alternative certification organizations are substantial. However, technologic innovations as well as increased acceptance of telelearning, teleradiology and remote testing during the COVID-19 pandemic have eliminated some barriers.

Despite alternative certification boards, the market dominance of the ABMS and its member boards has been supported by a large infrastructure of organizations that influence radiologists’ practices. The ABR should welcome new entrants, perhaps by sponsoring products developed by other organizations to catalyze evolution, innovation and improvement to benefit patients.

Representation, Governance and Leadership

To maintain independence and assert validity, the specialty boards understandably maintain an arm’s length relationship from candidates and diplomates. But to achieve balance and fairness, these individuals’ psychosocial and professional needs should be recognized by the ABR, and the ABR should be responsive to concerns about the content and conduct of certification programs. Without robust feedback mechanisms, any certifying body’s relevance to radiologists and their patients may decline.

Our review of certification in other specialty and professional organizations found that some sensibly chose to reflect the diversity of the overall group in their voting members. Although the ABR meets regularly with leadership from external organizations, such as APDR, the ABR could better connect with its candidates and diplomates by reserving some voting member positions on their boards for various constituencies. Representatives could reflect the diversity of diplomates not only in race and gender, but also by such factors as geographic location, practice type and career stage. Successful certifying bodies also tended to be highly feedback-driven, continuously seeking, vetting and acting on constituent feedback, with most making communication central to their operations.

Certification bodies most supported by their constituents engaged and collaborated with stakeholders and were process-driven and transparent in both design and operation. The ABR’s process of prioritizing the content and categories in testing has sometimes been complex and opaque. Volunteers and employees from academic radiology make up a majority of ABR leadership and boards. APDR members are well-represented as item writers, but private practice is underrepresented. Several years ago, the ABR was one of the first ABMS boards to establish an Advisory Committee comprised of diplomates. Recently, the Advisory Committee has been expanded to include trainees as well as academic and private practice physicians from all four disciplines. This is a positive step toward improving stakeholder input and communication. In addition, the ABR added a public member to the Board of Governors, which provides an important perspective and ties back to the core mission. These developments should be publicized to radiologists so that everyone is aware of the efforts being made to improve stakeholder input and inclusiveness. The ABR should continue consulting with APDR, the Association of Program Directors in Interventional Radiology, ACGME Review Committee, appropriate specialty and subspecialty organizations, and others regarding the content and complexity of the examinations. The ACR is committed to representing our members in a collaborative effort as the ABR continues to work toward implementation of the Vision Commission’s recommendations.

1. Iglehart JK, Baron RB. Ensuring physicians’ competence — is Maintenance of Certification the answer? N Engl J Med. 2012;367(26):2543–2549.
2. American Board of Medical Specialties. Standards for the ABMS Program for Maintenance of Certification. https://www.abms.org/wp-content/uploads/2020/11/standards-for-the-abms-program-for-moc-final.pdf. Published January 15, 2014. Accessed January 24, 2022.
3. The American Board of Radiology. Our history. https://www.theabr.org/about/our-history. Updated May 16, 2017. Accessed July 9, 2020.
4. The Physicians Foundation. 2018 Survey of America’s Physicians: Practice Patterns and Perspectives. September 2018. https://physiciansfoundation.org/wp-content/uploads/2018/09/physicians-survey-results-final-2018.pdf. Accessed July 9, 2020.
5. Glazier J, Kaki A. Recertification: a tale of good intentions but lots of strife. Am J Med. 2019;132(1):e1–e2.
6. Teirstein PS, Topol, EJ. The role of Maintenance of Certification programs in governance and professionalism. JAMA. 2015;313(18):1809–1810.
7. Jha S. Point: twin dogmas of Maintenance of Certification. J Am Coll Radiol. 2015;12(5):430–433.
8. Rosenkrantz AB, Hawkins CM, Ryu RK, Duszak, Jr. R. Clinical practice characteristics of radiologists based on American Board of Radiology Interventional Radiology Certification Status. AJR. 2020; 214:149–155.
9. Rosenkrantz AB, Nicola GN, Hirsch JA, Duszak, Jr R. Board certification characteristics of practicing neuroradiologists. Am J Neuroradiol. 40:1610–1616.
10. Sandhu AT, Dudley RA, Kazi DS. A cost analysis of the American Board of Internal Medicine's Maintenance-of-Certification Program. Ann Intern Med. 2015;163:401–408.
11. The National Board of Physicians and Surgeons. Letters, emails, and articles from doctors who have been harmed by MOC. https://nbpas.org/advocacy-center/harmed-by-moc/. Accessed July 10, 2020.
12. McCullough AC, Nsien IE, Chilton AD. Maintenance of certification: to MOC or not to MOC? https://www.natlawreview.com/article/maintenance-certification-to-moc-or-not-to-moc. Published September 20, 2019. Accessed July 9, 2020.
13. Centers for Medicare and Medicaid. Physician quality reporting system Maintenance of Certification program incentive guidance. https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/PQRS/downloads/MOC_Guidance_Final1_2_3.pdf Accessed July 10, 2020.
14. Edison M. My MOC failure. https://megedison.com/my-moc-failure. Published December 3, 2017. Accessed July 10, 2020.
15. Edison M. My MOC victory. https://megedison.com/my-moc-victory/. Published May 15, 2019. Accessed July 17, 2020.
16. The American Board of Radiology. MOC FAQs. https://www.theabr.org/diagnostic-radiology/maintenance-of-certification/moc-faqs. Updated June 19, 2017. Accessed July 10, 2020.
17. The American Board of Radiology. Online longitudinal assessment (OLA). https://www.theabr.org/online-longitudinal-assessment-ola Updated June 2, 2020. Accessed July 10, 2020.
18. The American Osteopathic Association. The American Osteopathic Association website. https://certification.osteopathic.org. Accessed July 15, 2020.
19. American Board of Physician Specialties. About the American Board of Physician Specialties. https://www.abpsus.org/about-abps. Accessed July 10, 2020.
20. The National Board of Physicians and Surgeons. The National Board of Physicians and Surgeons website. https://NBPAS.org. Accessed July 15, 2020.
21. American College of Radiology. Re: Continuing board certification: vision for the future commission draft report for public comment [letter]. https://www.acr.org/-/media/ACR/Files/Commissions-and-Committees/Comment-letter-to-Vision-Initiative-Commission.pdf. Published January 15, 2019. Accessed July 10, 2020.
22. American Board of Medical Specialties. Continuing board certification: vision for the future commission: final report. https://visioninitiative.org/wp-content/uploads/2019/02/Commission_Final_Report_20190212.pdf. Published February 12, 2019. Accessed July 10, 2020.
23. The American Board of Prosthodontics. The American Board of Prosthodontics website. https://www.abpros.org. Accessed July 14, 2020.
24. The American Association of Nurse Practitioners. The American Association of Nurse Practitioners website. https://www.aanp.org. Accessed July 14, 2020.
25. Irons MB, Nora LM. Maintenance of certification 2.0—strong start, continued evolution. N Engl J Med. 2015;372(2):104─106.
26. Kempen PM. Why do patients select and stay with their doctor? Implications regarding board certification and Maintenance of Certification and of licensure (MOC/MOL). J Am Phys Surg. 2012;17(2):53─56.
27. American Board of Medical Specialties. Facts about the ABMS consumer survey: lifelong learning and other qualities in choosing a doctor. https://dlpgnf31z4a6s.cloudfront.net/media/40431/abms-2010-consumer-survey.pdf. Published 2010. Accessed January 24, 2022.
28. Cook DA, Blachman MJ, West CP, Wittich CM. Physician attitudes about Maintenance of Certification: a cross-specialty national survey. Mayo Clin Proc. 2016;91(10):1336─1345.
29. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142(4):260─273.
30. Rosenkrantz AB, Berland LL, Heitkamp DE, Duszak Jr. R. Reply to "Broadening Stakeholder Perspectives on Maintenance of Certification Research." Am J Roentgenol. 2020;214(5):W83.
31. Rosenkrantz AB, Berland LL, Heitkamp DE, Duszak Jr. R. Diagnostic radiologists’ participation in the American Board of Radiology Maintenance of Certification program. Am J Roentgenol. 2019; 213:1284–1290
32. American College of Radiology. Resolution No. 50. ACR Position on Certifying Boards in Radiology. https://www.acr.org/-/media/ACR/Files/Annual-Meeting/ACR-2020/Res-50--ACR-Position-on-Certifying-Bodies-in-Radiology.pdf. Published May 2020. Accessed July 10, 2020.
33. Stempniak M. American Board of Radiology leadership meeting this week to mull virtual test-taking options. https://www.radiologybusiness.com/topics/leadership/american-board-radiology-governors-virtual-test. Published June 15, 2020. Accessed July 23, 2020.
34. The American Board of Radiology. ABR moving to remote exams. https://www.theabr.org/announcements/coronavirus-updates#updates. Published June 22, 2020. Accessed July 24, 2020.
35. Magudia K, Smith E, Harrington SG, et al. Carpe Diem: An opportunity for the ABR to support its trainees with family-friendly policies. Clin Imaging. 2021;69:148–149.
36. American Board of Radiology. Provisions for special requests regarding ABR examinations. https://www.theabr.org/wp-content/uploads/2017/06/Provisions-for-Special-Requests.pdf. Published October 18, 2013. Accessed July 15, 2020.
37. Stempniak M. American Board of Radiology responds to bathroom breast-pumping controversy. https://www.radiologybusiness.com/american-board-radiology-bathroom-breast-pumping. Published July 8, 2020. Accessed July 23, 2020.
38. Kempen PM. Maintenance of certification and licensure: regulatory capture of medicine. Anesth Analg. 2014;118(6):1378─1386.
39. Jackson VP, Balfe DM, Guiberteau MJ. Counterpoint: why things are going right with the ABR examinations. J Am Coll Radiol. 2016;13(11):1361─1362.
40. Fisher WG. US physician ABMS board certification: from voluntary to mandatory. http://drwes.blogspot.com/2018/07/us-physician-abms-board-certification.html. Published July 18, 2018. Accessed July 10, 2020.
41. National Committee for Quality Assurance. About NCQA. https://www.ncqa.org/about-ncqa/. Accessed July 10, 2020.
42. American Board of Medical Specialties. What is ABMS board certification? https://www.abms.org/board-certification. Accessed July 10, 2020.
43. Texas Medical Association. S.B. 1148 (2017) - Maintenance of Certification. https://tma.custhelp.com/ci/fattach/get/89053/0/filename/MOC+white+paper+1+29+2018.pdf. Published January 29, 2018. Accessed July 20, 2020.
44. American Medical Association. Licensure and discipline. Continuing Board Certification H-275.924. https://policysearch.ama-assn.org/policyfinder/detail/MOC?uri=%2FAMADoc%2FHOD.xml-0-1902.xml. Accessed July 10, 2020.
45. Schaeffer-Helmecki J. “Anti-Maintenance of Certification” laws. https://www.cga.ct.gov/2019/rpt/pdf/2019-R-0222.pdf. Published October 23, 2019. Accessed July 10, 2020.
46. Association of American Physicians and Surgeons (AAPS). https://aapsonline.org/moc-legislation-tracking. Published October 1, 2018. Accessed July 17, 2020.
47. The Maryland Health Care Commission. Maintenance of Certification Workgroup. https://mhcc.maryland.gov/mhcc/pages/home/workgroups/workgroups_maintenance_cert.aspx. Updated November 7, 2018. Accessed July 10, 2020.
48. Department of Justice. Comments on Maryland House Bill 857 [Letter]. https://www.justice.gov/atr/page/file/1092791/download. Published September 10, 2018. Accessed July 10, 2020.
49. Sandhu AT, Dudley RA, Kazi DS. A cost analysis of the American Board of Internal Medicine's Maintenance-of-Certification Program [published correction appears in Ann Intern Med. 2016;165(7):532]. Ann Intern Med. 2015;163(6):401–408.
50. White B. A deep dive into the tax returns of the American Board of Radiology. http://www.benwhite.com/radiology/a-deep-dive-into-the-tax-returns-of-the-american-board-of-radiology. Published August 28, 2019. Accessed July 10, 2020.
51. Young R. Board certification brouhaha: rads cry foul over ABR’s ways and means. https://www.radiologybusiness.com/topics/practice-management/board-certification-brouhaha-rads-cry-foul-over-abrs-ways-and-means. Published April 30, 2019. Accessed July 9, 2020.
52. National Board of Physicians and Surgeons. Journal club MOC literature review. https://nbpas.org/wp-content/uploads/2018/06/MOC-Journal-Club_2018-05-30.pdf. Published May 30, 2018. Accessed July 10, 2020.
53. Gray BM, Vandergrift JL, Johnston MM, et al. Association between imposition of a Maintenance of Certification requirement and ambulatory care-sensitive hospitalizations and health care costs. JAMA. 2014;312(22):2348–2357.
54. Hayes J, Jackson JL, McNutt GM, Hertz BJ, Ryan JJ, Pawlikowski SA. Association between physician time-unlimited vs time-limited internal medicine board certification and ambulatory patient care quality. JAMA. 2014;312(22):2358–2363.
55. Teirstein PS. Boarded to death—why Maintenance of Certification is bad for doctors and patients. N Engl J Med. 2015;372(2):106–108.
56. Sharp LK, Bashook PG, Lipsky MS, Horowitz SD, Miller SH. Specialty board certification and clinical outcomes: the missing link. Acad Med. 2002;77(6):534–542.
57. Awan OA. Have we done radiology trainees a disservice by eliminating the oral board examination? RadioGraphics. 2020;40:605–606.
58. Chapman T, Reid JR, O'Connor EE. Anonymity and electronics: adapting preparation for radiology resident examination [published correction appears in Acad Radiol. 2018;25(5):683]. Acad Radiol. 2017;24(6):657–663.
59. Shetty AS, Grajo JR, Decker S, et al. ABR Core examination preparation: results of a survey of fourth-year radiology residents who took the 2013 examination. Acad Radiol. 2015;22(1):121–129.
60. Patel MD, Tomblinson CM, Benefield T, et al. The relationship between US medical licensing examination step scores and ABR core examination outcome and performance: a multi-institutional study. J Am Coll Radiol. 2020;17(8):1037–1045.
61. Harden RM. Trends and the future of postgraduate medical education. Emerg Med J. 2006;23(10):798–802.
62. Institute of Medicine. 2004. In the Nation's Compelling Interest: Ensuring Diversity in the Health-Care Workforce. Washington, DC: The National Academies Press.
63. Edmond MB, Deschenes JL, Eckler M, Wenzel RP. Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews. Acad Med. 2001;76(12):1253–1256.