Notes from the Chair: Theodore M. Freeman, MD
6/14/2019
Theodore M. Freeman, MD, Chair
By now, everyone is familiar with the Continuous Assessment Program (CAP) Pilot.
This program was developed by the ABAI Board, in response to a survey given to our diplomates, driven by the principle that any good organization should continually try to improve the processes by which it functions. The survey established that the overwhelming majority of our diplomates desired a change from the 10-year high stakes recertification exam. Given current learning and education, theory stipulates that learning is facilitated by frequent assessment, and under the leadership of our immediate past President Dr. Stephen Wasserman, MD, the board developed CAP. This was instituted in January 2018, during the chairmanship of Dr. Kathleen May, MD.
Interestingly, many of our fellow boards were noting dissatisfaction with the Maintenance of Certification (MOC) process as established by the ABMS. Consequently and concurrent with CAP, the ABMS established the Continuing Board Certification: Vision for the Future Commission to examine and make recommendations about MOC. This commission was composed of a variety of stakeholders to encompass all parties interested in the process of continuing certification. Our current President, Dr. Michael Nelson, MD, PHD, was asked to be a member of this commission. The commission completed its work last year, and made recommendations that the ABMS Board has adopted and is proceeding to implement; these recommendations included moving away from high-stakes re-certification exams, to more frequent assessments that reflect current learning (formative assessments). It was recommended that these formative assessments not be the sole determinant of a summative assessment (i.e. continuing assessment) of a diplomate’s ability. It is encouraging to note that the ABMS Commission’s recommendations exactly reflected what we have implemented at the ABAI.
Up to this point, the articles selected and the questions asked for CAP, have been initially developed by a sub-committee of the ABAI Board of Directors called the MOC Committee. One director is assigned to each of the eight categories in the MOC rubric. They are asked to select five recent articles in any of the journals with whom we have obtained an agreement (primarily from JACI, JACI in Practice, Annals of Allergy, Asthma and Immunology, NEJM). A “recent” article usually means published within the last two years, although exceptions may be made for articles of critical importance. They have to write multiple questions about each article in the typical multiple choice, one correct answer format. Each answer must be clearly be found in the article. They have to also write six core questions from the same category. The specification is that the answer should be easily verifiable in a major medical text. They’re then required to write a rationale for each of the core questions by specifying why the answer is correct and all of the other options are wrong.
These questions are then shared with another board member who reviews the questions and verifies that the answers are found within the specified text, and suggestions are made for the author to implement. All of the questions are then reviewed during a face to face meeting, and each question is read aloud, considered, edited and either accepted or rejected by the MOC Committee. If there are an insufficient number of questions remaining, the article is eliminated from consideration. The committee co-chairs then decide which of the available articles and questions are used to populate each category of the block.
At the end of the six-month block, each question is re-evaluated statistically by an independent contractor. Questions where there is no correlation between the correct answer and the answers chosen are eliminated, and everyone receives credit for it; this happens very rarely, about once per block, which is less than 1% of the total questions. If the analysis shows a significant proportion of responders picked a particular incorrect answer rather than the correct answer, the MOC co-chairs re-evaluate the question, and if there is a good reason for this duality, then both answers are accepted. This has occurred with about a half dozen questions for each block so far. In addition, the MOC co-chairs review each diplomate’s feedback, looking for additional reasons a question might be placed into one of the above categories. This too has happened occasionally in the first two blocks. It is important to note that this rigorous item review process may raise or keep the same individual block performance compared to the raw score received upon completion of CAP; however, the diplomates score will not be reduced by this process. In addition, this process relies on having a full data set; therefore, adjustments to questions and scores cannot occur until the block concludes.
Most recently, an External Writing Committee has been constituted to select CAP articles and write the initial questions which will be evaluated and vetted just as the questions written by the MOC Committee. These articles and questions will appear starting with Block 1 of 2020. This committee is currently composed of prior Board members, but we hope to include more of our diplomates in this effort in the near future. Both of our parent boards (ABP and ABIM) have utilized test writing committees for years. There are plans to re-survey our diplomates in the near future to try to continue to improve CAP. The goal is to provide a tool that is reasonably easy to use, that can assist each of us in keeping up with the important allergy-immunology literature, and whose results can be used by the Board to assure our patients that those continuing to certify are indeed practicing quality allergy and immunology.