Wait! Don’t go…I know it’s Spring, and you are enjoying that little break between recruitment season and anticipatory planning for the next academic year, but this is a quick read to help you get the most of that In-service training exam (ITE) that your fellows are about to take. Some of you, like me, might love receiving quantitative data on your fellows’ performance and already have established ways in which you interpret and share that information. A few of you might be new program directors, still building your review process. Luckily, the APCCMPD In-service exam writing committee launched new individual and program level score reports in the past year to simplify your ability to do this. If you are working on ways to maximize the use of the ITE data, then read on!

 

Where it all began

The PCCM and CCM In-service exams were established in 2007 and 2011, respectively. The exams are based on the ABIM Certification Exam blueprints and the PCCM exam was shown to be an independent predictor of performance on both the Pulmonary and Critical Care Medicine ABIM certification exams(1). A committee of current and former fellowship program directors reviews the blueprint carefully for any updates, ensures that items reflect current evidence, and works with a psychometrician to ensure results are statistically accurate and fair.

Originally offered to all 1st-, 2nd- and 3rd-year Pulmonary Medicine, Critical Care Medicine and combined PCCM fellows each Spring, it is now additionally offered in July to incoming fellows. A new exam is developed every year and consists of 125 scored items with an additional 25 unscored items for pre-testing. Each fellow receives their raw results immediately after completing the exam. At the end of the testing window, every program director receives summary score reports compared nationally for each fellow, as well as group comparisons by fellowship year. Here are a few tips on how you can engage with this data.

 

Help your fellows design their individual learning plans

The majority of you and your fellows are likely already using the ITE results in this way. Each individual score report has data by subcategory on the fellow’s performance compared nationally to fellows at the same level of training. The report is a guide to where they should place more of their attention when studying. Although many fellows have developed individualized study habits in their long professional journey, program directors should still review how they plan to advance their medical knowledge and assist them in designing a specific, targeted strategy. Options include:

  • reading the primary literature for that subcategory by using the study guide sent from the In-service Exam committee, the ATS Reading list(2), or a local reference list provided by your program
  • listening to podcasts and searching the immense Free Open Access Medical Education (FOAMed) resources on PCCM topics
  • practicing test questions in these subcategories
  • forming group study sessions with co-fellows to review shared areas of need

 

Ensure your fellow is ready for the subspecialty boards

The PCCM ITE is an independent predictor of performance on the subspecialty board examinations for Pulmonary Medicine and Critical Care Medicine. Interestingly, program director ratings on Medical Knowledge at the time of this validation study (before the ACGME Next Accreditation System was launched) were not an independent predictor of performance on board exams(1). For this reason, some program directors use ITE performance as an early warning sign for a fellow that may need additional support. In the study by Kempainen et al, second-year fellows that scored ³ 90 questions correct (72% correct) on the ITE had a greater than 95% probability of passing the subspecialty board examinations. However, the majority of fellows will pass the ABIM subspecialty board exam; in 2023–94% pulmonary and 85% critical care(3). This makes it difficult to demonstrate that poor performance on the ITE predicts failure on the board examinations, which is why the ITE should only be used as a low-stakes, formative assessment tool for medical knowledge(4). A low ITE score should still prompt a more nuanced evaluation of a fellow’s readiness for the Boards.

  • Are they studying? How are they studying?
  • Are there untapped study resources available that might help them?
  • Could they use support from a learning specialist?
  • Do they need help with time management?
  • How is their overall wellbeing?

 

Fine tune your program’s didactic curriculum

You should now also be receiving an institutional report at the end of the testing window period. These reports are designed to share group level data for your program in each subcategory. Some of you may have been compiling this data and analyzing it on your own before, but this report should save you some time!

Are there trends in your fellows’ performance data – perhaps your fellows universally score lower in Lung Transplantation? Or Cardiovascular Critical Care? Consider adding or improving how you cover these content areas in your curriculum. If the topics are primarily lecture-based, could the fellows’ learning be enhanced with more interactive methods or case-based discussions? If these content areas include less common diseases, such as cystic fibrosis, pulmonary hypertension, or obstetrical critical care, then are their opportunities to expand their clinical experiences to immerse them in the evaluation of these patients?

 

Integrate with comprehensive program data for a robust review of your program 

Share this data with your Program Evaluation Committee. Consider how your fellows’ performance on the ITE interconnects with the comprehensive metrics that you compile about your program. Is there an area of low performance that is surprising to you and the committee? Maybe your fellows rotate in ILD clinic or have a neurosurgery ICU rotation but score low in those categories. Listening to feedback from your fellow and faculty PEC members might help you to identify ways to enhance their learning.

Hopefully these tips are helpful, particularly for those of you who are new program directors. The new scoring reports should simplify your ability to engage with the ITE data in a meaningful way, and hopefully save a few of you from needing to develop your own excel sheets to track data!

 

References
  1. Kempainen RR, Hess BJ, Addrizzo-Harris DJ, Schaad DC, Scott CS, Carlin BW, Shaw Jr RC, Duhigg L, Lipner RS. Pulmonary and critical care in-service training examination score as a predictor of board certification examination performance. Annals of the American Thoracic Society. 2016 Apr;13(4):481-8.
  2. American Thoracic Society Reading List. https://www.thoracic.org/professionals/career-development/residents-medical-students/ats-reading-list/. Accessed February 22, 2024.
  3. American Board of Internal Medicine, Certification Pass Rates. https://www.abim.org/Media/yeqiumdc/certification-pass-rates.pdf. Accessed February 22, 2024.
  4. McCrary HC, Colbert-Getz JM, Poss WB, Smith BK. A systematic review of the relationship between in-training examination scores and specialty board examination scores. J Grad Med Educ 2021 Feb 1;13(1):43-57.

 

Paru Patrawalla, MD is an Associate Professor of Medicine at the Icahn School of Medicine at Mount Sinai and the Fellowship Program Director for Pulmonary and Critical Care Medicine at Mount Sinai Morningside-West. She serves as the Vice-Chair of the APCCMPD In-service exam writing committee and the past Chair of the Ultrasound Domain Task Force at CHEST. Dr. Patrawalla’s medical education interests include point-of-care ultrasonography, competency assessment and development of expertise.