This has been an incredibly challenging year for physicians, particularly in Pulmonary and Critical Care, as well as educators and program leaders.  Now, with the new academic year right around the corner, the ACGME Milestones 2.0 loom before you. What changes can you expect with this roll-out, and more importantly, how can you efficiently and effectively retool your evaluations structures for this updated framework? In this blog post, we will discuss the intent of the ACGME Milestones, highlight the updates included with the revised Milestones 2.0, and outline a concrete 8-step framework for transitioning to Milestones 2.0 within your fellowship program.

 

The What, When, and Why of Milestones 2.0

The ACGME Milestones are a systematic assessment framework for medical trainees using the six core competencies of physician practice. This framework was originally introduced in 2014 in ACGME-accredited subspecialty programs, with a goal to align with the larger conceptual shift to more competency-based medical education.  While these Milestones provided a necessary framework for this novel strategy of assessment, there were a series of challenges with their initial implementation.   The original Milestones were lengthy, verbose, lacked specificity, and used educational jargon (making it challenging to implement within fellowship programs).  The revision to Milestones 2.0, created via a committee of PCCM faculty, fellows, and relevant stakeholders, aimed to address and overcome these issues.   The overall goal of Milestones 2.0 was to provide simplified behavioral anchors, fewer subspecialty domains, and improved specificity for PCCM programs, ultimately aiming to improve assessment for program leaders and fellows alike.

 

The Updates (and Upgrades) of Milestones 2.0

The PCCM Milestones 2.0 Committee aimed to create a system that was specific to the field of PCCM, with specialty-specific updates to medical knowledge and patient care Milestones.  A Milestone focused on end-of-life conversations was also included in the revision.   “Harmonized” Milestones were developed for the domains of professionalism, interpersonal and communication skills, practice-based learning and improvement, and systems-based practice.  Beyond the specialty-specific edits to the Milestones, the Milestones were also significantly shortened (with reduced behavioral anchors, and reduced number of sub-competencies). The language was adapted to improve the overall interpretation (avoiding vague descriptors and avoiding educational jargon).

All PCCM programs (including Pulmonary and CC-only programs) are expected to implement these revised Milestones for the 2021 academic year (see the complete Milestones here: https://www.acgme.org/Portals/0/PDFs/Milestones/PulmonaryCriticalCareMilestones2.0.pdf?ver=2020-12-15-153735-067).

We recognize that revamping the Milestones can seem like a daunting task, so we have broken it down to the following 8-step framework.

 

Eight Step Recipe for Success: How to Transition from Milestones 1.0 to 2.0

Step 1: Review the Milestones and Supplemental Guide
First, prior to working with any evaluation tool or software (like MedHub, or New Innovations), we would recommend that you review the complete Milestones 2.0 (https://www.acgme.org/Portals/0/PDFs/Milestones/PulmonaryCriticalCareMilestones2.0.pdf?ver=2020-12-15-153735-067) and the associated ACGME Milestones 2.0 supplemental guide. (https://www.acgme.org/Portals/0/PDFs/Milestones/PulmonaryCriticalCareSupplementalGuide.pdf?ver=2020-12-16-084546-467) The Supplemental Guide is intended to provide educators and program leaders some concrete examples of the intent of each milestone, with associated examples of a fellow at each level (to assist the members of the Clinical Competency Committee) in PCCM programs.

Importantly, the ACGME also has a Milestones Guidebook (intended for fellows), a Clinical Competency Committee guidebook (for members of the CCC) that you can consider disseminating to the relevant groups in your institution.

We hope that after review of these documents, you will be pleasantly surprised at the revisions overall. Again, the goal is to improve specificity and overall utility for you, your programs, and your fellows!

Step 2: Read the Global Intent of the Milestones
Each milestone has a specified intent that aligns along the six core competencies.  For example, the intent of the “Patient Care 4: Pre-Procedure Assessment” was to ensure a fellow could “counsel patients regarding indications, risks, benefits, and alternatives of common procedures.”

As you review the suggested intent of each subcompetency (located in the ACGME Milestone Supplemental guide, https://www.acgme.org/Portals/0/PDFs/Milestones/PulmonaryCriticalCareSupplementalGuide.pdf?ver=2020-12-16-084546-467) begin to reflect on where these different milestones may be evaluated within your program.  In what rotations, inpatient or outpatient, or other experiences? Are there opportunities for EPAs (entrustable professional activities) to evaluate these domains? Are there opportunities for 360-degree feedback?

Step 3: Review your current rotations and current evaluations
Each Milestone rating should be determined by several evaluations (and different evaluation tools) to provide the full picture of a fellow’s skillset and competency level.  With that being said, after reviewing the changes in Milestones 2.0, we would recommend reviewing your current fellowship rotations (and associated evaluations) and reviewing your existing evaluation tools.  You may decide to keep the same evaluation tools as you update the link to Milestones 2.0, or may decide to take this opportunity to revise your evaluation tools for given rotations.  This process may highlight areas of your evaluation system and/or curriculum that need revision, so that this as an opportunity to consider some changes and improvements to your program.

Importantly, many of you have previously developed evaluation tools that aligned with Milestones 1.0.  Do not feel obligated to start from scratch! You can likely retool many of these old evaluations within the updated Milestones 2.0.  Milestones 2.0 is meant to guide your assessment framework and evaluation system, but should not serve as your sole assessment mechanism or evaluation system.

Step 4: Decide which to keep and where you need new evaluations
Again, with the new Milestones 2.0, you won’t have to necessarily reinvent the wheel entirely, but rather remap and revise some of your older evaluations. Look at your evaluations critically and decide which ones you want to keep, which ones you want to abandon, and consider if there are new opportunities for evaluation, like EPAs or 360-evals. This is an excellent opportunity to implement newer evaluations in different domains (such as more 360-evaluations from interprofessional team members).

Step 5: Create your evaluation map
After deciding on your evaluations, the next step will be to create an Excel spreadsheet where you match up each rotation with each milestone. This will highlight where the different milestones are being evaluated across different rotations. Here are examples of both an empty matrix and a filled in matrix:

 

Step 6: Check for competencies that aren’t measured
This process will most likely highlight certain subcompetencies that are not effectively measured with your current evaluations.  For example, in the revised Professionalism 3 milestone (Well-Being and Resiliency), an end-of-rotation evaluation is likely not an effective strategy to assess this domain.  Instead, you can embed this into the Program Director’s mid-year or end-of-year evaluation.  Other potential areas for assessment include through direct observation, group interviews, or self-assessment with a personal learning plan (as outlined in the ACGME Milestones 2.0 supplemental guide).  Importantly, each milestone should be assessed in at least three different areas, which can include end-of-rotation evaluations, OSCEs, 360 evaluations, among others.

Step 7: Think about novel areas for evaluation
Are there new opportunities for novel areas for evaluation, for example, EPAs or 360-degree evals? EPAs can be particularly useful for more discrete tasks, such as procedures, family meetings, or code leadership. 360-degree-evals can be useful to get input from interprofessional team members like nurses, APPs, bronchoscopy technicians, PFT lab technicians, and more.  Check out the ACGME Supplemental Guide for additional evaluation ideas, and be sure to check out the resources on the APCCMPD webpage (which include shared tools across programs).

Step 8: Input your mapped evaluations into your evaluation management software
The final step of the process is to work with the individuals in your institution to map your evaluations into the evaluation management software like MedHub or New Innovations. Don’t have a fancy software platform? You can use a simple Qualtrics survey to send to faculty that maps to the Milestones.

 

A Few Final Tips
  • Review the resources provided by the ACGME (both the main Milestones and the Supplemental Guide), as well as the resources provided by the APCCMPD.
  • Don’t reinvent the wheel, but don’t be afraid of change either
  • Be creative with your evaluations – some milestones are not easily assessed in end-of-rotation evaluations
  • Special note: If you have Pulmonary-only or Critical Care-only trainees, the Milestones are slightly different (in case you plan to use the same evaluations, the mapping may be slightly different).

 

Closing Thoughts

Remapping your Milestones can be an intimidating process, but we hope this 8-step framework will help demystify the process.  Want some more in-depth knowledge? Check out the APCCMPD website where some programs have posted their own evaluation forms, and feel free to contact us with additional questions. Again, be sure to review the relevant resources on the ACGME website, including the outline of the revised Milestones 2.0 and the associated Supplemental Guide.  There are also a series of articles on the JGME (Milestones 2.0 supplement) with additional input on the various domains, including CCC best practices.  Best of luck with your Milestones 2.0 Mapping process!

 

Further Reading
  1. ACGME Milestones 2.0: https://www.acgme.org/Portals/0/PDFs/Milestones/PulmonaryCriticalCareMilestones2.0.pdf?ver=2020-12-15-153735-067
  2. ACGME Supplemental Guide: https://www.acgme.org/Portals/0/PDFs/Milestones/PulmonaryCriticalCareSupplementalGuide.pdf?ver=2020-12-16-084546-467
  3. Journal of Graduate Medical Education Milestones 2.0 Supplement: https://meridian.allenpress.com/jgme/issue/13/2s
  4. Heath, J. K., Dempsey, T. M., Santhosh, L., Edgar, L., & Fessler, H. E. (2020). Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment. ATS Scholar1(1), 33-43.
  5. Edgar, L., Roberts, S., Holmboe, E. Milestones 2.0: A Step Forward. J Grad Med Educ 1 June 2018; 10 (3): 367–369.

 

Additional Resources
Download the Milestones and shared mapping matrices and evaluations here.

 

Lekshmi Santhosh, MD, MAEd is an Assistant Professor of Pulmonary/Critical Care Medicine and the Associate Program Director of the University of California San Francisco Pulmonary/Critical Care Fellowship Program. She participated in the Milestones 2.0 Workgroup for Pulmonary/Critical Care Medicine fellowships, along with Dr. Heath, and has previously published with Dr. Heath on this topic.
Janae Heath, MD, MSCE is an Assistant Professor of Medicine within the Pulmonary and Critical Care Division at the University of Pennsylvania and the Associate Program Director for the Internal Medicine Residency Program, and a core faculty member for the Pulmonary and Critical Care Fellowship program. Her academic interests are assessment in medical education, and she has participated in the Milestones 2.0 Workgroup for Pulmonary/Critical Care Medicine fellowships, as well as the Milestones 2.0 Workgroup for Internal Medicine residency.