Craig A. Piquette, MD
“Oh, how I hate evaluations with my boss!”
How many times have you said that? We often feel that those meetings are a waste of time and we never learn anything from them. Our fellows probably feel the same way but we have an opportunity to make the semi-annual evaluation a journey of exploration and a collaborative educational process. You and the fellow can explore deficiencies together and maybe uncover unexpected means by which they may be addressed. You also have an opportunity to get to know the fellow on a more personal level that can strengthen the bond between you, the mentor, and the fellow as mentee. The semi-annual evaluation requires the accumulation of a lot of data which provides a clearer picture of the educational processes within the fellowship. In the end, it should be as valuable to you, as the program director (PD), as it is to the fellow. It should never be a ‘trip to the woodshed’ but rather a valuable experience for the participants that provides an opportunity for a collegial conversation.
The semi-annual evaluation process begins with the collection of evaluations of the fellows done at the completion of each rotation in the previous six months. These evaluations come from many of the members of the teams with which they worked. Residents and advance practice providers evaluate the fellows on practice-based learning and improvement, system-based practice, interpersonal and communication skills and professionalism. These multi-source evaluations provide a different perspective than the faculty evaluations of fellows. Rotation and clinic attendings assess fellows on all six competencies including medical knowledge and patient care. To keep these evaluations manageable, only 7-9 milestones are evaluated during any given rotation. We created a matrix, similar to that used at Emory (see Reporting Matrix posted under the heading Toolkit/Mapping Reporting Milestones to Rotations at apccmpdscholars.org) to ensure that all 24 milestones are evaluated during a six-month period when all available evaluations are combined. Each milestone will be evaluated three or four times over that time. In addition, comments are solicited from the fellows’ patients at clinic check-out and these are collated in a separate document. Procedure lists for the previous six months are obtained along with attestations from the supervising attending. A list of scholarly activity for each fellow is compiled and includes local, regional, and national abstract presentations and program conference presentations. Attendance at the weekly Chest Conference is tracked and reviewed as part of the semi-annual evaluation. Finally, the fellow is asked to self-assess their progress towards attainment of all 24 milestones every six months. Fellows’ evaluations of faculty, rotations, and program organization occurs through a separate process organized by the PD and the Program Evaluation Committee.
Once the data for each fellow’s performance in the preceding six months is compiled, this data is used to fill in portions of the Global Assessment/Semi-Annual Evaluation Form and given to a core faculty member who did not directly supervise the fellow during that period. This faculty member reviews all the data and completes a short comment section on the Global Assessment Form for presentation at the Clinical Competency Committee (CCC) meeting. The role of this faculty member is to serve as an advocate for the fellow. They may suggest commendation or remediation depending on the tenor of the evaluations overall.
The associate program director (APD) serves as the chair of the CCC, as recommended by the ACGME, and the meeting agenda is arranged to allow approximately 10-minutes of discussion per fellow. For fellows who are experiencing difficulties or whose performance is not following the proper trajectory, more time may be allotted and additional meeting time may be required. The Global Assessment Form, which includes the domains of procedural competency, in-training exam performance, scholarly activity and homework from the last review, is available to each reviewer and the Committee’s discussion of each fellow’s performance in these areas is recorded in the minutes of the meeting. Once discussion of all the areas of assessment on the form are completed, discussion moves to the next fellow on the agenda. In this way, the CCC can review six fellows in one hour.
Once the CCC has completed their review and the comments of the CCC and their specific recommendations are recorded on the global assessment form for each fellow, the semi-annual performance evaluation with the fellow is scheduled. Thirty minutes is allotted per fellow to allow time for review and discussion with the PD or APD. There are five documents that are reviewed with the fellow: the Global Assessment Form, written comments from all evaluations, the procedure list with totals for the six months and for the entire length of fellowship, composite evaluations of each milestone by faculty as reviewed by the CCC, and the self-evaluation by the fellow of their performance on all 24 milestones. The fellow may request copies of any or all of the documents reviewed with them during the meeting. The PD or APD reviews these documents prior to meeting with the fellow to ensure a meaningful conversation. The focus of the meeting is on the Global Assessment Form and the faculty and fellow’s evaluations of milestone performance. The fellow should have seen the written comments from the evaluations at the end of each rotation and the fellows log their procedures and should know if the numbers recorded are consistent with what they have performed. These documents serve as a confirmatory mechanism to avoid surprises. The Global Assessment Form is reviewed with particular attention to the homework from the last meeting and the homework as defined by the CCC for the next six months. Procedural proficiency is reviewed and if proficiency is not established, the fellow is advised to enlist faculty to attest to their proficiency prior to the next review. In-training exam performance is reviewed in light of the probability of passing the certifying exam as defined by Kempainen et al. (Ann Am Thorac Soc. 2016 Apr;13(4):481-8). Study habits are discussed and if standardized testing has been a challenge, resources are suggested and a plan for follow-up established. Scholarly activity is reviewed with the fellow and plans for additional projects are discussed. Finally, the comments of the CCC are reviewed which included a global assessment of clinical skills and areas for improvement. These inform the homework for the upcoming six months. The final two documents to review are the faculty’s milestone evaluations and the fellow’s self-evaluation. Usually the assessments are quite close but a discrepancy is an opportunity for discussion with the fellow as to the reasons for the discrepancy. The fellows often comment that this discussion is very helpful. The meeting with the fellow ends with questions regarding any concerns they may have and progress towards plans for the future, such as career planning, job search, etc.
This process for the semi-annual evaluation has evolved over time and has not been subjected to scholarly scrutiny but overall it seems to satisfy the fellows. Many of the elements have been borrowed from others and some are available on the APCCMPD resource page. I want to thank my colleagues in the APCCMPD for their support and to my colleagues and fellows at the University of Nebraska who have taught me so much over my years as program director.
Craig A. Piquette, MD is Professor, Internal Medicine Division of Pulmonary, Critical Care, Sleep & Allergy at the University of Nebraska Medical Center. He participated in the development of the ACGME’s Internal Medicine Subspecialty Reporting Milestones.
Sunita Kumar, MD
I use the semi-annual evaluation meetings with the fellows not only to evaluate their clinical progress but also as a counseling/guidance session discussing career choices and wellness/burnout. Though wellness and career guidance are discussed during the course of the year at fellow meetings and workshops, the semi-annual meeting provides an opportunity to have a personalized discussion with each fellow. I use the checklist below to keep me organized through the meeting.
- Academic and clinical performance: Following the introduction of milestone reporting by the ACGME, we created new end-of-the-rotation evaluations, mapping the 24 milestones to different evaluations based on how the rotation is structured at our institution and including at least one milestone for each of the ACGME competencies (patient care and procedural skills, medical knowledge, practice-based learning improvement, system-based practice, professionalism and interpersonal and communication skills). For example, system-based practice (SBP4) is included for pulmonary consult rotation, while SBP1 and 2 are relevant for the MICU and transplant rotations. This allowed for all milestones to be accounted for without making the evaluation too arduous for faculty to fill out. We use New Innovations (NI) for our evaluations and our program coordinator generates a “milestone report” (collating the milestone ratings from all the evaluations) just before the semi-annual evaluation. We check to see if all milestones are accounted for in the report. In the event that there are some missing, we reach out to faculty with incomplete evaluations that will help fill the gap and/or account for the missing data (fellows with few clinical rotations due to research). Fellows involved in research have evaluations filled out by their research mentors semi-annually. Fellows’ attendance at educational conferences, pulmonary grand rounds, didactic lectures and journal club are included in the report to the Clinical Competency Committee (CCC). Fellows have to attend a minimum of 75% of each conference after allowing for excused absences (MICU and off-site rotations). At the CCC meeting faculty discuss each fellow’s progress, taking into account the milestone report from NI, faculty comments and other data such as procedure logs and 360 evaluations, and make a recommendation for each milestone which is subsequently submitted to the ACGME after it is reviewed by me. This final milestone report from the CCC is provided to fellows at the time of their meeting with me. During the meeting I orient the fellows to their milestone trends, showing them how they have progressed (or declined) since the last evaluation. I review if they are meeting program expectations and identify areas for improvement. For the semi-annual evaluations in June, I discuss the results from the in-service examination (also arranged through the APCCMPD) and the national graphs which indicate where they stand among their peers. For the fellows taking boards their scores are used to discuss their probability to pass boards (Ann Am Thorac Soc. 2016 Apr;13(4):481-8). At this evaluation fellows are also informed if they have been recommended for promotion to the next year by the CCC.
- 360 evaluations: I review evaluations by nursing and housestaff and highlight compliments or address concerns that have been reported.
- Procedure log: Fellows’ procedure log is reviewed to see if they are on track to meet the ACGME and program targets for various procedures. In addition, I check to see if they are completing competency requirements for procedures not required by ACGME such as EBUS, indwelling pleural catheter insertion etc, so that these may be listed on their final note to file and remind them if they have not done so.
- Scholarly activity: I review the evaluations from presentations at pulmonary grand rounds and journal clubs, research and case reports abstracts submitted to national conferences, and comments from their research mentors. This is the time to nudge them to submit their case reports for publications, and I review the avenues, online or print, where they can submit these. Often these “nudges” are entered in their 6-month goals as noted below.
- Quality Improvement: Our fellows are encouraged to join quality improvement initiatives in the MICU or bronchoscopy lab. A list of available activities is circulated at the beginning of each academic year. Their participation and ongoing project progress are reviewed.
- Duty hours/Wellness/Burnout: The semi-annual evaluations provide an opportunity to discuss with every fellow how they are coping with work-life-study balance, identify problems and review the institutional resources available to help them cope with challenges, as needed. Moonlighting activities are reviewed and forms are updated if the fellow has added any additional sites where they moonlight (GME requirement). Fellows are required to log their duty hours daily and any recent violations are reviewed.
- Career counseling: It is never too early to start planning for the career after fellowship. I discuss options for the career path the fellows would like to take from their very first semi-annual evaluation. While this may change during the course of their training, it allows them to seek opportunities early on to enhance their portfolios for the track they are interested in: clinician educator, physician scientist and/or additional subspecialty training. I discuss the resources available at our institution as well as those through CHEST, ATS and SCCM. For senior fellows, suggestions on job search, networking, reaching out to program alumni and/or shadowing opportunities are discussed and arranged.
- Mentoring program: We recently started a mentoring program in our fellowship. Fellows create an Individual Development Plan (IDP) with their mentor and this is reviewed.
- Goals for the next 6-months: Based on the data reviewed, I create 2-4 short term goals to be met by their next evaluation. Goals from their last evaluation are also reviewed to see if these have been met and barriers to meeting these are reviewed.
The semi-annual evaluation process noted above has gone through several iterations through time and I have incorporated ideas from other program directors and adapted from the resources available on the APCCMPD website (Program Director Toolkit).
Sunita Kumar, MD is a Professor of Medicine at Loyola University Chicago, Stritch School of Medicine in the Division of Pulmonary, Critical Care and Sleep, and the Pulmonary and Critical Care Fellowship Program Director. She was President of the APCCMPD in 2016-17. She is also the Medical Director of Loyola Center for Sleep Disorders.