In the 2018 interview season, our program received 466 applications for our 8 fellowship spots (6 in the match for PCCM, 2 outside of the match for our CCM track). This represents a 25% increase over applications from a year ago (25.2% increase, actually). This increase means that the job of sorting through potential candidates to select for interview and match is a Herculean task. Here, I will outline what I think are important considerations for Program Directors and fellowship selection committees to consider in their philosophy, process of evaluating ERAS applications, and interview methodology.


Philosophy toward the application review process


Programs and program directors should be explicit when they outline their fellowship mission and vision prior to the application review process. It is reasonable that there are a variety of goals regarding the types of trainees each program desires to train. Some will want to create NIH-funded researchers; some may want to create clinical leaders; others may focus more on clinician experts in a certain field; some will want to train a combination of these. All of these are reasonable goals toward which a fellowship can strive; but the first step to a successful recruiting season is to agree upon what the goals are and articulate them clearly. I suggest putting them on paper and agreeing on language with those who will participate in the interview process­–this can be a helpful exercise in program consensus.


Process in evaluating ERAS applications


Once the program philosophy is clearly outlined, a process for evaluating the fitness of applicants should be established. The Electronic Residency Application Service (ERAS) provides a standard number of variables for each candidate (listed in the table to the right). Some of these variables are objectively valued (asterisked in the table, ie, AOA status, USMLE scores, if they were a chief resident), while others are subjective and their value is variable, depending on the assessor (ie, quality of residency, quality of publications, Letter of recommendation value).


At Indiana University, we used these variables to develop a way to formalize weighting each of these parts of the ERAS application and combined that with a similarly weighted interview day score and compared it to our usual method of evaluating candidates; we then did this at five other centers. This method uses a “Wisdom of Crowds” method to democratize the value that our fellowship selection committee places on the different variables that go into selecting candidates. This score is one that we can use to compare across years to evaluate the quality of our candidate pool and dig into those variables that select for the highest quality candidates.


Regardless, I suggest that programs have a standard process and a scoring system that is as reproducible as possible. Of course, I am partial to the process that we use, but there are a myriad of ways to do this well, so long as you are consistent and the scoring process is a manifestation of your program’s mission and vision. (If you don’t have a process and you want to develop one, here is a how-to-guide to get started on developing your program’s own similar evaluation tool.)


Interview methodology

There are a ton of data in Human Resources literature about how best to utilize the interview process to select candidates. Much of these data suggest that most postgraduate medical education programs are doing it wrong. I have found this article to be very helpful in developing my interview strategy. Based upon this article, I do one thing that many think is uncomfortably unconventional when interviewing, and one thing that is becoming more the norm that I think people should consider adopting.


First, I use ERAS to select the applicants to interview in July, but I do NOT re-review their application before their interview day. This satisfies Principle #2 from the paper above: “Know as little about the candidate as possible.” This allows me to focus on those things that I think should be variables judged in the interview: communication skills, judgement, ability to share vulnerabilities, and insight. I find that if I know the intimate details contained in their ERAS application beforehand, I spend much of the time confirming things I can get from the application, rather than trying to know who they are as a person. I like to think that this strategy makes me a better judge of professionalism and interpersonal skills (I admittedly have no data to back this up). I have been interviewing this way for the past three years, and I will never go back to the old way. I find out things I never would have before, and have insights unobtainable by “traditional” interviewing.


Second, I use behavioral and situational interviewing questions, rather than the traditional “where do you see yourself in 10 years?” interview questions. There is a reasonable amount of literature for doing this–quite simply, it leads to better interview outcomes. This has been known since the early 1980s, when randomized trials demonstrated that candidates hired by behavioral interviewers performed far better in student ratings of teacher performance than did traditional interviews. In the study from 1982 outlined in the graphic to the right, teaching assistants were randomized to being interviewed using “traditional interview” questions or using behavioral interview questions and these interview scores were compared to their teaching performance after hiring–the correlation coefficient was far higher for the behavioral interview-style questions (0.54) than for the traditional interviews (0.07). There are a myriad of places online to find good behavioral interview questions. I use the same seven questions for all candidates. These have been perfected over time and have served me well. If you are interested, Geneva Tatem, MD and her team at Henry Ford Hospital have published on their experience with behavior-based interviewing.

Editorial narration: The peer reviewer for this blog post asked if I would share my behavioral interview questions in this blog. While I agree it would help to improve the quality of this post, it would also give away my interview questions for future years. So, I will not!


How do I operationalize this? I warn applicants that I have not re-reviewed their application (after the initial application review to invite for interviews) and the reasons for this. I ask my behavioral interview questions. I score my interview. Then, and only then, do I open the ERAS application to review it. This way my interview score (which is seemingly more objective) is not tainted by the ERAS application.


So, as we approach interview season, I would encourage you to reflect on your process and consider doing the following:

  1. Clearly outline your program philosophy and goals.
  2. Convene your recruitment committee (or equivalent) to assess and weight the ERAS variables you will use to select candidates to interview.
  3. Develop/refine a scoring system that allows you to compare candidates over time.
  4. Challenge interviewers to know as little about interview candidates as possible when interviewing them (consider not giving them access to ERAS applications until interview scoring is done)!
  5. Use behavioral and situational interviewing questions.


Gabriel T. Bosslet, MD, MA is Associate Professor of Clinical Medicine in the Department of Pulmonary, Critical Care, Sleep and Occupational Medicine at Indiana University and an affiliate faculty member at the Charles Warren Fairbanks Center for Medical Ethics. In addition, he is an Assistant Dean in the Office of Faculty Affairs and Professional Development, and is the Fellowship Director for Pulmonary and Critical Care Medicine. He is also the co-director of IU Talk, a communication skills workshop for practicing physicians at Indiana University.