The American Society of Hematology’s (ASH’s) Hematology In-Service Exam (ISE) is a low-stakes exam that has been offered to hematology fellows since 2009. The 200-question medical knowledge assessment is typically administered by fellowship programs each March and allows trainees to gauge their readiness to take the hematology board exam, while also offering feedback to training programs on the effectiveness of their curricula.
“The ISE is designed to assess knowledge – and knowledge gaps – for individual trainees and their programs, which allows them both to better fill in those knowledge gaps,” said Alice Ma, MD, Co-chair of the ASH Test Materials Development Committee and an Associate Editor of ASH Clinical News. Dr. Ma also is a Professor of Medicine and Director of the Hematology-Oncology Fellowship Program at the University of North Carolina, Chapel Hill.
The exam’s content includes both malignant and nonmalignant hematology and mirrors the American Board of Internal Medicine’s Hematology Certification exam blueprint.
In 2020, 163 training programs – both in the U.S. and internationally – registered for the ISE. In total, 1,411 trainees have taken the exam this year, though in a slightly different format adapted to the unique circumstances of the COVID-19 pandemic.
ISE Goes Virtual
In 2020, the In-Service Exam was originally scheduled to be administered in March – just a few weeks after the COVID-19 pandemic hit the U.S. ASH staff, in consultation with the National Board of Medical Examiners (NBME), which helps produce the exam, moved quickly to postpone the exam and set up a secure virtual option for trainees.
“The main driver in terms of postponing it was to communicate to trainees that we care about them,” said Matthew Ulrickson, MD, Co-chair of the ASH Test Materials Development Committee and Section Chief of Hematology at Banner MD Anderson Cancer Center in Phoenix, Arizona. “It was a very stressful time, especially early on when the pandemic was first starting to affect the U.S., so adding an exam on top of it just wasn’t at the heart of what we wanted to do.”
The 6-hour, multiple-choice ISE traditionally is administered at a university testing center with an in-person proctor. Due to COVID-19, the NBME implemented remote proctoring guidance that allowed trainees to take the exam in a computer lab, from home, or from offices spread out across their campuses. Trainees were required to enable their cameras through Zoom or WebEx so they could be monitored during the exam.
The exam took place on June 9 and 10 and was remote proctored by each institution. ASH was able to return scores to trainees and programs in July.
In 2021, the exam will be back on its traditional March schedule, with scores expected in May, but programs can select remote proctoring or in-person testing at the time of registration.
Dr. Ulrickson said the Test Materials Development Committee is open to the idea of continuing remote proctoring beyond 2021, provided they can guarantee the security of the exam.
Even before COVID-19, ASH was adjusting the exam to make it more useful for trainees and fellowship programs. Over the last 2 years, ASH began providing a “teaching point” for missed questions.
Previously, when a trainee missed a question on the exam, their results would identify only the subject area or diagnosis missed, which Dr. Ma noted wasn’t very helpful to guide studying.
Now, the feedback is more granular, with trainees and programs receiving specific details on areas needing improvement. Instead of learning that the missed question related to treatment of hemophilia A, for example, trainees will now receive a “teaching point” indicating that they need to review the impact of emicizumab on patients’ partial thromboplastin time.
“The In-Service Exam is designed to assess knowledge – and knowledge gaps – for individual trainees and their programs.”
—Alice Ma, MD
Designing the Exam
Creating the exam is a year-long process that begins even before the previous exam is administered. The ASH Test Materials Development Committee, comprising about a dozen physicians with expertise in malignant and nonmalignant hematology, starts writing exam questions in February. Those initial questions are put into a standard format by the NBME, then reviewed by the full committee in July. The group usually meets in person to review and revise the questions based on feedback. In 2020, that process happened virtually.
The NBME constructs the exam using a combination of new and previously used questions, with the goal of creating an exam that is robust and sound, Dr. Ma said. “Sometimes when we write the questions, we don’t realize how they might be misinterpreted,” she explained. Through these discussions, “we learn whether a question might have two right answers, especially when we’re talking about how we treat certain hematologic malignancies or certain bleeding disorders.”
Producing the individual questions and evaluating the exam is a continual process.
When selecting previously used questions, NBME looks at metrics from past exams. The ideal question will be answered correctly by 30 to 90% of trainees, Dr. Ma said. If the percentage of trainees who answer the question is too high, the question is considered too easy; too low and the question is too difficult.
The draft exam is reviewed by the Test Materials Development Committee co-chairs and NBME before it is ready. But, even after the exam is administered, it is reviewed once more. The NBME identifies questions that performed poorly, the, the co-chairs review them and vote to either accept the question but flag it for rewriting for the next year’s exam, or reject it as invalid. If a question is rejected, it is not scored as part of the exam.
Dr. Ulrickson said he learns a great deal as an educator from writing the exam questions. In some cases, that process alerts the committee to potential knowledge gaps. For example, if questions in a particular topic area underperform for 2 to 3 years, it could indicate a problem with training curricula.
“For underperforming questions, we have started to reach out to program directors to share what we found,” Dr. Ulrickson said. “It might be worthwhile for them to use an ASH resource or other resource to provide additional education to trainees.”
Dr. Ulrickson added that the Test Materials Development Committee is always looking for new members. Interested physicians can attend a question writing workshop hosted by ASH to learn more about the process and how to craft exam questions.