How Guidelines Happen

Clinical practice guidelines serve as guideposts for providing evidence-based care for patients, assisting clinicians in decision-making. They may focus on providing the best value to patients, and in an era of health-care reform, they can drive funding decisions by policymakers or lead payer reimbursement decisions.

Practice guidelines seem to appear out of nowhere, as if handed down from above as the next Ten Commandments for treating patients. But just how does a clinical recommendation or idea become a guideline? And how do guideline writing panels sift through the mountains of sound and not-so-sound evidence to craft trustworthy, useful recommendations? As might be expected, it is an often time-consuming process that involves numerous stakeholders, a commitment to transparency, and a thorough investigation of existing research.

Panels of experts often spend several years developing clinical questions and diving into research pertaining to those questions, to ensure that recommendations are based on the most current clinical science.

If done right, the process yields a well thought-out, evidenced-based document that serves as a guide to practicing clinicians that highlights areas where additional research may be needed and drives future research and funding decisions where data are lacking.

ASH Clinical News spoke with experts involved in the development of guidelines for hematologic disorders and cancers about how evidence ultimately becomes a guideline.

Narrowing the Focus

As one might imagine, the first step in guideline development is deciding what to focus on. With new evidence emerging each day about hematologic conditions, deciding where to direct guideline development resources isn’t a simple process.

“There is a huge amount of evidence that gets published every day, every week, every month, every year,” Adam Cuker, MD, MS, assistant professor at the Hospital of the University of Pennsylvania in Philadelphia, told ASH Clinical News, explaining the rationale behind clinical practice guidelines. “It’s pretty much impossible for a busy clinician to keep up with all of the new information. One of the major roles of clinical practice guidelines is to synthesize and rigorously, systematically appraise this evidence and distill it down to recommendations for the practicing clinician.” Dr. Cuker serves as chair of an expert panel that is coordinating a new ASH effort to develop comprehensive new guidelines on venous thromboembolism (VTE) (SIDEBAR).

ASH has previously published guidelines on a few topics, most recently on immune thrombocytopenia (in 2011), and the organization has endorsed guidelines developed by others. The effort to develop ASH guidelines on VTE is distinguished by the use of a new, standardized process; greater staff support; and dedicated resources for a systematic, rigorous review of available evidence by independent researchers.

In 2014, ASH formed a new Committee on Quality, which is responsible for overseeing all ASH quality-related activities. Reporting to this committee, a new Guideline Oversight Subcommittee will manage the organization’s guideline development efforts. Julie Panepinto, MD, MSPH, professor at the Medical College of Wisconsin, chairs the new subcommittee. “Part of our task will be to prioritize disease areas for future guideline development. For the near future, the Committee on Quality has already selected some topics, including VTE and a revision of the guidelines on immune thrombocytopenia.”

According to Dr. Panepinto, a major effort of the past year by the Committee on Quality has been to define standard procedures for each step of guideline development to ensure the same level of rigor and transparency is applied to all topics.

According to Anita Rajasekhar, MD, of the University of Florida, who oversaw the creation of ASH’s guideline development process and serves on ASH’s Committee on Quality, the Guideline Oversight Subcommittee takes several factors into account before making any decisions.

First, they consider the needs of ASH members and which topics are in high demand, including suggestions from other internal committees, professional societies, policymakers, payers, patient groups, and government and non-government agencies.

“Next, we consider the disease itself and its characteristics,” Dr. Rajasekhar explained. “Is it a highly prevalent disease? Is it a very costly disease to treat and, if so, even though it may be rare, is it something that could potentially lower health-care costs if we have a good set of treatment guidelines available?”

Other factors that may drive the decision are whether there is large variation in clinical practice across the country, or if a clinical area is seeing rapidly changing evidence. “This is a common scenario in oncology, but it is also happening in hematology, particularly with new oral anticoagulants coming out,” Dr. Rajasekhar added.

So, why is ASH focusing on VTE for its first guideline development effort? “We chose venous thromboembolism because it is a very common disease that hematologists are involved with. There has also been rapid evolution of evidence in recent years,” Dr. Cuker said.

Finding the Right Team

Once the guideline topic has been decided, it’s time to assemble the guideline writing team. Panel selection is a careful process that involves multiple levels of organizational approval, beginning with the Guideline Oversight Subcommittee and ending with the ASH Executive Committee or its officers. An important goal is to create a multidisciplinary group that is able to look at the topic from all angles. “Despite our rigorous development process, panels must still use judgment to evaluate evidence and form recommendations. For this reason, we aim to balance expertise, perspective, and background,” Dr. Panepinto said. Face validity is also important, she acknowledges. “Readers may trust a guideline on the basis of a strong development process, but they also pay attention to who served on the panel.”

The process begins by identifying a panel chair, which Dr. Rajasekhar says is usually determined from a short list of candidates who possess a certain mixture of characteristics. “Obviously, they have to have content expertise, but they also have to have some expertise in developing guidelines, because this process is so involved, and we want someone who can bring both skill sets to the table,” she said.

Once the chair of a guideline panel is approved by the Executive Committee, the Guideline Oversight Subcommittee and the chair select the rest of the team. Dr. Panepinto notes that panelists are not just “friends of friends.” At every level of approval, individuals are likely to be included who are unfamiliar to the chair and to each other. The goal is to create a group that is balanced, diverse, and multidisciplinary.

“We want to include those in academics and practicing community hematologists, methodology experts and content experts, and even experts from other relevant fields [such as health economics],” Dr. Rajasekhar said. ASH has also made efforts to include junior faculty members, women, minorities, and international experts to ensure there are diverse backgrounds and ideas represented on the panel as well.

Research has supported the use of such multidisciplinary teams to create a balanced panel that can ultimately deliver clinically meaningful guidelines, according to Holger Schünemann, MD, PhD, MSc, chair of the Department of Clinical Epidemiology and Biostatistics at McMaster University in Ontario.

Creating Credibility

Another important factor in the selection process is minimizing potential conflicts of interest – financial or research-based – or any circumstance that would prevent a panelist from approaching the question or recommendation with an open mind. “There is a lot of evidence that indicates that people with potential conflicts make recommendations that are not necessarily in the best interests of those who are at the receiving end, so these conflicts of interests and considerations have started to play a very important role in eligibility for these panel positions,” Dr. Schünemann, who is also serving as the vice-chair of the VTE coordination panel, said.

Guided by an ASH stand-alone policy created to help manage conflict of interest in guideline development creation, those who select the panel vet each candidate to assess any potential conflicts.

“In [ASH’s] particular policy, we mandate that a majority of each guideline panel, including the chair and the vice chair, cannot have any current financial interest with companies that could be positively or negatively affected by the guidelines,” Dr. Rajasekhar said.

This policy is intended to be consistent with recommendations of the Institute of Medicine (IOM) as well as the Council of Medical Specialty Societies, a membership organization that represents most U.S. medical specialty societies including ASH, according to Dr. Panepinto. “The ways guideline developers think about conflict of interests has evolved greatly. Not long ago, conflicts were managed mainly through disclosure and transparency. In 2011, IOM raised the bar by recommending limits on the participation of individuals with conflicts.”

However, ASH guideline panels aren’t completely free of conflict of interest by design.

Members who have what’s perceived as a conflict of interest could also be those who have the most clinical or research experience in a given area – their expertise could be a valuable part of the panel. The key, Dr. Rajasekhar said, is preventing bias (not conflict of interest) from entering into the process. “It is crucial to have that scientific, clinical, methodologic, or other expertise and ensure that you are preventing bias from those conflicts,” she said.

Patients can also be a valuable part of the process, as well. Experts say having patients on guideline panels can provide the team with a unique and essential perspective.

“The doctors don’t receive the treatment; the patients do,” Dr. Schünemann remarked, “Patients are the individuals who can probably best consider whatever the interventions are or what the actual outcomes are that are being prevented or caused by treatment.”

“The IOM recommends that patient representatives should be included in the guideline development process, at a minimum during the question formulation step,” Dr. Panepinto explained. “We are piloting involving them on our VTE guidelines.” With the support of patient advocacy organizations, including the National Blood Clot Alliance, ASH staff identified more than 100 individuals who personally experienced VTE and were interested in volunteering their time to help on the guidelines. The individuals were vetted for enthusiasm and motivation, ability to understand technical information, experience serving on deliberative groups, and lack of conflicts of interest. The Executive Committee approved 20 of these individuals to serve as patient representatives on ASH VTE guideline panels.

Framing the Question

Once the topic is decided and the panel is in place, the next step is generating the clinical questions these guidelines hope to answer. Just what goes into a good question? When it comes to guideline development, a lot.

In Dr. Rajasekhar’s opinion, generating a good clinical question is “the most important part” of the guideline-development process. Three aspects must be met, she explained: a question needs to be relevant, answerable, and usable for the audience.

To create and frame these questions, ASH uses the PICO format: Patient, Intervention, Comparator, and Outcome. These four components need to be addressed to create questions that are specific and answerable. For example, one question may be: “For patients having major orthopedic surgery, how does warfarin compare with low-molecular-weight heparin in preventing venous thromboembolism?”

On the VTE committee, Dr. Cuker said, panel members tried to be as broad-minded as possible when developing the questions, drawing from their own experiences as clinicians caring for patients with VTE and looking at the available evidence.

Following the Evidence

After the clinical questions have been carefully defined, the guideline development panel is ready to enter the next, and often most lengthy, part of the process: the systematic evidence review.

Guided by the clinical questions, research and methodology experts systematically pore through all of the available literature, including what’s known as “grey literature,” or articles that are never fully published, to find the relevant evidence that could inform the answer to a particular question. Then, the group summarizes the literature and grades the evidence based on its strength or weakness. The results of this grading process influence the direction and strength of any recommendation ultimately made, Dr. Schünemann said.

“As part of these systematic literature reviews, one typically appraises the quality of the data that one obtains,” Dr. Schünemann explained. This judgment about quality is informed by multiple factors. For example, when evaluating evidence for a treatment studied in clinical trials, one would consider the study designs, the consistency of the treatment effect observed across multiple studies, the magnitude of the effect, the certainty in the estimate of the effect, and the possibility of publication bias (i.e., unpublished negative findings). The result of this judgment is a quality rating that reflects the confidence that one can have in the data that are being published and presented, he added. “This also needs to be a structured process so that others understand why certain quality ratings are made.”

For the VTE guidelines, ASH has externally commissioned this portion of the process to the GRADE Centre at McMaster University. Using the GRADE (Grading of Recommendations, Assessment, Development, and Evaluation) approach, the panel members will create guidelines and include information about the strength of each guideline based on the information available.

“If a study has many weaknesses – for example, is biased in some important way – that is going to be taken into account when the evidence is evaluated and recommendations are made,” Dr. Cuker said.

Even if the evidence is weak, however, Dr. Schünemann said there is still value in making a guideline recommendation. Not making a recommendation wouldn’t help anybody, he noted, because an individual clinician will rarely have the time – or, potentially, the qualifications – to look through all the available evidence, even if it is of low quality, to come up with a clear understanding of the issue and the best course of action to take based on what is known. “The requirement for decisions does not go away just because the evidence is not of the type of quality that we would like to see,” he said.

In guideline documents, treatment or testing recommendations are published along with the strength of the supporting evidence; this allows clinicians to make their own decisions about how the recommendation can be applied to patients in their clinical practices.

For the ASH VTE guidelines, the process of framing clinical questions and reviewing evidence is just beginning. After recommendations are formed, the guidelines will go through both an internal and external review process. Once these reviews are complete, the guidelines are published.

Dr. Cuker and his colleagues have high hopes for the VTE guidelines and, once the process is complete, hope it will offer real value in the field. “Ultimately, of course, I hope that these guidelines are both rigorous – based on the highest-quality evidence and the most rigorous methods – and user-friendly so that clinicians are able to apply them readily to their practice,” he said.

Reflections after Publication

What happens when the evidence is not of high quality – for instance, when there is a lack of randomized clinical trials in a specific disease setting, or for a certain patient population? The guideline writing panel for the National Heart, Lung, and Blood Institute’s (NHLBI) “Evidence-Based Management of Sickle Cell Disease: Expert Panel Report” ran into just that problem.1

In the guideline development process, which began in early 2009, George R. Buchanan, MD, co-chair of the Expert Panel, noticed one concerning trend: a lack of high-quality research in the area of sickle cell disease (SCD) and a low number of rigorous randomized controlled trials in this area. The panel, made up of about 12 individuals, looked at five major topic areas: ongoing health maintenance, managing acute complications, managing chronic complications, hydroxyurea therapy, and blood transfusions.

“Literally, there are about 15 or 20 randomized controlled trials in this area total, and some of those trials were not even that good,” said Dr. Buchanan, who is also professor of pediatrics and internal medicine at University of Texas Southwestern Medical Center in Dallas. “We found ourselves then having to rely on studies that weren’t as high-quality – observational studies or flawed studies with a selection bias. Often we felt compelled to make recommendations based upon less-than-optimal data. The end result was our inability to make strong recommendations for several of the recommendations that were made.”

The five-year process, however, did yield a document that helps set the standards for the treatment of SCD, Dr. Buchanan said, while also paving the way for future initiatives, additional research, and future funding possibilities. Ultimately, he said, “the report can help point researchers toward areas where more evidence is needed and to help set future research priorities.”

The process wasn’t without its flaws. In retrospect, Dr. Buchanan said, the report’s intended audience may have limited its long-term relevance. “This report was directed toward the primary-care physician. The assumption throughout the process – although some members might have had reservations about it – was that primary-care physicians would continue to be the main clinicians encountering this disease,” he said. However, that situation is not ideal. SCD is so complicated that it will require collaboration among primary-care physicians, hematologists, general internists, emergency room physicians, and hospitalists. Perhaps, Dr. Buchanan, reflected, the audience for the Expert Panel Report should have been broader.

Time for a New Approach?

Although specific methodology may change among organizations, the current standard for developing guidelines remains essentially the same: Select a topic, choose the guideline experts, narrow the question, and conduct a meticulous review of the available evidence.

However, after the IOM released the “Clinical Practice Guidelines We Can Trust” report in 2011, many guideline developers have been retooling their processes to meet the IOM standards.2

The report has an impact industry-wide, with the National Guideline Clearinghouse (NGC) implementing new inclusion criteria after its release. While the NGC had at one time more than 2,300 guidelines, there are now just 219 that meet the new inclusion criteria. Medical professional organizations across the country, including ASH, are striving to develop guidelines that meet the new NGC criteria. This push has caused some organizations to rethink their guideline development process.

Today, developing evidence-based guidelines has become a more rigorous process than ever before. Guideline developers must overcome significant obstacles as they try to create timely and reliable guidelines that inform both clinicians and payers.

For instance, cost is a significant factor due to the high price tag often associated with performing systematic evidence reviews. Managing conflicts of interest and involving patients in the process are other hurdles to overcome before guidelines can be created that meet NGC criteria.

ASH has invested heavily in its guideline development process to ensure it is meeting these challenges and creating guidelines of the highest quality to guide physicians in the years ahead.

Of course, after these potential problems are addressed, the guideline still faces its next – and arguably, most important – challenge: implementation.—By Jill Sederstrom 


  1. Yawn BP, Buchanan GR, Afenyi-Annan AN, et al. Management of sickle cell disease: summary of the 2014 evidence-based report by expert panel members. JAMA. 2014;312:1033-48.
  2. Institute of Medicine (US) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines; Graham R, Mancher M, Miller Wolman D, et al., editors. Clinical Practice Guidelines We Can Trust. Washington (DC): National Academies Press (US); 2011.


ASH’s First Guideline Development Initiative to Use the New Standardized Process: Venous Thromboembolism

Since developing a set of standard procedures for guideline development, ASH is embarking on its first guideline development initiative: venous thromboembolism (VTE).
The goal of the ASH VTE Guideline Coordination Panel, which began in 2014, is to produce a rigorous and transparent set of clinical guidelines that can help inform clinicians and treatment efforts. The panel, led by chair Adam Cuker, MD, MS, is in the midst of what is ultimately projected to be an 18-month process to develop a series of rigorous guidelines related to VTE.

The coordination panel has organized VTE into 10 logical topics, to focus research efforts as they move through the process. Ten expert panels have been formed to address each of these topics.

The first step in the process was to determine who would serve on the panels. ASH chose panel members after giving careful consideration to any potential conflicts of interest and ensuring balanced representation by bringing together a diverse and multidisciplinary group of experts.

“We wanted to be really thoughtful about having the panels reflect different perspectives,” Dr. Cuker said. “For example, there is a panel on prevention of venous thromboembolism after surgery, with a number of surgeons serving on that panel. We also involved a variety of other specialists, including an urologist, a neurosurgeon, an orthopedic surgeon, pulmonologists, cardiologists, general internal medicine specialists, hospitalists, and epidemiologists.”

Each of the 10 panels is now developing clinical questions they hope to answer through the guideline development process. Once that step is complete, ASH has contracted with McMaster University to conduct an eight-month systematic literature review that will find, summarize, and grade the strength of the evidence discovered.

Each panel will then use that evidence to develop guidelines.

“We know that guidelines that can’t be applied in practice are essentially useless,” Dr. Cuker said. “The ultimate goal of this guideline process is to improve practice and to improve outcomes for patients, so our absolute top priority is to ensure that these end up being really useable for the end user.”