From gaps to guidelines: a process for providing guidance to bridge evidence gaps

Abstract

Background

Despite the proliferation of clinical research that can be used to inform Clinical Practice Guidelines there remain many areas where the number and quality of research studies vary widely. Using the Canadian Clinical Practice Guideline for Moderate-to-Severe Traumatic Brain Injury (MOD-SEV TBI) as an example, there is a lack of robust research evidence, derived from randomized controlled trials, meta-analyses, and systematic reviews to inform the recommendations. Randomized controlled trials in this field often have limitations, such as smaller sample sizes and gender and racial disparities in enrollment, that reduce the level of evidence they can provide. Notably, evidence is often lacking in the priority areas identified by people with lived experience (PWLE) and guideline end-users.

Methods

The Canadian Clinical Practice Guideline for MOD-SEV TBI rehabilitation is a Living Guideline that implemented a robust and replicable process to mitigate these issues. This process includes: 1. Identification of Priorities by PWLE of MOD-SEV TBI and Guideline End-Users; 2. Involvement of Diverse Multidisciplinary Expert Panels, Including PWLE; 3. Compilation, Review and Evaluation of Published MOD-SEV TBI Evidence; 4. Identification of Gaps in the Published Literature; 5. Formulation of Recommendations, Rigorous Grading of Available Evidence and Formal Voting; 6. Creation of Knowledge Translation and Mobilization Tools and 7. Publication of the Updated Living Guideline.

Results

Since 2014–15, the Canadian TBI Living Guideline has implemented and refined this process to produce high-quality expert consensus-based recommendations and knowledge translation and mobilization tools across 21 comprehensive domains of TBI rehabilitation. There are 351 recommendations in the current version of the Canadian TBI Living Guideline; 68% of these are primarily consensus-based recommendations. Developing a comprehensive guideline in areas where research may not be present or strong ensures that the Guideline is comprehensive and addresses the priority needs of clinicians and PWLE.

Conclusions

The use of robust, transparent, and replicable evidence reviews and expert consensus building process produces clinical guidelines that are relevant and applicable even when empirical data are lacking or absent. This process of developing consensus-based recommendations can be used to develop guidelines in other content areas and populations facing similar challenges.

Background

There is a common call for timely, evidence-based, and appropriate practical guidance across multiple healthcare and scientific fields. Yet, oftentimes, clinical guideline development groups are reluctant to make recommendations in areas that are supported by weak evidence out of concern that they lack foundation or feel that they should not because having no recommendations in an area will highlight directions for further research [6]. The purpose of this paper is to share the experience, challenges, and successful strategies of developing a comprehensive clinical practice guideline (CPG) for Moderate-to-Severe Traumatic Brain Injury (MOD-SEV TBI) rehabilitation as an example of research- and consensus-based guideline development. The Canadian Living Clinical Practice Guideline for MOD-SEV TBI (The Canadian TBI Guideline)—https://kite-uhn.com/brain-injury/en/guidelines formerly known as INESSS-ONF Clinical Practice Guideline for the Rehabilitation of moderate to severe TBI is a living CPG that was established in 2014–15 in Canada and since then has gone through multiple update cycles using a rigorous review process involving multidisciplinary expert panels. Although the content used in this example is TBI focused, the general principles of guideline development are applicable to other areas.

The Institute of Medicine’s guidebook on developing CPGs highlights the need for systematic guidance to support clinicians in making appropriate medical/clinical decisions and points out that better healthcare quality and outcomes can be achieved using a rigorous process that combines research evidence, clinical experience and values and priorities of PWLE [10]. Although many CPG development groups are reluctant to make recommendations when the relevant evidence is of low-quality or absent, research shows demand for and utility of such recommendations. Clinicians prefer having recommendations and evidence summaries presented together and this can facilitate good clinical decision making [14]. Neumann and Schünemann [15] discuss the value of health guidelines in supporting clinicians and PWLE in areas where evidence is sparse or inconclusive and urge expert panels to make recommendations in this context, emphasizing that guideline panels should exercise transparency and use explicit frameworks to describe the considerations and challenges encountered in the process.

In the recent years, there has been a trend to apply higher standards for grading evidence, which has resulted in downgrading or removal of earlier recommendations in new updates of existing guidelines [7]. There is an 81% chance that a guideline recommendation supported by at least one RCT will be preserved after an update is conducted [21]. Paradoxically, the application of stricter grading criteria has resulted in a higher perceived methodological rigor of guideline development, but a simultaneous loss of clinical appeal along with reduced perceived relevance for decision making. This paradox illustrates the gap between rigorous evidence and clinical practice and highlights the importance of expert consensus in helping to bridge the gap when high-grade evidence is unavailable or insufficient [12, 23].

RCTs are recognized as the gold standard in evidence-based care; however, conducting high-quality RCTs in many fields, including MOD-SEV TBI rehabilitation is challenging and not always feasible. Challenges include difficulties in blinding participants and study staff, finding appropriate placebo or attention–control sham interventions for control groups, and ethical concerns about withholding treatment from control group participants. In addition, RCTs take several years to conduct and publish, and the complex and multifaceted nature of TBI requires controlling for multiple variables [8]. The need for larger sample sizes and complex study designs involving multiple care providers often makes these studies cost-prohibitive [20].

Evidence derived from RCTs may not be easily translated into practice or informing overarching principles of care, as these studies often focus on intervention efficacy in highly controlled environments that may not accurately reflect their implementability and effectiveness in a real-world setting where multiple uncontrolled variables are present [5, 22]. Consequently, it is necessary to include expert opinion grounded in clinical and lived experience to understand contextual and environmental factors and consider patient experiences and barriers to implementation, such as systemic inequities and availability of services and resources.

A review of RCTs in moderate to severe TBI by Teasell et al. [20] examined 662 published studies, finding that only 34.7% focused on rehabilitation, with these trials typically having smaller sample sizes (mean n = 56.4) compared to medical/surgical trials (mean n = 195.8). Moreover, the review found that most rehabilitation RCTs were conducted in the chronic phase post-injury (≥ 6 months), with few studies occurring in the acute phase. Rehabilitation trials represented only 8.7% of RCTs conducted in the acute phase post-injury [20]. A review of 135 clinical trials in TBI research published from 2008 to 2022 found marked gender and racial disparities in enrollment that did not improve over 14 years [2].

High-quality consensus-based recommendations are particularly needed in priority areas lacking published evidence. A rigorous guideline development process is necessary to address the needs and include perspectives from guideline end-users and PWLE, mitigate individual biases and ensure the relevance and implementability of guideline recommendations.

These themes are prominently present in the development of the Canadian TBI Guideline as the Guideline development Expert Panels often need to make recommendations in priority areas where research evidence is of low quality or absent. MOD-SEV TBI is a complex and lifelong condition that requires a multidisciplinary approach across the continuum of care to address the diverse needs of people recovering from TBI [8]. A comprehensive guideline is necessary to guide practice and support end-users, including clinicians and clinician managers and secondarily to inform researchers, PWLE, caregivers, and system planners. Despite advances in MOD-SEV TBI management, guideline end-users, including physicians and other regulated healthcare professionals, PWLE and caregivers, and rehabilitation program managers express the need for specific guidance on various aspects of clinical care, including the intensity, frequency, and duration of rehabilitation sessions and benchmarks for the appropriate length of stay, innovative treatment modalities, and continuity of care [4, 11, 19]. In addition, PWLE and family members/caregivers often present with questions about treatments they have heard about or areas of need that they want help with that may not be routine parts of inpatient rehabilitation programs (e.g., sexuality and intimacy).

The quality and quantity of published evidence supporting various aspects of MOD-SEV TBI rehabilitation vary considerably. Although certain areas are supported by high-quality randomized controlled trials (RCTs) and systematic reviews [5], many others rely heavily on expert consensus and observational studies [3].

The objective of this paper is to describe the process for integrating consensus-based recommendations, reflecting the needs of healthcare providers and the priorities of PWLE, into a comprehensive living CPG, thus bridging gaps in the availability and strength of available evidence.

Results

Since 2014–15, the Canadian TBI Guideline team has implemented the rigorous and replicable multistep living review process designed to evaluate all types of evidence (scientific, clinical and experiential) to produce a robust and comprehensive guideline that bridges any evidence gaps where published literature is absent or insufficient to produce high-quality consensus-based recommendations. This process was used to produce the first edition of the Guideline in 2014–15 and subsequent updates to the Living Guideline format that was implemented to address the rapidly growing body of research on MOD-SEV TBI rehabilitation.

The multistep process is outlined in Fig. 1, and the rationale for each step is presented in Table 1.

Fig. 1

Living guideline review cycle

Table 1 Rationale for the process components

By following the process, 351 recommendations have been developed across 21 chapters, each focusing on different domains of MOD-SEV TBI rehabilitation. Expert Panel members have created high-quality recommendations in all areas, but particularly in areas identified by the TBI community (clinicians and PWLE) and/or where there is weaker or absent evidence in priority areas.

Identification of priorities by PWLE of MOD-SEV TBI and Guideline end-users

The priorities for the Guideline are identified by consulting with PWLE of MOD-SEV TBI and Guideline end-users. PWLE are invited to be part of this process through broad invitations sent out through brain injury associations, by word of mouth, by targeted invitation to ensure broad regional representation (e.g., urban, rural, different regions, different injury severities, etc.) and by approaching PWLE engaged in post-secondary graduate work. PWLE complete online surveys and participate in Focus Groups to identify their priorities, but there is flexibility used at each time point to allow PWLE to participate using their preferred method of online survey or focus group. A process of feedback checking using individual and group summaries is used to allow PWLE to confirm and/or make any needed edits to the feedback about the Guideline content, format and/or process. This process was used for the initial development of the Canadian TBI Guideline in 2014–15 (then known as INESSS-ONF Clinical Practice Guideline for the Rehabilitation of moderate to severe TBI [4]) and has been repeated multiple times since, with the most recent consultation conducted in 2023.

End-users are identified through professional associations and networks, healthcare provider rehabilitation facilities, and brain injury collaboratives/networks. Input and feedback from end-users are collected via surveys, ongoing focus groups, interviews, conferences, knowledge mobilization activities, and through the Canadian TBI Guideline website engagement and user feedback form. In addition, two virtual summits have been held (2020, 2022) to review the entire Guideline with key partners (clinicians, researchers and PWLE) to ensure that the Guideline was meeting the expectations and needs. Through each of these consultations, priorities were established for any edits.

In addition to the initial priority setting survey [19], there have been ongoing surveys and focus groups conducted as part of the Neurotrauma Care Pathways Initiative https://www.neurotraumapathways.ca/, where PWLE emphasized the need for guidelines addressing pain and headaches, intimacy and sexuality, patient and family education, and the importance of continuing rehabilitation after discharge into the community. PWLE also highlighted the importance of maintaining hope for improvement and acknowledging the active role and agency of PWLE in their own recovery and incorporating considerations for equity-deserving populations.

Responding to these needs, new recommendations have been developed in the review cycles from 2021 to 2023, addressing priority areas identified by PWLE, where the evidence is emerging or newly developed. A new chapter addressing issues related to intimacy and sexuality has been added, along with the start of a process to incorporate considerations for equity-deserving groups. Ongoing feedback from PWLE has indicated support for the recent modifications and that they feel included and valued in the Living Guideline process.

Involvement of diverse multidisciplinary expert panels, including PWLE

A multidisciplinary Expert Panel is formed for each chapter of the Guideline. Some of the chapter Expert Panels work together due to a considerable overlap of membership. The Canadian TBI Guideline team invites volunteers and secures involvement from the leading experts in the field (e.g., researchers, clinicians and regulated health professionals) and PWLE of MOD-SEV TBI. The Expert Panels are standing committees that are active during the review of the relevant chapter but also can be called upon between review cycles if new or updated guidance is needed. Expert Panel members are provided with an orientation to their role and are supported as needed by the Guideline Team to fulfil their roles.

During each review cycle, Expert Panel members attend a series of virtual meetings to review the evidence that has become available since the previous update. Ground rules for the Expert Panel meetings are established, endorsed by the group and followed to promote an equitable and productive working environment. An experienced group facilitator leads the process, and a research coordinator provides support and assistance, retrieving necessary materials and liaising with the Guideline Team and relevant collaborators. Facilitators are identified through self-selection and confirmed by group consensus based on their experience in the field and interest in leading the group discussion. PWLE participate in a co-lead position on various topics.

The Expert Panels utilize shared working documents for synchronous and asynchronous review and editing, a virtual voting platform and templates for structured evidence summaries and tools to support knowledge mobilization and implementation. During virtual meetings, the Expert Panel reviews the latest evidence as well as existing tools and resources, discussing clinical implications, adjusting, formulating new or updated recommendations, additions, and revisions to the supporting materials as necessary.

To ensure balanced representation in the Expert Panels, a table is prepared that outlines some of the key characteristics of each member of the panel (e.g., type of clinician or researcher, PWLE, area of brain injury expertise, geographic location/province/country, rural/urban location). At each review iteration membership of the Expert Panels is reviewed to ensure that current leaders in the chapter area are engaged in the process and the group has representation from different clinical and experience backgrounds. New members are added as needed. Over the years, the participation of PWLE has increased due to targeted efforts of the Guideline Team.

Currently, there are 9 Expert Panels, involving 98 people, active in the Living Guideline process, including researchers, healthcare professionals and PWLE of diverse clinical and experience backgrounds. Each Expert Panel comprises 4–14 individuals who bring their expertise to further develop and update the Guideline and improve the development process. The disciplines represented in the Expert Panels include PWLE, physicians (including physiatrists, neurologists, and family physicians), nurses, physical and occupational therapists, speech–language pathologists, dietitians, pharmacists, social workers, psychologists, neuropsychologists, clinical administrators, unit managers, researchers and educators and others. Disciplines are matched to the content of the section and the new evidence to be reviewed. Additional experts from other fields are identified and invited if necessary. Not every Expert Panel includes PWLE and there are continued efforts to ensure PWLE representation on each Expert Panel. Where an Expert Panel does not have a regular PWLE member, the content of the Chapters and proposed revisions have been circulated to PWLE through brain injury association networks via online surveys for comment and review.

Compilation, review, and evaluation of published MOD-SEV TBI evidence

A comprehensive review of evidence published since the previous review is conducted by the Evidence-Based Review of Moderate-to-Severe Acquired Brain Injury (ERABI) (https://erabi.ca). This involves a thorough search of multiple databases, including MEDLINE, EMBASE, CINAHL, Cochrane, and PsychINFO. The aim is to identify all relevant intervention studies and evidence, including RCTs and other forms of research where the majority of the study participants are PWLE of MOD-SEV TBI. The Expert Panel members can also suggest literature to review that meets the inclusion criteria that might not have been identified in the ERABI Process (e.g., papers published after the cutoff date used by ERABI).

Transition to a living guideline model allows the Canadian TBI Guideline Team to incorporate the most current research evidence in the Guideline by shortening the review cycle to no more than two years before the next review. The living review process also allows for seminal research or important new findings to be integrated into the relevant Guideline chapters as needed. Between group meetings, Expert Panel members engage in offline work, which includes reviewing additional materials, engaging their professional networks for feedback, and refining the wording of recommendations to ensure they are clear, accurate, and easily implementable in a clinical setting. Expert Panel meetings can be restarted at any time in response to changes in evidence (e.g., if a high-quality RCT is published) and priorities before the regularly scheduled review.

Collaboration with ERABI has allowed hundreds of published papers to be reviewed to ensure that no more than two years have passed since previous reviews. This process has been able to include all relevant new literature.

Identification of gaps in published literature

The Expert Panels conduct a review to assess the quality and relevance of the current literature, clinical practices, and lived experiences related to MOD-SEV TBI and the potential impact on the recommendations in the designated chapter for review [17]. The Expert Panels participate in a synchronous discussion using in-person or videoconference meetings to identify key areas for development or updating of recommendations. This involves evaluating the quality and relevance of the evidence, discussing it in the context of real-world environment, considering the experiences and insights of PWLE of MOD-SEV TBI and identifying any key areas that are not represented by the research evidence.

The ERABI reviews of peer-reviewed literature are comprehensive, but the Expert Panels have added searches of the grey literature, nonintervention studies and qualitative studies to ensure that all concepts and findings relevant to providing clinical guidance for the rehabilitation of people after TBI are considered in developing recommendations. These additional searches combined with clinical and lived experience have been the foundation for the majority of the consensus created recommendations.

Any relevant evidence brought forth by the Expert Panel members in addition to the summary of published evidence provided by ERABI (e.g., recent or qualitative studies or nonintervention studies) is checked against the literature inclusion criteria to make sure there was not a systematic exclusion of any relevant literature in the peer reviewed literature review process. The Expert Panel receives evidence and summaries in advance of meetings to facilitate guided structured discussion using a modified Delphi approach to achieve consensus on inclusion of evidence in the formulation of recommendations.

In priority areas where published evidence is absent or insufficient, experts may also incorporate recommendations based on the relevant published evidence in related populations, such as mild TBI or stroke, and clinical opinion and lived experience of people with brain injury.

Since 2014–15, this process has been repeated several times and now involves review of the new literature and evidence in the context of existing recommendations and prior evidence.

At this stage, feedback received since the last review of the Guideline chapter will be integrated into the discussions. Often this feedback will identify areas where clinicians are requesting additional guidance or areas where PWLE have identified that the healthcare professionals are not effectively addressing during rehabilitation and recovery. In areas where research evidence is emerging it may be that there is now enough published evidence in an area where previously there was not enough evidence to even create consensus recommendations. Where the research evidence is not strong the consensus building process will be used to identify any appropriate recommendations. This step has been essential to ensure that the Guideline provides the guidance that is needed to optimize recovery, in particular the recovery that matters to PWLE.

Formulation of recommendations, rigorous grading of available evidence and formal voting

The available evidence is rigorously graded using established methodologies including the Physiotherapy Evidence Database (PEDro) scale [13] and a standardized system for evidence grading used in numerous CPGs for TBI, spinal cord injury and stroke [17]. The Expert Panels evaluate the quality of supporting studies and other evidence, assess the strength of resulting recommendations (integrating research, clinical and lived experience evidence) and consider their potential benefits and risks. The level of evidence supporting each recommendation and indicating the current state of the field is defined as follows:

• Level A: Supported by meta-analyses, systematic reviews, or RCTs.

• Level B: Supported by cohort studies, well-designed single subject experimental designs, or small sample size RCTs.

• Level C: Supported primarily by expert opinion (clinical and lived experience), qualitative research, and uncontrolled case series.

All the evidence goes through the same rigorous review, applying a modified Delphi approach and formal consensus-building and voting process to ensure endorsement by the Expert Panel for the Level of Evidence. At least one high-quality RCT (PEDro score of 6 or higher), meta-analysis, or systematic review is required to assign Level A evidence.

Once the evidence (research, clinical, and lived experience) has been reviewed and graded, the Expert Panels review and discuss the content in the context of updating existing recommendations and/or drafting new recommendations and updating previously determined levels of evidence for the recommendations. Where the research literature is strong and of good quality, recommendations are developed consistent with the research evidence; in areas where the research literature is absent, of poor quality or insufficient, consensus-based recommendations were developed. The strength of the recommendation is then determined by the Expert Panel taking into account the expected impact of the recommendation on the quality of healthcare and patient, the level of supporting evidence, how it relates to the needs and priorities identified by the PWLE and Guideline end-users. The strength of each recommendation is indicated by wording chosen by the Expert Panel, e.g. “should be used”, should be considered”, “may be used”, “may be considered”. In addition, key recommendations are assigned Priority or Fundamental status based on the following definitions:

• Priority: Clinical practices or processes deemed most important to implement and monitor during the course of rehabilitation for people having sustained a TBI. These practices are most likely to result in positive outcomes for people with TBI. Note: The guideline development team strongly believe that implementation of the priority recommendations would be difficult without the fundamental recommendations in place first.

• Fundamental: Defined as the elements that settings/programs (where rehabilitation is provided) need to have in place, in order to build the rest of the system properly. These are primarily for program managers and their leaders as they reflect the service conditions for optimal rehabilitation provision.

Once consensus has been reached within the Expert Panel the final set of recommendations for a chapter is created. The changes are submitted for formal voting using an electronic voting platform, allowing respondents to accept, accept with modification or reject any recommendation. A minimum of 75% agreement with the recommendation by at least 80% of the expert panel members is required for a proposed revision to be approved. This high threshold ensures that the recommendations reflect strong consensus among the experts. Further rounds of voting for selected recommendations may occur after modifications have been incorporated. The draft recommendations are sent to 3–4 experts who were not part of the development of the recommendations but who have the expertise/experience to comment on the relevance and appropriateness of the recommendations. After any final edits have been made and the voting and review process is concluded the recommendations are sent for French translation, as the Guideline is produced in both official languages of Canada.

The process for developing recommendations and voting has been repeated many times since 2014–15 and the Expert Panel members report a high comfort level and feel that the process is both comprehensive and efficient. In 2023, the Guideline Team conducted an analytical overview of the Guideline content to characterize the level of evidence supporting different domains of MOD-SEV TBI rehabilitation. Recommendations where the level of evidence is high are grouped within a small number of chapters where the volume of research is high (see Fig. 2). These chapters are I—Disorders of Consciousness, J—Cognitive Functions, K—Cognitive Communication, L—Dysphagia and Nutrition, M—Motor Function and Control, and R—Neurobehavior and Mental Health. Notably, of the 351 recommendations at the time of review, 239 were Level C, representing 68% of the guideline. Three domains consist entirely of Level C recommendations, many of which are consensus-created. These chapters are F—Brain Injury Education and Awareness (4 recommendations), G—Capacity and Consent (3 recommendations), and H—Comprehensive Assessment of the Person with TBI (10 recommendations).

Fig. 2

Levels of evidence

Consensus-level recommendations are particularly prevalent in chapters focusing on priority areas identified by PWLE where guidance was requested, including Brain Injury Education (4 out of 4 recommendations; 100%), Intimacy and Sexuality (9 out of 10 recommendations; 90%), and Pain and Headaches (19 out of 21 recommendations; 90.5%).

Creation of tools to support knowledge mobilization and implementation

There has been a focus on developing knowledge translation/mobilization tools and identifying resources to support the integration of the consensus recommendations within the priority areas for PWLE as these areas might not be as commonly understood and familiar to healthcare providers. Clinicians have requested these take the form of summaries of key practice points and decision algorithms.

The development/selection of tools to support knowledge mobilization and implementation begins with identifying the needs of end-users through input gathered from the website comments, focus groups, summits, presentations and webinars, Expert Panel meetings, and a review of current literature and resources. This is followed by an assessment by Expert Panel members to identify key clinical messages that should be highlighted in these tools and whether or not existing tools and resources exist. If validated tools already exist, they will be added to the chapter in the tools and resources section. Where there is a gap in tools, the Expert Panel will develop tools that fill the gap. The first step involves identifying the specific target audience who would be using each tool, which may include clinicians, administrators, and/or PWLE. The purpose of the tool is then determined whether it is to inform, change practice, or provide support for implementing best practices in clinical care. Finally, an appropriate format for the tool is selected, which could include informational brochures, posters or infographics, checklists, decision rules, algorithms, or executive summaries to ensure it effectively meets the needs and preferences of the intended audience.

Following the initial development stage, tools are reviewed and revised by the relevant Expert Panels to ensure they are accurate and suitable for their intended audience and purpose. A small group of end-users are then invited review the tools. The tools are revised and finalized based on this feedback and submitted to the Expert Panel for voting to secure approval.

Publication of the updated Living Guideline

The Guideline has been transitioned to a living guideline where evidence reviews and consultations are made regularly to ensure up-to-date and relevant guideline recommendations that are responding to current best practice evidence and needs. Chapters are reviewed on a schedule where preferably no more than two years elapse between chapter reviews.

The first major update was conducted in 2020 when the Guideline was transitioned to a Living Guideline exclusively available as an interactive website (https://kite-uhn.com/brain-injury/en/guidelines). For every update, the final set of recommendations with companion resources are posted. It is possible to toggle easily between both the French and English versions of the Guideline. Manuscripts have been published on Lessons Learned [17], Intimacy and Sexuality [18], Cognition and Cognitive Communication and Guideline processes, and outcomes have been presented at international, national and local scientific and clinical conferences and invited talks, gathering interest from the TBI and CPG development communities.

Finalized recommendations and related knowledge translation and mobilization tools approved through the review and voting process are published on the Canadian TBI Guideline website. The recommendations are freely available bridging the gap in MOD-SEV TBI care knowledge translation, especially where access to primary publications is limited or cost-prohibitive.

Visitor engagement patterns for the Guideline website suggest that it is consulted by users from around the world looking for guidance on MOD-SEV TBI care.

Since adopting the Living Guideline update model, the Guideline website has substantially grown its audience. Over the period of 12 months from January to December 2024, the Guideline website attracted 33,350 visitors, including 30,240 accessing the English version and 3110 accessing the French version of the Guideline. This represents a 202% audience growth compared to a 12-month period from April 2021 to March 2022 when the Guideline website was accessed by 11,061 visitors from 135 countries.

Simultaneously, the supporting materials for the chapter, including rationale, system implications, key indicators, and summary of evidence, are updated to reflect the most recent review cycle. If during the chapter update process, the Expert Panel determines that there is enough new material to warrant a peer-reviewed publication, the group will collaborate on the preparation of a manuscript for submission to a peer-reviewed journal. The updates are disseminated using multiple strategies to reach different audiences, including researchers, clinicians, brain injury associations and networks, PWLE, caregivers and community care providers, and facilitate the ongoing review and uptake of recommendations. These strategies include presentations at healthcare and professional conferences and meetings, and via e-mail newsletters and social media.

As part of our ongoing quality improvement processes, we encourage feedback at any time. Feedback collected at the end of the review cycles through surveys and focus groups has been positive and has supported our processes to facilitate effective collaboration for all Expert Panel members, particularly PWLE. Feedback collected at the two Summits has added that PWLE would like to continue to have expanded roles in the review process and that the Expert Panels continue to develop/provide user friendly practical tools. There was strong support for the use of the robust multistep consensus processes to create recommendations in priority areas that have not been addressed in previous iterations, or in other relevant guidelines.

Expert Panel members and PWLE are surveyed about the working processes used by the Living Guideline. After each review cycle, the Guideline Team uses formal and informal processes to receive feedback including direct communication with the Expert Panel members, which helps identify and quickly address concerns such as the use of professional jargon, technical issues (e.g. font size, background noise) and group dynamics (e.g. making sure everyone has an opportunity to express their points during discussion). Anonymous survey feedback collected at the end of each review cycle and two Summits conducted in 2020 and 2022 to discuss the Guideline as a whole, provided valuable insights into Expert Team’s experiences with the development process and results that could be used to inform the subsequent Living Guideline review cycles.

Methods

In developing this process, the Guideline team integrated methods and tools from existing methodological frameworks. We surveyed the existing CPG development methodology through ongoing review of published literature, including Institute of Medicine’s CPGs We Can Trust framework [10], professional education, and expert’s involvement in the development and evaluation of guidelines in related fields including Canadian Stroke Best Practice Recommendations [9], Canadian Spinal Cord Injury Best Practice (Can-SCIP) Guideline [16], INCOG 2.0: Guidelines for Cognitive Rehabilitation Following TBI [5] among many others and adopted best practices. Established and effective tools were selected to support the processes, including the PEDro scale [13] for evidence evaluation and the Modified Delphi technique for consensus development. The Guideline team applied principles of project management and utilized techniques and tools, such as a Kanban board, to develop timelines and optimize processes for the Living Guideline updates. Feedback regarding the Guideline development process was collected systematically through summits and during each review cycle and used to evaluate and refine the processes.

Discussion

The Canadian TBI Guideline employs a rigorous, multicomponent process for developing and updating recommendations, ensuring that they are evidence-based, patient-centered, reflect the consensus of diverse and representative experts, and respond to the needs and priorities identified by PWLE. Involving a diverse, multidisciplinary group of experts (with relevant clinical, research, administrative, and lived experience) has created a community with a wealth of expertise and experience that allows the Guideline to maintain the up-to-date review format and produce high-quality recommendations. This process balances high-quality published evidence with practical expert consensus-based to develop recommendations supporting comprehensive patient care. By implementing the Living Guideline update model, the recommendations are updated more frequently and this allows for confidence in the relevance of the Guideline. Each chapter is routinely updated every 1.5–2 years, compared to every 3–5 years or more when following a traditional guideline development model. Having the Guideline freely available online has allowed for worldwide access to this resource, including low- and middle-income countries who otherwise might have barriers to access this guidance [1].

Multiple strategies were used to ensure the validity of the Canadian TBI Guideline recommendations including: the multidisciplinary Expert Panels include recognized experts from different jurisdictions and backgrounds to ensure integration of diverse perspectives and experiences, and mitigation of research, clinical or personal biases. Expert Panel members complete a declaration of conflict-of-interest form, and any conflicts or potential biases are addressed as needed, but there could still be some bias that could enter into the process.

The processes for literature identification, classification and review are clearly identified and followed for each Expert Panel, ensuring consistency and transparency throughout each update cycle. The bar for voting is purposely set at a high level (80% of members to achieve 75% consensus) and a process of external review is used to create confidence in the recommendations. The Guideline team and Expert Panel members also create and adapt tools to support knowledge mobilization and implementation, evaluate them, and vote on their suitability.

One concern that was raised is that funding for research often follows trends and that some areas that should have been more rigorously evaluated in research have not been. To a certain extent, including a consensus recommendation process mitigates this source of bias. People after brain injury are a very heterogeneous group, and it could be that the PWLE who were part of this process were not representative of the larger population of PWLE, and their priorities may not generalize more broadly. We tried to mitigate this by engaging a broad group of PWLE through various means and across different jurisdictions.

It is important to recognize that the level of evidence supporting each recommendation does not directly indicate how important the recommendation is to implement and the expected size of its impact. Consequently, many recommendations supported by Level C evidence include clinical practices or processes deemed most important to implement and monitor during rehabilitation for people who have sustained a TBI and most likely to have a positive impact on the outcomes. To address this concern and distinguish the recommendations that are important to implement, recommendations deemed important by the Expert Panels are marked as Fundamental or Priority in addition to the icons indicating the level of supporting evidence (A, B, or C).

Although the living Canadian TBI Guideline has been developed for the use by healthcare providers in a variety of rehabilitation settings, PWLE and their families are an end-user group. The fact that PWLE have been involved in each step of the Guideline update process and are involved in reviews of the Guideline content and structure helps ensure that the Guideline will increase in usefulness for both clinicians and PWLE. The feedback from PWLE has been integral to developing the tools and resources that healthcare providers can provide for their patients but also that PWLE can guide their own understanding of the recovery process and help build reasonable expectations of their rehabilitation.

The Guideline team took steps to integrate PWLE into the Expert Panels as members and promote equal participation and meaningful involvement of PWLE in the Guideline development process. PWLE were invited to join multidisciplinary Expert Panels as members and participated in co-lead positions on several topics. Facilitators and other Expert Panel members made sure everyone had the opportunity to express their ideas and participate in the discussion. Strategies were used to help mitigate barriers to participation (e.g., French-speaking experts, experts who may have language or cognitive impairment, and personal factors) and minimize the use of professional jargon. When scheduling review meetings over videoconferencing, accessibility was considered (e.g. use of screen sharing with appropriate font size, providing materials to review in advance of the meeting, and scheduling meetings during hours that facilitate PWLE participation.

After each review cycle, the Guideline Team used formal and informal processes to receive feedback including anonymous survey feedback and direct communication with the Expert Panel members. Summits were conducted in 2020 and 2022 to discuss the Guideline as a whole and solicit feedback from all experts. The process of developing the Living Guideline methodology itself is living and evolving, looking back at what was effective and what should be improved and using these insights to map out the next steps in the process.

The outcome of this process does have some limitations, but the process used could be generalized to any setting or context where a guideline is appropriate. It should be acknowledged that the content of the consensus guidelines may not be relevant to all models of healthcare delivery because the Canadian TBI Guideline has been developed in a high-income country with publicly funded healthcare, but the multistep process used could be tailored to any setting. The criterion of 75% agreement with the recommendation by at least 80% of the panel members required to adopt recommendations was selected by the Expert Panel; however, it could be that a higher standard might be required.

Conclusions

The Canadian TBI Guideline uses a rigorous, transparent, and replicable evidence review and expert consensus process to create high-quality recommendations for MOD-SEV TBI rehabilitation that respond to the needs of PWLE and Guideline end-users. This multistep approach was developed and implemented to bridge the gap between published research and clinical practice and provide concrete and timely guidance, particularly in the areas where published research is absent or insufficient.