Conduct of scoping reviews in health professions education

Scholarly reviews involve thorough evaluations and integrations of research, practices and advancements within a field. These reviews may take various forms, including systematic reviews, narrative reviews, scoping reviews and meta-analyses, each with its specific methodology and focus. Regardless of the chosen methodology, there is a structured and standardized approach to any scholarly review [1]. Scoping reviews in health professions education (HPE) are common knowledge syntheses that aim to explore and map out the existing literature on a particular topic or question. They are broader in scope than traditional systematic reviews, aiming to provide a comprehensive overview of the available evidence. Scoping reviews help identify gaps in knowledge, inform future research directions and support evidence-based decision-making in HPE. Over the last 5 years, a notable surge in scoping reviews has occurred within the HPE field, with a discernible increase of 50%. Moreover, within the simulation literature, scoping reviews have grown with a parallel trajectory, marking a notable increase of 53% during the same timeframe (PubMed Search data, Appendix 1).

Arksey and O’Malley originally described their scoping framework in 2005 [2], which HPE scholars adopted to guide the conduct of early scoping reviews. Levac et al. extended this original scoping study framework to enhance the methodological process and provide additional guidance on how to implement the six-step approach [3]. The original Arksey and O’Malley method defined and endorsed an optional sixth step that was termed ‘consultation exercise’ to inform and validate the main findings. In 2020, JBI (formerly known as the Joanna Briggs Institute) once again refined the approach for conducting scoping reviews [4] by extending methodological guidance and describing a framework for knowledge user (KU) inclusion [5]. This framework comprised multiple enhancements to the Levac et al. recommendations, listing various roles to include in the review team including information technologists, experts and KU throughout the review process.

A recent study examining published scoping reviews in HPE identified potential impacts of this methodological approach, highlighting limitations and offering recommendations to strengthen the review’s impact [6]. These recommendations specifically, included (a) linking research questions to the rationale of the review, and (b) considering the valuable but routinely ill-defined role of external consultation since engaged research contributed to high-impact, high-quality research [6]. Buus et al. critically reviewed consultation exercises in scoping reviews employing Arksey and O’Malley’s sixth optional step and found poorly described consultation processes, often with inadequately reported effects of these consultations on review findings [7]. Buus et al. re-imagine the consultation process as a consideration throughout the review process rather than an optional or distinct final step [7]. While this positive development enhances consultation activities, the lack of well-defined processes or frameworks for this engagement may limit engagement and contribution. To this end, the JBI Scoping Review Methodology Group offers more specific guidance for KU engagement and encourages scoping review teams to adopt a co-creation approach [5]. Table 1 presents an overview of the evolution of the published recommendations for the conduct of scoping reviews in HPE, and specifically KU engagement.

Table 1:

Published recommendations for engaged research in the conduct of scoping reviews

Authors	Recommendations for engagement in scoping reviews
Arksey and O’Malley (2005)	• Defined the five-stage scoping review framework. 1) Identify the research question; 2) Identify relevant studies; 3) Study Selection; 4) Charting the data 5) collating, summarizing and reporting the results. • Proposed as an optional extra (sixth) and final step in the framework. • Described as a Consultation Exercise, with three groups adding value to their review.
Levac et al. (2010)	• Described consultation as an ‘essential component’ in the framework. • Suggested preliminary findings can be a catalyst for the consultation that should occur after Stage 5 of the framework. • Recommended defining the objective of the consultation including: 1) Who will be invited to participate? 2) What they will be asked to contribute? 3) How data will be collected, analysed, reported and integrated? • Recommended opportunities for knowledge transfer/exchange with KUs in the field.
Buus et al. (2022)	• Reviewed reported consultation exercises in published scoping reviews. • Identified the process was poorly described, under-reported and lacked any description of impact or outcome of the consultation process on the review. • Recommended the development of methodological guidance for the conduct and reporting of KU involvement for rigour and transparency. • Recommended KU involvement be considered mixed-methods research, and should be approached and reported accordingly.
Pollock et al. (2022)	• Encouraged KU involvement in scoping reviews. • Favoured a shift from consultation to co-creation. • Provided detailed guidance for each stage of review development. 1) Strategy development; 2) Scoping review protocol development; 3) Conducting the review; 4) Disseminating results of the review. • Recommended KU involvement from the start and throughout the review. • Provided a summary of nine key considerations for research teams in engaging KUs.

The importance of language: knowledge users

The term ‘stakeholder’ routinely used in scoping review literature and engaged research was deliberately excluded from this paper due to its negative connotations and the potential to perpetuate colonial harm, which may undermine its intended positive impact [8]. This conscious decision reflects the authors’ commitment to examining and evolving terminology in the field of HPE, acknowledging the significance of language. Consequently, we use the term ‘knowledge user’ or ‘knowledge users’ to describe individuals outside the core review team who engaged in the research process. In this paper, the term ‘knowledge user’ is appropriate as the individuals involved were either working as a simulated patient (SP), or working directly with SPs and utilizing and researching SP Methodology, and therefore, this group would use the findings of this scoping review.

Consultation and co-creation in scoping reviews

In HPE, co-creation is defined as collaborative knowledge generation by academics working alongside other KUs [9] and moves beyond a potentially tokenistic consultation approach to authentic KU participation [7]. Pollock et al. suggest a shift from traditional consultation practices towards co-creation in scoping reviews to generate richer findings and improve dissemination and implementation. However, these authors advocate for greater transparency in reporting processes highlighting the need for frameworks to guide KU engagement [5]. A critical challenge remains: while scoping review teams may use various models to support consultation and co-creation processes, we lack a full understanding of when and how to best apply these models. Furthermore, the distinction between ‘consultation’ and ‘co-creation’ within the context of scoping reviews remains unclear. To advance these efforts, we believe a discussion on their relative differences, merits and advantages would provide additional clarity on the various approaches. Co-creation can be time-consuming and resource-intensive, relevant contextual factors that inform decision-making [5,10]. Although co-creation is an important and lofty goal, it is not always feasible. A better understanding of alternative models would help researchers make more informed choices when designing their scoping reviews.

In this paper, we describe our approach to engaged research in scoping reviews and outline a version of co-creation we achieved through our consultation activities. In doing so, we use ‘co-creation’ and ‘consultation’ to distinguish potentially different activities. We believe that a clear differentiation between these terms would advance our discourse:

Consultation: information gathering activities by academics seeking insights and feedback from KUs on various aspects of the conduct and/or results of a scoping review. KUs are consulted but not directly involved in decision-making, and are not accountable for the project.

This paper proposes an update to the methodology for conducting a scoping review and provides a fine-grained description of our process, which we have termed ‘co-creation through consultation’ conducted by AJD, CS, CC, CM, MOT, DN, MA, FC, NMN, RMA and WE [11]. In this essay, we describe and reflect on the impact of our ‘co-creation through consultation’ approach, as a practical application of the widely accepted six-step scoping review methodology [2,12] within our specific review context. Our approach differs from previous guidelines and recommendations described in Table 1, as we propose an approach that involves longitudinal KU consultation, from review inception to manuscript publication. However, our approach has broader implications for future KU engagement practices. Our review team, with scoping review methodology expertise, aims to encourage future researchers and academics interested in KU engagement to adopt, evaluate and refine this process.

Context: Best Evidence in Medical Education Guide No. 86

We implemented this process in our Best Evidence in Medical Education (BEME) scoping review, BEME Guide No. 86, that investigated simulated participants’ training for role portrayal and feedback practices in communication skills training for HPE [11]. A simulated person is an individual who portrays the role of a patient (SP), family member, healthcare provider or other role, in order to meet the objectives of the simulation, for learners to develop and practise their clinical skills [14].

The BEME review process involved additional steps to the standard scoping review process, including (a) topic registration with the BEME committee; (b) the preparation, submission and BEME committee acceptance of our scoping review protocol, and (c) development of a 6-month progress report submitted to the BEME committee. Figure 2 illustrates the additional steps involved in the BEME process for scoping reviews.

Figure 2:

Best Evidence Medical Education (BEME) process as it aligned to the scoping review activities in Doyle et al. [11].

Core Review team and KU recruitment

The BEME scoping review project team included the Core Review team and the KU team. The Core Review team comprised simulation researchers and educators at an Irish health professions university. The Core Review team was grant-funded to conduct this research by the Irish Higher Education Authority Innovation and Transformation Program 2021–2022. The Core Review team developed a targeted approach to the recruitment of KUs. Six KUs with varied backgrounds participated: an experienced SP with a background in academia (MA), and KUs with expertise in SP methodology and SP education from Europe, UK, USA, Canada and Australia (CC, FC, RMA, NMN, DN).

KUs were selected on the basis of their significant expertise in SP methodology and SP education. Our network of collaborators and previous colleagues served as a sample population and was utilized to identify experts in SP feedback practices and training methodologies. No one declined our invitation; this may have been due to the fact that this review began in 2021, during coronavirus disease 2019 (COVID-19) restrictions, and there was an increased focus on remote engagement. This may not have been the case in a pre-COVID world where such an undertaking may have been abstract before many in the HPE field were compelled to pivot to online. Our core review team had expertise in patient and public involvement and recognized the importance of SP feedback in this particular review. Therefore, as an active KU, their feedback was sought.

The geographically diverse group of KUs brought broad expertise to ensure that KU input supported the proposed review objectives. Once potential KUs were identified and purposively sampled, they were invited by email to individual online meetings to discuss our research objectives and discuss their involvement. The SP who engaged in the co-creation through the consultation process was remunerated for their involvement in line with our institutional SP Engagement Policy.

We viewed KUs as partners throughout the review process, from developing review questions to publishing the manuscript (see Figure 1). Careful ethical consideration was given to the KUs’ level of engagement, and in line with International Committee of Medical Journal Editors (ICMJE) guidance (https://www.icmje.org/), we determined that the KU contributions clearly warranted co-authorship on the manuscript.

Preparation for KU engagement activities

KU engagement required a schedule of preparatory activities (see Figure 1 – Preparation column). These activities were necessary for KUs to have a detailed understanding of each stage, alongside their required input. At most stages, this information was sent in advance of meetings, utilizing synchronous and asynchronous means of engagement to optimize contact time with KUs. Information was targeted to the audience and often included documents for KUs to review and a PowerPoint slide deck outlining the meeting agenda, an overview of progress between meetings, and the planned activities.

As we progressed, stage-specific data were included in the circulated documents, such as drafts of the search strategy, potential inclusion/exclusion criteria and literature samples to be included or excluded from the review. Additionally, the PowerPoint slide deck served as a communication tool, enabling the core review team to present the KUs with critical challenges or questions that required input and collaborative discussion.

This preparation for meetings and information sharing via email or shared repositories ensured that everyone involved was briefed and had time to review information, form opinions and offer recommendations ahead of live meetings. As the review progressed, drafts of the manuscript were circulated to KUs via email, often in manageable sections or chunks to facilitate more streamlined review and comment, inviting feedback and input before further meetings. The regular meeting schedule with KUs helped manage expectations and deadlines for feedback and iterative refinement.

KU engagement activities

KU engagement involved both synchronous and asynchronous communication techniques to interact purposefully and directly, including video conferencing; email; shared repositories for data and literature; PowerPoint presentations with accompanying pre-recorded verbal explanations and in-person discussions at international meetings. This blended approach led to productive and focused interactions with KUs. With our KU recruitment strategy, we sought a geographically diverse group of experts in SP methodology. As we achieved this aim, mutually convenient times across various continents meant that meetings with the full review team and all KUs were not always possible. To ensure inclusion, we recorded meetings for those who could not attend and arranged one-to-one meetings to accommodate KUs in diverse time zones. After considerable review progress or an acute need for feedback and KU input, we circulated pre-recorded PowerPoint presentations to share interim data and associated queries, enabling KUs to engage, digest and reflect in their own time and then feed into our collective deliberations asynchronously. We conducted the BEME review over an 18-month period; throughout this time, international research meetings and conferences facilitated opportunistic in-person discussions with review team members and KUs. Our targeted approach for KU engagement respected KU time and facilitated their invaluable input into decision-making and the development of future plans.

At various stages from inception to conclusion, KUs contributed their considerable and specific content expertise to help guide and direct aspects of the review process. This included thoughtful discussions regarding review objectives, research questions, search strategy including possible databases, and the identification of data through hand searching. Importantly, KUs tapped into their professional networks via mailing lists and at networking events to identify training materials. In co-designing the review protocol, we acknowledged that potentially valuable data may have been unpublished and therefore potentially inaccessible to us. KUs reached out to their networks and to professional organizations including the Association of SP Educators (ASPE) and the Association for Simulated Practice in Healthcare (ASPiH) to identify gray literature such as relevant SP training materials.

Impact of KU engagement in the BEME scoping review

The impact of KU engagement in the scoping review was determined by the consensus of the review team and included valued KUs’ perspectives. Initial KU engagement helped to identify research questions relevant to the SP research community. This, in turn, refined the scope of the review and our research objectives, and shaped data included in our review through iterative refinement of inclusion/exclusion criteria. Initially, we considered how SPs were trained primarily for feedback practices. Through KU consultation, we identified that in their role portrayal, SPs are frequently instructed to or can improvise feedback in their role. We subsequently captured this tranche of data, adding richness and depth to our findings about feedback practices because of the KUs’ valuable inputs.

The co-creation of our search strategy was crucial to understand the nature of the data required to answer our research questions. Our primary focus was SP feedback for communication skills training, not technical skills or clinical performance per se. While this limits the evidence base, we wanted this level of specificity. KUs supported and were able to suggest several approaches to building and refining a robust search strategy to address this focus. Early in the process, we identified that data might exist outside peer-reviewed literature in unpublished training materials and local procedural guidelines. Our KUs reached out to their professional organizations and networks to help to identify this unpublished information. Ultimately, this call for data yielded no substantial additional return, however, that in itself reinforced the need for more thorough and rigorous reporting of SP training processes which in turn shaped the implications for practice and our recommendations for future research.

From the outset, our geographically diverse group of KUs was uniquely positioned to help identify gaps in current practice related to the global implementation of SP methodologies. KU involvement ensured the broader applicability of recommendations for future research beyond our context and made the implementation of SP methodology more generalizable. This was highlighted in a paper evaluating the use of BEME reviews, in which the authors identified a need for relevant knowledge syntheses that actually inform educational research and practice [15]. Our KU engagement ensured that our recommendations were contextualized for HPE readership, and identified tangible and important future research directions.

Through the data extraction process, we identified cultural differences and sensitivities in the literature related to specific discourses around SP methodologies. Historically, the terms ‘simulated’ and ‘standardized’ were used interchangeably to describe the SP role, a finding our review also identified. However, our geographically diverse KU group provided more cultural context and sensitized the review team about certain pervasive yet dehumanizing technical language choices that might perpetuate unhelpful stereotypes in HPE [16]. For example, SPs can take on roles other than patients, such as family members or healthcare staff during simulated encounters, in which case they have traditionally been referred to as ‘confederates’. However, this term was removed from the most recent edition of the Healthcare Simulation Dictionary [14] in favour of ‘embedded participants’ [17], a more neutral term. This invaluable KU perspective allowed us to integrate this evolving discourse and communicate our findings and recommendations using culturally inclusive language. Furthermore, we were able to invite the research community to remain mindful of this evolving discourse in their own future publications.

Our core review team oversaw two parallel processes: (a) managing BEME expectations, timelines and peer-review process, and (b) coordinating our own milestones for completing the review process in a timely manner. Although BEME added an additional element, the BEME process also prompted additional consultation opportunities, scaffolded the review process, and provided discrete time points to reflect on the scope and direction of the work. The documents produced at the topic registration and protocol development stages for BEME (see Figure 2) proved to be valuable artefacts that structured communication for the core review team and KUs. Also, the 6-month BEME report represented an obvious reflection point that accelerated and refined plans to complete review activities and write the manuscript. The BEME scoping review protocol included a comprehensive description of the rationale, anticipated outcomes and findings of a pilot search to demonstrate sufficient literature existed to perform the review. When the BEME Collaboration reviewed our protocol, they raised methodological questions related to the suitability of the review question to the proposed scoping methodology. Our KUs significantly shaped our response to the BEME Collaboration about our planned scoping approach and subsequently alleviated BEME concerns, reinforcing our articulation of the need to identify and synthesize the available evidence to guide educational practice. This dialogue with KUs then identified areas for further research and clarified our understanding of SP’s specific contributions to communication skills training in HPE.

Reflections and lessons learnt

We have described and reflected on the impact of our ‘co-creation through consultation’ approach in a specific scoping review context. However, our approach has more general implications for future practice in KU engagement. Our core review team, in conjunction with a scoping review methodology expert (HC), hopes that by describing our approach in a transparent way and sharing our reflections, future researchers and academics interested in KU engagement may adopt our approach in future scoping reviews and indeed evaluate and further refine this process.

Managing a collaborative review team of 15 individuals, with some members participating remotely, could potentially lead to tension or conflict. First, the lack of in-person interaction may hinder the development of strong relationships and open communication channels, potentially resulting in misunderstandings or miscommunication. Additionally, differences in time zones and schedules may challenge synchronous collaboration, delaying feedback and decision-making processes. Relying exclusively on digital communication tools may lead to technical issues or barriers in accessing important information, further complicating the workflow. Furthermore, disparities in work styles, cultural backgrounds or personal circumstances among team members could contribute to differences in expectations, workloads or levels of engagement, potentially causing friction within the team. However, this was not our experience and we believe that our positive outcome may be attributed to our proactive approach, fostering clear communication and team cohesion to mitigate potential conflicts.

The core review team adopted a targeted approach to engagement both to respect KU time and to optimize their contribution. Through the preparation of resources, we curated which information to present to KUs and in this way, the core team drove the review. From the outset, however, we consulted KUs to co-create our process of collaboration at specific stages of the review. We never expected KUs to engage with ‘raw data’ (e.g. databases of unscreened manuscripts) to prevent fatigue and minimize the burden on this in-demand group of experts. Additionally, we did not invite KUs to participate in the review activities such as screening or data extraction. We chose this specific approach for two main reasons. First, BEME placed considerable time constraints with an 18-month period from protocol acceptance to manuscript submission (see Figure 2). While this timeline facilitated an efficient and focused review workflow, the prescribed deadline meant the core review team needed to be responsible for active work at each stage of the review to meet internal milestones. Second, our geographically diverse group of KUs crossed extreme time zones, making it impossible to consistently arrange suitable meeting times for all 15 members. Screening and data extraction require consensus within and across pairs, and this was best achieved by the core review team. Although our approach suited our context, we do not know if this impacted the finalized review. However, where there were contentious issues, we shared these at KU meetings to seek wider input. This gave us confidence in the resolution of issues.

Considerable work was involved in the preparation and engagement activities for KUs. Scholarly reviews are effortful and involve thorough and meticulous planning. Significant additional work was required to prepare materials, synopses, summaries and examples for KUs. Preparation and pre-briefing activities demanded an additional layer of organization and planning, and the process of scheduling meetings for the geographically diverse KU group posed additional challenges. To overcome these challenges, we used synchronous and asynchronous means of engagement and polling software to choose meeting times. Additionally, where necessary and at their request, we followed up with individual KUs who were unavailable at scheduled meeting times. Our preparation, engagement and integrated approach enabled efficient engagement of KU resources, easing the decision-making process.

We invited our KUs to participate based on their expertise related to the proposed review. While five KUs were chosen due to their experience as academics and researchers of SP methodologies, one KU was chosen because they were an experienced SP. One SP was engaged in this work as this study occurred in parallel with other research conducted by our team and we were mindful of our expectations from our SP colleagues and their professional and research commitments.

Our review project team was familiar with scholarly reviews and AJD provided scoping review just-in-time training via initial project team meetings. This familiarity and training helped the review team eliminate many challenges associated with trying to understand the specific processes associated with reviews. Without this familiarity and training, our team would have required additional resources and preparation, necessary for successful collaboration. KU involvement should not be attempted without adequate preparation and appropriate consideration. While these types of resources take time to produce, they are critical for effective collaboration. Without them, collaboration risks becoming tokenistic. Our model also clarifies concerns about remuneration, which has been described as an important but challenging aspect of patient and public involvement in research [18]. Our model could be used to clearly identify the contact hours and KUs’ preparation time, facilitating an explicit description and quantification of their engagement.