Study protocol of a national multicentre prospective evaluation study assessing the validity and impact of the Dutch Paediatric Early Warning Score (PEWS) in the Netherlands
•,,,,,,.
...
Abstract
Introduction Early recognition of clinical deterioration and timely intervention are important to improve morbidity and mortality in paediatric care. The Paediatric Early Warning Score (PEWS) is a scoring system aiming to identify hospitalised children at risk for deterioration. Currently, there is a large heterogeneity of PEWS systems in the Netherlands, with a considerable number remaining unvalidated or self-designed. Therefore, a consensus-based Dutch PEWS has been developed in a national study using the Core Outcome Measures in Effectiveness Trials initiative. The Dutch PEWS is a uniform system that integrates a core set of vital parameters together with pre-existing risk factors and uses risk stratification to proactively follow-up on patients at risk (so-called ‘watcher patients’). This study aims to validate the Dutch PEWS and to determine its impact on improving patient safety in various hospital settings.
Methods and analysis This national study will be a large multicentre evaluation study, in which the Dutch PEWS will be implemented and evaluated in 12 hospitals in the Netherlands. In this study, a mixed methods methodology will be used and evaluated on predefined outcome measures. To examine the validity of the Dutch PEWS, statistical analyses will be undertaken on quantitative data retrieved from electronic health records. Surveys among physicians and nurses; semistructured interviews with healthcare providers and parents; and daily evaluation forms are being conducted to determine the impact of the Dutch PEWS. The study is being conducted from December 2020 to June 2024.
What is already known on this topic
There is an extensive heterogeneity of Paediatric Early Warning Score (PEWS) systems not only in the Netherlands but also worldwide.
The validation of PEWS systems is limited to University Medical Centers. None of the systems validated is considered superior in the performed meta-analysis.
What this study adds
This study will provide valuable insights into a PEWS system’s performance in the context it is used as well as the chain of care between hospitals.
How this study might affect research, practice or policy
This study protocol can serve as a blueprint for validating PEWS systems in other countries to solve problems with heterogeneity. The utilisation of a nationwide PEWS system applicable to the entire chain of interhospital care potentially increases the quality and safety of paediatric care.
Introduction
Serious adverse events in hospitalised patients are often preceded by deteriorating physiological parameters, such as blood pressure, heart rate and respiratory rate.1 The Early Warning Score (EWS) is a numeric score aiming at early recognition of clinical deterioration to give appropriate care.2 This EWS must be embedded in a system consisting of an afferent and efferent limb. The afferent limb refers to monitoring physiological parameters. The efferent limb consists of the response to an abnormal observation, such as mobilising medical emergency teams (METs).3 METs consist of physicians and nurses trained in advanced life support. These teams are mobilised to intervene early in acute clinical deterioration and this is correlated with the reduction of morbidity and mortality.4
EWS are also developed for and used in paediatric patients who are admitted to a hospital.5 6 However, the application of the Paediatric Early Warning Score (PEWS) is complex due to cognitive and physical variances between different age categories.7–9 It is believed that PEWS systems facilitate healthcare providers to monitor physiological deterioration and may improve situational awareness.6 In addition, there is evidence that PEWS systems have the potential to enhance multidisciplinary teamwork, communication and confidence in recognising and making clinical decisions in worsening patients.9 Despite the considerable amount of literature regarding PEWS, there is limited evidence supporting its effectiveness and several systematic reviews failed to demonstrate more than a positive trend in clinical outcomes.2 9–11 Further, most validation research on PEWS systems is conducted in University Medical Centers (UMCs), limiting its generalisability to large teaching hospitals and general hospitals with different infrastructure, case-mix of patients and more time-consuming paediatric intensive care unit (PICU) access.12
In 2011, the Dutch national patient safety initiative ‘Prevent Harm, Work Safely’ (VMS safety programme), recommended paediatric departments to implement a PEWS system. Most paediatric departments in the Netherlands have since adopted this recommendation. However, there is a substantial heterogeneity of PEWS systems in clinical practice of which many are unvalidated or self-designed.13
Due to the lack of an evidence-based PEWS system, a consensus-based Dutch PEWS has been developed in a national study using the Core Outcome Measures in Effectiveness Trials initiative in 2019.14 15 The establishment of the Dutch PEWS created a uniform system that integrates the following two components for predicting clinical deterioration: (1) a core set of vital parameters and (2) the assessment of pre-existing risk factors (so-called ‘watcher signs’). The watcher signs include worried sign (by parents and/or healthcare providers), predefined risk treatments and local contextual factors. The Dutch PEWS applies risk stratification to categorise each clinical patient into a risk category with appropriate measures to monitor the clinical course closely and prevent deterioration. This generic instrument is also modifiable for local contexts, this enables implementation in various clinical contexts. The infographic of the Dutch PEWS system is presented in figure 1.14 16–19
Infographic Dutch Paediatric Early Warning Score (PEWS).
This research project aims to answer the following research questions:
What is the validity of the Dutch PEWS in various clinical contexts in hospital care? How does this relate to the application of risk stratification?
What is the impact of the Dutch PEWS in different clinical contexts in hospital care? How does this relate to local adaptation of the system?
What is needed for proper implementation of the Dutch PEWS in hospital care?
The primary goal is to implement and study Dutch PEWS in daily performance, thereby answering the research questions above. Due to lacking evidence on performance of systems currently used in the Netherlands, no quantitative comparison can be made with the Dutch PEWS.
Methods
Study design and context
This national research project is practice oriented and can be considered as an evaluation study. The Dutch PEWS will be implemented and evaluated in 12 hospitals. The implementation will be carried out in three tranches, with four hospitals participating in each tranche.
The study is primarily based on the four-step model for improvement: the plan–do–study–act cycle.20 In this convergent parallel mixed methods study, data will be prospectively collected. Prior to setting up this study, an internal pilot study was performed to evaluate the feasibility of the statistical methods.
The study is conducted by the Dutch PEWS research collective consisting of experts in the field representing relevant national care associations [i ii]. Patient representation is covered by the participation of the Dutch Child & Hospital Foundation.
Setting and participants
The hospitals are selected by block randomisation considering hospital setting, geographical location and unique patient groups (case-mix). The participating hospitals are a representative allocation of general hospitals, large teaching hospitals and UMCs (including one national paediatric oncology centre) in the Netherlands. Table 1 and figure 2 show an overview of the participating hospitals and the geographical distribution. The projected time period for the study is from December 2020 to June 2024.
Geographical distribution of the participation hospitals in the Netherlands.
The population that will be studied is heterogeneous and includes all children admitted to a general paediatric (oncology) department, aged between infants (>37 weeks’ gestational age) and 18 years. The eligibility criteria are presented in table 2 .
Table 2
|
Eligibility criteria for hospital selection
Outcome measurements
In this study, predefined outcome measurements will be evaluated as reported in table 3. The outcome measures are divided into three categories: validity, impact and implementation. In addition, the participating hospitals could also formulate local outcomes.
Table 3
|
Outcome measurements hospitals
The outcome measures for validity (1–4) aim to indicate the ability of the Dutch PEWS to early recognition of clinical deterioration. Unplanned PICU admission and resuscitation (1–2) are commonly used outcome measures but occur rarely in large teaching hospitals and general hospitals.6 12 To study PEWS’ performance in these settings more closely, two additional, more frequently occurring outcome measures (3–4) were formulated that might provide valuable insights in timely detecting these patients. The outcome measures for impact (1–7) focus on healthcare providers’ experiences of the Dutch PEWS on quality of care. However, the validity and the impact of the Dutch PEWS can only be measured if the implementation (8) in the hospitals is proper. Proper implementation could be defined as the sustainable use of the system in daily practice. It should be in the capillaries of the hospital staff and processes. This of course is hard to measure. Hence, a surrogate protocol adherence indicator and qualitative data will be used. The protocol adherence indicator is based on a calculation of the total number of completed PEWS forms per hospital divided by the number of admission days in the study period, this is a rough indicator which mainly helps to show trends in protocol adherence over time in a specific hospital. Therefore, no specific outcome was defined to indicate proper implementation. In addition, the implementation process and outcomes will be measured qualitatively using surveys and interviews. Together, this provides valuable insights towards local implementation (difficulties) over time and effects of improvement measures.
Data collection
Quantitative data
Most of the participating hospitals use the electronic health records Epic and ChipSoft in which the Dutch PEWS is fully integrated to reduce workload. During this study, sessions will be organised to further improve the digital usability within these health records in codesign with manufacturers. To examine the validity of the Dutch PEWS, hospitals will provide a baseline dataset and additional data to the research group. During the study period, nurses on duty will collect PEWS data from each patient at least once during each shift as part of regular care. Based on these PEWS data, the risk category will be determined at least once every 24 hours. All scores will be registered in the electronic health records.
The participating hospitals will upload the data in their personalised Castor Dutch PEWS research environment. Castor is a cloud-based clinical data management platform that enables to securely capture and integrate data.21 The baseline dataset provides information about: (1) the course of PEWS during admission (PEWS component), (2) the absence or presence of pre-existing risk factors (pre-existing risk factor component), (3) the classification of the patient in a risk category: high-medium-standard risk (Dutch PEWS risk category), (4) the outcome measures paediatric patients did or did not accomplish during the admission period and (5) basic patient characteristics. It is expected that data from around 18 000 admissions is eligible for inclusion. In case of ambiguities or missing data, additional data will be requested from the concerned hospital.
To determine whether achieved outcome measures should be included or excluded from the analyses an algorithm will be used. The algorithm has been established by the research group and is presented in table 4.
Table 4
|
Algorithm for the inclusion and exclusion of achieved outcome measures
Qualitative data
The impact and implementation degree of the Dutch PEWS will be determined through surveys, semistructured interviews and daily evaluation forms. These data will be collected at predetermined moments and are presented in figure 3. The questions in the surveys, interviews and evaluation forms are mainly based on the previously mentioned outcome measures, literature and experience from a preliminary study.5 6
Surveys will be provided to physicians and nurses before implementing the Dutch PEWS (T=0) and at 3, 6 and 12 months after implementation (T=3, T=6, T=12). T=3 is mainly focused on implementation. The surveys contain closed and open questionnaires. The closed questionnaires consist of (1) one question for consent, (2) background questions (eg, years of work experience) and (3) statements based on a 7-point Likert scale. Themes are the experiences with the current PEWS, effects of the Dutch PEWS, implementation (difficulties), usability of the Dutch PEWS, communication effects, chain of care, suggestions for improvement and overall rating. The surveys will be conducted digitally through the software SurveyMonkey. SurveyMonkey will be managed by the Dutch society for Paediatrics.
Semistructured interviews are scheduled with physicians and nurses at T=0, T=6 and T=12. Two of the six interview guides are presented in online supplemental appendices 1 and 2. At T=9, the study proceedings are evaluated with local project leaders. Besides, the experiences of parents of children who are admitted to a participating hospital will also be studied by semistructured interviews. After containing consent, interviews will be conducted digitally and recorded.
Outcomes measures must be recorded on occurrence using daily evaluation forms (table 5) during everyday shift transfers.
Table 5
|
Daily evaluation form
Data analysis
Quantitative data
The statistical software Statistical Package for the Social Sciences (SPSS Statistics 29) will be used for statistical analyses.22 The threshold for statistical significance will be p<0.05. To measure the uncertainty for the sample, 95% CIs will be computed.
Descriptive statistics will be used to describe the baseline characteristics of the sample. The following measures will be used to estimate the validity: sensitivity, specificity, positive and negative predictive value, receiver operating characteristic curve and area under the curve. The sensitivities will also be computed over time to examine the course of sensitivity in the 24 hours before an outcome measure occurs. The differences between the participating hospitals and hospital settings will be analysed using the previously mentioned measures and descriptive statistics.
Qualitative data
The audio records of all interviews will be transcribed verbatim and analysed using the software programme ATLAS.ti.23 At the baseline measurement of the first tranche, relevant aspects were fragmented and the initial categories and codes were formed. These categories and codes will be used as the basis for subsequent interviews. After open coding, a thematic analysis will be performed using an inductive strategy.24 The analyses for each interview will be performed by two independent researchers. Regular meetings for consistency and iteration will be organised to reach consensus. After analysing all interviews individually, central themes will be constructed using frameworks for implementation, communication and situational awareness. Member checking will be performed during sessions with project leaders and the research group.
Current status of the study
Since the start of the national research project, the first phases have been completed. At present, the Dutch PEWS has been implemented in all 12 participating hospitals. In addition, three hospitals (from tranches 1 and 2) have provided quantitative data into their Castor environment. Regarding the qualitative data collection, all predetermined evaluation moments have been completed in tranches 1 and 2 (T=0, T=3, T=6, T=9, T=12). The evaluation of the implementation (T=3) is currently being finalised in tranche 3. Data so far shows that the study protocol is properly designed to meet its purposes.
Recruitment and consent
Before the start of the study, a cooperation agreement was signed by project leaders and hospital board members. In addition, wherever appropriate, a supplemental declaration of enforceability was acquired from the local medical ethics committee.25 Paediatric patients and their parents will be informed about the study at the start of admission to a participating hospital. If patients and/or parents did not wish to participate, this will be registered in the electronic health record by the paediatrician to exclude their data.
Data collection and deposition
All data from electronic health records will be provided in a pseudonymized way to the research group. According to General Data Protection Regulation (Algemene Verordening Gegevensbescherming) all involved patients will be coded by a subject number.26 The participating hospitals will keep the key file to the pseudonymised data in a separate registry.
The Dutch society for Paediatrics will share the data of the surveys anonymised via a secure route and then destroys the original data. All interviews will be recorded and transcribed verbatim using pseudonymisation. All audio records will be destroyed immediately afterwards. The transcripts and the keys will be kept on a protected disk with the documents separated by two layers.
All data together will be stored for maximally 10 years on a protected disk within Radboudumc. The protected disk is only accessible to the research group of the Dutch PEWS.
Incentives
Paediatric patients admitted to a hospital, their parents, project leaders and healthcare providers do not receive any special incentive, compensation or special treatment by participating in the study.
Contributors: JF and JdG conceptualised the Dutch PEWS study. JS and ML wrote the original research protocol, with the contribution of JdG, JB, HR, JH-S and JF. JS, JdG and JF wrote the manuscript of the study protocol. All authors approved the final manuscript.
Funding: This study is funded by the Foundation Quality Funds Medical Specialists (in Dutch: Stichting Kwaliteitsgelden Medisch Specialisten; funding number 6908773).
Map disclaimer: The inclusion of any map (including the depiction of any boundaries therein), or of any geographic or locational reference, does not imply the expression of any opinion whatsoever on the part of BMJ concerning the legal status of any country, territory, jurisdiction or area or of its authorities. Any such expression remains solely that of the relevant source and is not endorsed by BMJ. Maps are provided without any warranty of any kind, either express or implied.
Competing interests: None.
Patient and public involvement: Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Data availability statement
This is the study protocol and there is no data available/published yet.
Ethics statements
Patient consent for publication:
Not applicable.
Ethics approval:
This study is conducted according to the principles of the Declaration of Helsinki, set up by the World Medical Association (WMA), Fortaleza Brazil, October 2013. The CMO Region Arnhem-Nijmegen approved the study and a non-WMO waiver is provided (2020-6966) applicable to all study locations. The Dutch PEWS research collective will facilitate the dissemination of the study findings. The findings will be published in peer-reviewed journals, presented in conference presentations, and shown on the website (www.dutchpews.com).
Petersen JA. Early warning score challenges and opportunities in the care of deteriorating patients. Dan Med J2018; 65. Google Scholar
Smith MEB, Chiovaro JC, O’Neil M, et al. Early warning system scores for clinical deterioration in hospitalized patients: A systematic review. Ann Am Thorac Soc2014; 11:1454–65. doi:10.1513/AnnalsATS.201403-102OC•Google Scholar
Williams G, Alberto L. Recognizing and Managing the Deteriorating Patient: the Role of Rapid Response Systems, Critical Care Outreach Nurse and Medical Emergency Team. 2020;
de Vries A, Draaisma JMT, Fuijkschot J, et al. Clinician perceptions of an early warning system on patient safety. Hosp Pediatr2017; 7:579–86. doi:10.1542/hpeds.2016-0138•Google Scholar
Teheux L, Verlaat CW, Lemson J, et al. Risk stratification to improve pediatric early warning systems: it is all about the context. Eur J Pediatr2019; 178:1589–96. doi:10.1007/s00431-019-03446-0•Google Scholar
Stichting Kind en Ziekenhuis. Kinderen zijn geen kleine volwassenen.
Lambert V, Matthews A, MacDonell R, et al. Paediatric early warning systems for detecting and responding to clinical deterioration in children: A systematic review. BMJ Open2017; 7. doi:10.1136/bmjopen-2016-014497•Google Scholar
Chapman SM, Grocott MPW, Franck LS, et al. Systematic review of Paediatric alert criteria for identifying hospitalised children at risk of critical deterioration. Intensive Care Med2010; 36:600–11. doi:10.1007/s00134-009-1715-x•Google Scholar
Trubey R, Huang C, Lugg-Widger FV, et al. Validity and effectiveness of Paediatric early warning systems and track and trigger tools for identifying and reducing clinical deterioration in hospitalised children: a systematic review. BMJ Open2019; 9. doi:10.1136/bmjopen-2018-022105•Google Scholar
Van Sambeeck SJ, Vos G, Theeuwes B, et al. De pediatric early warning score (PEWS) en Veilige (re) Zorg in Nederland. Praktische pediatrie2016;
Prinsen CAC, Vohra S, Rose MR, et al. Core Outcome Measures in Effectiveness Trials (COMET) initiative: protocol for an international Delphi study to achieve consensus on how to select outcome measurement instruments for outcomes included in a ‘core outcome set'. Trials2014; 15. doi:10.1186/1745-6215-15-247•Google Scholar
Bouwens J, Zachariasse J, Snijders C, et al. Dutch PEWS in de Algemene Kindergeneeskundige Praktijk Implementatie en Lokale Toepassing. Praktische pediatrie2021;
Fuijkschot J, Stevens J, Teheux L, et al. Development of the National Dutch PEWS: the challenge against heterogeneity and implementation difficulties of PEWS in the Netherlands. BMC Pediatr2023; 23. doi:10.1186/s12887-023-04219-3•Google Scholar
England NHS, Improvement NHS. Plan do study act (PDSA) cycles and the model for improvement. 2022;
Gorham TJ, Rust S, Rust L, et al. The vitals risk index—retrospective performance analysis of an automated and objective pediatric early warning system. Pediatr Qual Saf2020; 5. doi:10.1097/pq9.0000000000000271•Google Scholar
Parshuram CS, Hutchison J, Middaugh K, et al. Development and initial validation of the bedside Paediatric early warning system score. Crit Care2009; 13. doi:10.1186/cc7998•Google Scholar
VMS. Vroege herkenning en behandeling van de vitaal bedreigde patiënt. 2008;
Hansen G, Hochman J, Garner M, et al. Pediatric early warning score and deteriorating ward patients on high-flow therapy. Pediatr Int2019; 61:278–83. doi:10.1111/ped.13787•Google Scholar
Parshuram CS, Bayliss A, Reimer J, et al. Implementing the bedside Paediatric early warning system in a community hospital: A prospective observational study. Paediatr Child Health2011; 16:e18–22. doi:10.1093/pch/16.3.e18•Google Scholar
de Groot JF, Damen N, de Loos E, et al. Implementing Paediatric early warning scores systems in the Netherlands: future implications. BMC Pediatr2018; 18. doi:10.1186/s12887-018-1099-6•Google Scholar
Orr RA, Felmet KA, Han Y, et al. Pediatric specialized transport teams are associated with improved outcomes. Pediatrics2009; 124:40–8. doi:10.1542/peds.2008-0515•Google Scholar
Petrillo-Albarano T, Stockwell J, Leong T, et al. The use of a modified pediatric early warning score to assess stability of pediatric patients during transport. Pediatr Emerg Care2012; 28:878–82. doi:10.1097/PEC.0b013e31826763a3•Google Scholar
