Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events

INTRODUCTION

Delayed recognition and failure to provide timely intervention to patients developing clinical deterioration events result in suboptimal prognoses.^1,2 Patients often develop physiological instability hours before clinical deterioration,^3–5 and thus many are potentially avoidable.⁶ Early warning score systems have been developed to assist clinicians in recognizing early signs of physiological deterioration, allowing them to provide timely intervention.⁷ Early warning scores (EWS) typically take physiological measurements to calculate risk scores of clinical deteriorations.^8–12 Different levels of action are recommended to corresponding healthcare professionals if a patient exceeds certain thresholds.¹³ These actions typically include increasing monitoring and examinations, initiating treatments, and eventually activating the Rapid Response Team to elevate patient care. A Rapid Response Team (RRT) is a group of clinicians that respond to patients with early signs of deterioration to prevent respiratory and cardiac arrest in the hospital.^14,15 The implementations of EWS and RRT are associated with decreased mortality and in-hospital cardiac arrest.^16–19 In recent reviews of EWS, most data-driven EWS use structured data (eg, vital signs, lab data, demography) from electronic health records (EHR),^20,21 but a few include clinical concern where clinicians answer structured questions if he/she is subjectively concerned about the patient.^22,23 In clinical practice, experienced clinicians recognize patterns of patient deterioration based on expert knowledge and clinical assessment skills and from “knowing the patient” rather than solely on objective physiological measurements.^24–26 These subjective expert judgements, often made before any physiological derangement is detected by machines, are captured in EHR data.^25,27–29

It is widely known that the EHR is not a direct reflection of patient’s pathophysiology but rather a record of the healthcare process with noise.^30,31 Time is an important component that represents the dynamic and positive feedback loop of healthcare processes.^30,31 Additional measurements and clinical records are often made when clinicians are concerned about their patients’ conditions.^25,27 For example, a series of vital sign measures in early morning implies a different healthcare process than a set of those recorded as a routine measurement at scheduled times. Therefore, features reflecting the healthcare process (eg, time and co-occurrence of measurements, order placement, prescriptions, notes) can likewise be utilized for model building. In the National Institute for Nursing Research-funded Communicating Narrative Concerns Entered by RNs (CONCERN) study, our team demonstrates that the inclusion of documentation and measurement frequency in an EWS triggered an alert 5–26 hours earlier than the first Modified Early Warning Score (MEWS) alert.³² In our more recent work, the CONCERNv2.0 model,, including additional nursing note features and temporal features, further increases the “lead time” to 42 hours compared to MEWS and National Early Warning Score.³³ Most EWSs rely solely on physiological measurements, but we believe the use of features that represent clinicians’ concerns could augment early detection.

We propose an algorithm that utilizes only timestamps of longitudinal EHR data (eg, time and co-occurrence of vital sign measurements, flowsheet comments, order entries, and nursing notes) to classify clinical deterioration events. These time-series data reflect nurses’ decision-making related to patient surveillance.^25,28 We emphasize that our data for analysis do not include any measurement values (eg, heart rate = 90mHg). This study aims to 1) validate the proposed models built on sequences of timestamps of underlying clinical data that reflect the healthcare process, and 2) evaluate the impact of including time-of-day and time-to-outcome information in the model.

MATERIALS AND METHODS

This project was approved by both Columbia University Irving Medical Center and Brigham and Women's Hospital Institutional Review Board for the Protection of Human Subjects.

This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration among patients in intensive care units (ICU) using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes in electronic health records. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural network (RNN) algorithms are tested on a training dataset against a composite outcome of death, cardiac arrest, and RRT calls. We choose single layer Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), 2 well-established modalities, as our RNN classifiers.^34,35 Then we validate our algorithms on a holdout dataset with Area Under the Precision-Recall Curve (AUPRC) sensitivity, Positive Predictive Value (PPV), and F-score. The code is accessible on the online repository: https://github.com/lf2608/Timestamps_EHR_Deterioration_Predict.

RESULTS

There are 8367 unique ICU admissions including 200 outcome events in this dataset. These are broken into 76 cardiac arrest and 124 RRT calls. Since these were the first outcome events that the patients had, patients who developed cardiac arrest then died were recorded as cardiac arrest in this dataset. After removing ICU admissions shorter than 36 hours, the final dataset consists of 6720 ICU admissions. (Table 3) There are 161 samples with composite outcomes of death, cardiac arrest, or RRT call which consist of 2.40% of the dataset. A total of 830 578 timestamps are included in the final analysis (Supplementary Material Appendix 3).

The GRU model, utilizing timestamps of vital signs, order entries, flowsheet comments, and nursing notes on the time-to-outcome dataset of timesteps of 60 minutes, has the best predictive power with an AUPRC of 0.101 (0.06, 0.137). This model performs better than the logistic regression with L2 normalization AUPRC 0.093 (0.09, 0.096), but the difference is not statistically significant. (Table 4) The best GRU model has a sensitivity of 0.443 and a PPV of 0.092 at the threshold of 0.6. In general, the RNN models on time-to-outcome datasets have better AUPRC than those on time-of-day datasets. The performance of the RNN models varies on datasets of different length of timesteps with no significant differences.

The values on the scale bar represent the difference between the average count of a given feature from ICU stays with outcomes and without outcomes. These averages are calculated for every feature per hour per patient. The redder the color for a particular feature, the more frequent that feature occurs in an ICU stay with outcome.

DISCUSSION

Our implementation of machine learning algorithms that use only timestamps from EHR metadata results in well-performing discriminative power. We emphasize that this approach doesn’t include the clinical measurement value (eg, heart rate = 90) but instead uses the frequency and co-occurrence of each data entry and documentation pattern that reflect clinicians’ practice patterns and clinical workflow. Additionally, our RNN algorithms display their abilities to learn temporal patterns, which results in improved discriminating power. This result supports our hypothesis that metadata in EHR, a proxy signal of the clinician expert’s objective and subjective assessments, provide valuable information on clinical outcomes, and thus have great potential to inform clinical prediction models.^25,32,33,42

We curate the threshold for our best RNN model to achieve a discriminative power with sensitivity 0.443 and PPV 0.092 using the National Early Warning Score (0.395, 0.152) and Advanced Alert Monitoring (0.489,0.162) as benchmarks.⁴⁰ The former is an expert opinion-based algorithm while the latter is a discrete-time logistic regression model derived from a database of 21 hospitals, and both have been in actual use in clinical settings. However, we are not able to directly compare our model against these benchmarks because those models require measurement values to compute and were validated on distinct study cohorts.

Clinical data is full of missing data and is irregularly measured. These characteristics have been a major challenge in longitudinal data analysis.⁴³ Many EWS studies apply discrete time survival analysis on longitudinal data that completely exclude these characteristics as biases.^{10,40,44–46} Numerous RNN models incorporate information about missing data and irregularity as additional features.^47–50 Our novel approach discretizes data into a multivariate time series representing the frequency and co-occurrence of clinical data entry. Consequently, we are able to preserve the irregularity characteristics as well as the missingness without adding additional artifacts and noise to the dataset. Finally, our results suggest that the dataset of timesteps of 60 minutes has sufficient granularity to represent the patterns of different clinical processes 36 hours before outcomes.

Time is an important component in clinical data because healthcare is not static.⁵¹ Observational measurements are often made when patients are ill. Clinicians make decisions based on their observations and previous understandings of their patients. Documentation of these observations, above and beyond regulatory requirements, are often made when clinicians are concerned about the result of a specific measurement or a patient’s clinical status.^25,27 In the literature, the temporal pattern of lab orders also provides additional information on clinical outcomes regardless of the results.^52,53 By exploiting time, we can capture this nonlinear recording process and positive feedback loop where measurement and health status affect each other.^30,31,51 Our results have shown that time-to-outcome information further boosts RNN models’ AUPRC scores when compared to the baseline logistic regression model. Figure 3 demonstrates the difference of temporal patterns between ICU stays with outcomes and those without. The RNN models are able to learn these temporal patterns of different healthcare processes. We hypothesize that the time-of-day dataset provides additional information, indicative of records that happened in the rare hour of the day, which could further increase the model’s performance. However, our RNN models including the time-of-day information don’t achieve better performance. One explanation is that our method does embed the time-of-day information into the dataset but renders the original sequence of healthcare processes which affects our RNN models’ ability to recognize patients with higher risk.

There are several limitations in this study. Our analysis ran on a relatively small dataset from a healthcare system in Northeast US. This composite dataset includes various types of ICUs where the data represent different clinical processes and documentation patterns within the system. However, potential biases could derive from distinct patient population, healthcare process, regional practice, and, finally, individual clinician’s care pattern. Our results might not be generalizable to other healthcare systems. As a result, we plan to conduct further investigations on a larger dataset to verify the reproducibility of these findings and to externally validate our models. Secondly, our analysis only includes data from 36 to 12 hours before outcomes which limits our models’ ability to predict outcome in clinical practice. A future study will include model derivation on a forward-facing time window so that our models can be validated in a more realistic clinical setting. Moreover, our models don’t include measurement values that contain the information about the pathophysiological status of a patient. Therefore, we can’t directly compare our models with other well-established benchmark EWSs which require such values. A future study will include measurement values in order to validate our model against other benchmarks. Finally, our models are trained on timestamps of the underlying clinical data. There is often a delay when clinical data are entered into EHR which could lead to another bias if our models run on timestamps from EHR. However, our data preprocessing pipeline discretizes data into multivariate time series instead of using timestamps directly, which could potentially mitigate the bias created by the discrepancy between the timestamps of clinical data and EHR data. This discretization method also gives our model the potential to be implemented in real-time settings. Our models can generate predictions at any chosen timestep and requires no data imputation.

CONCLUSION

Clinicians’ recording and documentation patterns result in complex and diverse EHR data collection processes. This characteristic is seen by many as bias and is consequently overlooked in most clinical prediction modeling.²¹ Still, we believe it can also be exploited to yield valuable information for prediction model. This study demonstrates that our RNN models, using only timestamps of longitudinal EHR data that reflect healthcare processes, achieve well-performing discriminative power.

FUNDING

This work was funded by the National Institute for Nursing Research (NINR) funded CONCERN Study #1R01NR016941.

AUTHOR CONTRIBUTIONS

All listed authors substantially contributed to this work, including the study design, the acquisition, analysis, and interpretation of data. All authors participated in either drafting or revising this article and finally approved this version to be published.

DATA AVAILABILITY STATEMENT

The data underlying this article cannot be shared publicly due to inclusion of protected health information. The data will be shared on reasonable request to the corresponding author. All codes of our work are accessible on a public repository.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None decared.

Supplementary Material