Patients in tertiary care hospitals are more complex than in the past, but the implications of this are poorly understood as “patient complexity” has been difficult to quantify. A “Complexity Ruler” was validated and tested in a case-control model of all patients with major adverse events at a tertiary care pediatric hospital from 2005 to 2006. Above empirically derived cutoffs, 24-hour and lifetime cognitive complexity of the medical records were risk factors for major adverse events.
Abstract
Context:
Patients in tertiary care hospitals are more complex than in the past, but the implications of this are poorly understood because “patient complexity” has been difficult to quantify.
Objective:
We developed a tool, the Complexity Ruler, to quantify the amount of data (as bits) in the patient’s medical record. We designated the amount of data in the medical record as the cognitive complexity of the medical record (CCMR). We hypothesized that CCMR is a useful surrogate for true patient complexity and that higher CCMR correlates with risk of major adverse events.
Design:
The Complexity Ruler was validated by comparing the measured CCMR with physician rankings of patient complexity on specific inpatient services. It was tested in a case-control model of all patients with major adverse events at a tertiary care pediatric hospital from 2005 to 2006.
Main Outcome Measures:
The main outcome measure was an externally reported major adverse event. We measured CCMR for 24 hours before the event, and we estimated lifetime CCMR.
Results:
Above empirically derived cutoffs, 24-hour and lifetime CCMR were risk factors for major adverse events (odds ratios, 5.3 and 6.5, respectively). In a multivariate analysis, CCMR alone was essentially as predictive of risk as a model that started with 30-plus clinical factors.
Conclusions:
CCMR correlates with physician assessment of complexity and risk of adverse events. We hypothesize that increased CCMR increases the risk of physician cognitive overload. An automated version of the Complexity Ruler could allow identification of at-risk patients in real time.
Introduction
Patients in tertiary care hospitals are more complex than in the past, and complexity varies greatly between patients. Because there are finite limits on the brain’s ability to assimilate and process information, greater patient complexity increases the risk of physician cognitive overload and medical error.1–16 Therefore, if one could quantify patient complexity, it is possible that this would provide a useful risk stratification tool.
Cognitive overload is a risk factor for cognitive failure in many domains, and techniques have been created in other fields to quantify complexity.17 Although there is no perfect method of quantifying complexity, one widely used operational definition is “the minimum amount of information required to fully describe something.” Conceptually, one writes down the most parsimonious but complete description of something (eg, a car, a patient) and counts the number of bits required to store that description.18,19
It is obviously impossible to write a perfect description of a patient and his/her illness or illnesses. We therefore hypothesized that the amount of data recorded in the medical record, which we designated the “cognitive complexity of the medical record” (CCMR), would be a useful surrogate for true patient complexity. Our testable hypothesis was that patients with greater CCMR are at higher risk of adverse events.
The algorithm for calculating CCMR is complex, but conceptually it is fairly simple. We count 1 bit for every character of English language text, and 3 bits for every individual numeric digit recorded in the record. (Although it takes 8 bits to store an English letter in ASCII code, English-language text is highly constrained, and thus 1 bit per character is a reasonable estimate of text complexity.18) For images, which require artificially inflated storage space (every chest film in our institution is stored in a 5-megabyte file), we count the number of characters in the dictated report. For headings on standardized charts (eg, “blood pressure”) we count the heading once per 24 hours.
To the best of our knowledge, no prior studies have attempted to quantify the data stored in the medical record, and there is thus no gold standard for CCMR. We therefore tested our tool by 1) comparing its ranking of CCMR to the complexity rankings of experienced clinicians and 2) testing its ability to predict major adverse events in a tertiary care pediatric hospital.
Methods
Development of the Complexity Ruler
As discussed in the introduction, we used an estimate of one bit per character for English-language text and 3 bits per digit for numeric data. We counted redundant headings once per 24 hours, and counted the dictated reports rather than the data storage requirements of actual images. We included all data entered into the record, whether by nurse, physician, or other clinician, and included laboratory results, x-ray films, and any other information of any kind. The result is a single number to represent the CCMR for each patient during a given time period. The instructions for the full Complexity Ruler are available online at: www.thepermanentejournal.org/files/Winter2014/cr.pdf.20
Face Validity
Three senior physicians rank ordered the complexity (not the acuity) of patients on four different inpatient medical services. We measured the CCMR of five randomly selected inpatients from each of these four services and estimated their lifetime CCMR. We hypothesized that agreement between the Complexity Ruler and expert clinicians would constitute face validity of the Complexity Ruler.
Cognitive Complexity of the Medical Record versus Major Adverse Events
We conducted a case-control evaluation of the 39 patients who had major adverse events at Boston Children’s Hospital, Boston, MA, in 2005 and 2006. A major adverse event was de-fined as an event of sufficient gravity to require reporting to an outside regulatory agency. Most of these events involved permanent injury to the patient or major complications, for example, unanticipated abdominal or cardiac surgery. Controls were 78 patients who were randomly selected from all admissions to Boston Children’s Hospital during 2005 to 2006.
24-Hour versus Lifetime Cognitive Complexity of the Medical Record
The 24-hour CCMR was measured in the 24 hours before the adverse event in the cases (eg, from 3 am the day before to 2:59 am if the event occurred at 3 am). Each control was randomly assigned a date and time of his/her “event,” and 24-hour CCMR was calculated for the 24 hours before the event. Cases and controls whose event occurred in the first 24 hours after admission were excluded from the 24-hour CCMR analysis.
The current Complexity Ruler is too labor-intense to apply to long periods of time. We therefore estimated the lifetime CCMR by determining their lifetime number of inpatient hospital days and multiplying it by the average CCMR of an inpatient day, stratified by days in the intensive care unit (ICU) vs non-ICU days. All cases and controls were included in the analysis of lifetime CCMR.
Data Analysis
Mean CCMR was compared for cases (reportable event) and controls (no reportable event) using the 2-sample t test with unequal variances. Patient characteristics, admission characteristics, and nursing factors were compared for cases vs controls using the Wilcoxon rank sum test for continuous variables and Fisher exact test for categorical variables. Factors significant at the 0.10 level were included in logistic regression models predicting major adverse events; the odds ratio (OR) and 95% confidence intervals were estimated. The CCMR was dichotomized at a value that maximized the OR for predicting major adverse events.
The c statistic was calculated for each logistic regression model. The c statistic is a measure of the model’s ability to discriminate between patients who experienced a major adverse event and those who did not; a value of 0.5 is no better than random, whereas a value of 1.0 means the model predicts outcome perfectly. Multivariate analysis was performed using forward selection; a p value ≤ 0.05 was required for a variable to be retained in the final model. Statistical analyses were performed using SAS Version 9.2 (SAS Institute Inc, Cary, NC).
Results
Face Validity
All 3 senior physicians ranked patients on the short stay service as least complex and patients in the ICU as most complex. Two of the 3 physicians ranked cardiology patients as more complex than general pediatrics patients, and 1 reversed this order. Both 24-hour and lifetime CCMR agreed with the rankings of experienced clinicians (Figures 1 and 2).
Figure 1.
Mean 24-hour cognitive complexity of the medical record (CCMR) measured by the Complexity Ruler compared with physician rankings of complexity.
CCMR was measured from an average of 5 randomly selected patients on each of 4 inpatient medical services.
MICU = Medical Intensive Care Unit.
Figure 2.
Mean 24-hour cognitive complexity of the medical record (CCMR) and estimated CCMR for the preceding 3 years.
Data are for the same patients shown in Figure 1. The average intensive care unit patient had more than 3.5 million bits over the previous 3 years—a staggering amount of information for the human brain to assimilate.
MICU = Medical Intensive Care Unit.
Lifetime Cognitive Complexity of the Medical Record versus Major Adverse Events
All 39 cases and 78 controls were included in this analysis. The mean estimated lifetime CCMR was much higher for cases (736,033 bits) than controls (119,707 bits), (p < 0.001). Having more than 70,000 bits in the lifetime medical record was associated with an OR of 6.5 for major adverse events (p < 0.001). Compared with more than 30 other risk factors, lifetime CCMR was 1 of 16 statistically significant risk factors for major adverse events and had the third highest OR of these factors (Table 1).
Table 1.
Univariate analysis: All cases and controlsa
| Variable | Odds ratio | p value |
|---|---|---|
| ≥ 70,000 Bits of lifetime CCMR | 6.5 | < 0.001 |
| ≥1 G-tube medication | 12.7 | 0.007 |
| English not first language for parents | 6.9 | 0.02 |
| ≥ 1 Nebulized medication | 5.6 | 0.02 |
| ≥ 1 Assistive device | 4.5 | < 0.001 |
| ≥ 2 Oral medications | 4.4 | 0.009 |
| ≥ 4 Medications | 4.2 | 0.002 |
| Critically ill just prior | 4.0 | 0.03 |
| Intubated | 4.0 | 0.001 |
| Day of week, Monday-Friday | 3.8 | 0.01 |
| ≥ 2 IV medications | 3.4 | 0.02 |
| Any surgical or invasive treatment | 3.2 | 0.01 |
| Elective admission | 3.1 | 0.007 |
| ICU stay | 2.4 | 0.09 |
| ≥ 2 Ongoing medical conditions | 2.1 | 0.06 |
| Any ongoing medical condition | 1.6 | 0.3 |
For this dataset, 24-hour CCMR information is not available. The following 24 risk factors did not have statistically significant results: admission type, median age, patient type, month of the year, primary service, median length of admission, median number of ongoing conditions, any serious ongoing medical condition, purpose of admission, condition at admission, any diagnostic evaluation, any medical treatment, any nonsurgical invasive treatment, any surgical treatment, time of event, level of assistance with daily living, level of nursing monitoring required, communicative ability of patient, continuous oxygen, noninvasive continuous positive airway pressure/biphasic positive airway pressure, treated with nebulizers, frequent respiratory treatment, median number of dressing changes, parent/caregiver present.
CCMR = cognitive complexity of the medical record; G-tube = gastrostomy tube; ICU = intensive care unit; IV = intravenous.
In a multivariate analysis, lifetime CCMR was an independent predictor of adverse events. Patients with a lifetime CCMR greater than 70,000 bits had an OR of 7.7 for adverse events (p < 0.001; Table 2). We compared the discrimination of the univariate model used to evaluate lifetime CCMR (c statistic of 0.72) with the goodness of fit from a multivariate analysis that excluded CCMR (c statistic of 0.78).
Table 2.
Multivariate analysis: All cases and controlsa
| Clinical factor | Odds ratio | 95% Confidence interval | p value | c statistic |
|---|---|---|---|---|
| Most predictive multivariate model—all factors | 0.82 | |||
| > 70,000 bits of lifetime CCMR | 7.7 | (3.0–19.6) | < 0.001 | |
| Day of week, Monday–Friday | 5.1 | (1.6–17.0) | 0.007 | |
| Admission type: elective | 3.2 | (1.2–8.1) | 0.02 | |
| Univariate analysis of lifetime CCMR | 0.72 | |||
| > 70,000 bits of lifetime CCMR | 6.5 | (2.8–15.3) | < 0.001 | |
| Multivariate analysis with CCMR excluded | 0.78 | |||
| > 1 Assistive device | 3.2 | (1.2–8.5) | 0.02 | |
| > 4 Medications | 3.6 | (1.4–9.5) | 0.01 | |
| Day of week, Monday–Friday | 4.4 | (1.3–14.6) | 0.02 | |
24-hour CCMR is not available for this dataset.
CCMR = cognitive complexity of the medical record.
24-Hour and Lifetime Cognitive Complexity of the Medical Record versus Major Adverse Events
We hypothesized that 24-hour CCMR and lifetime CCMR might have different predictive values for major adverse events. However, for the analysis of 24-hour CCMR, we could evaluate only those patients whose event happened at least 24 hours after admission and, thus, for whom 24-hour CCMR could be measured (17 cases and 42 controls). We therefore analyzed this group of patients including both 24-hour CCMR and lifetime CCMR as potentially independent predictive variables. Because the datasets are not identical, the lifetime CCMR results are slightly different than in the analysis of all patients (Table 1).
Mean 24-hour CCMR was higher for cases (31,323 bits) than controls (14,454 bits), a significant difference (p = 0.008). Inclusion of more than 15,000 bits in 24 hours was associated with an OR of 5.3 for a major adverse event (p = 0.008). Compared with more than 30 other risk factors for adverse events, both 24-hour and lifetime CCMR were among the 10 statistically significant risk factors, and lifetime CCMR had the second highest OR of all factors (Table 3).
Table 3.
Univariate analysis: 17 cases and 42 controlsa
| Clinical factor | Odds ratio | p value |
|---|---|---|
| ≥ 70,000 Bits of lifetime CCMR | 16.7 | < 0.001 |
| ≥ 1 G-tube medication | 19.2 | 0.02 |
| ≥ 1 Nebulized medication | 15.5 | 0.007 |
| ≥ 4 Medications | 11.0 | 0.001 |
| ≥ 1 Assistive device | 7.3 | 0.003 |
| ≥ 2 Oral medications | 6.6 | 0.03 |
| Intubated | 4.8 | 0.01 |
| ≥ 2 IV medications | 4.3 | 0.03 |
| Elective admission | 3.6 | 0.04 |
Cases and controls included only those for whom 24-hour CCMR data were available. The following 28 risk factors did not have statistically significant results: median age at admission, patient type, day of the week, month of the year, primary service, location of service, median length of admission, any ongoing medical conditions, median number of ongoing conditions, any serious ongoing medical condition, purpose of admission, condition at admission, any diagnostic evaluation, any medical treatment, any nonsurgical invasive treatment, any surgical treatment, English not first language of parent, time of event, condition before event, level of assistance with daily living, level of nursing monitoring required, communicative ability of patient, treated with continuous oxygen, treated with noninvasive continuous positive airway pressure/biphasic positive airway pressure, treated with nebulizers, frequent respiratory treatment, median number of dressing changes, and parent/caregiver present.
CCMR = cognitive complexity of the medical record; G-tube = gastrostomy tube; IV = intravenous.
In a multivariate analysis after controlling for other factors, 24-hour and lifetime CCMR were not statistically significant. The c statistic for goodness of fit for the most predictive multivariate analysis was 0.87. When 24-hour CCMR was forced into the model, greater than 15,000 bits in 24 hours had an OR of 9.7 for a major adverse event (p = 0.02, c statistic = 0.86). When lifetime CCMR was forced into the model, greater than 70,000 bits had an OR of 19.3 for a major adverse event (p = 0.02, c statistic = 0.88; Table 4).
Table 4.
Models of multivariable analysis: Limited dataseta
| Models | Odds ratio | 95% Confidence interval | p value | c statistic |
|---|---|---|---|---|
| Model 1: Most predictive model | 0.87 | |||
| > 4 Medications | 21.6 | (1.8–265.8) | 0.02 | |
| > 1 Nebulized medication | 15.2 | (1.2–200.3) | 0.04 | |
| Model 2: Most predictive model that forces in 24-hour CCMR | 0.86 | |||
| > 15,000 Bits of 24-hour CCMR | 9.7 | (1.4–69.9) | 0.02 | |
| > 1 Nebulized medication | 9.8 | (1.0–93.7) | 0.05 | |
| Model 3: Most predictive model that forces in lifetime CCMR | 0.88 | |||
| > 70,000 Bits of lifetime CCMR | 19.3 | (1.5–240.5) | 0.02 | |
| > 1 Nebulized medication | 16.3 | (1.3–207.3) | 0.03 | |
The limited dataset includes only the 17 cases and 42 controls for which we have 24-hour CCMR.
CCMR = cognitive complexity of the medical record.
Discussion
Many authors have commented on the phenomenon of increasing patient complexity and hypothesized that complexity is a risk factor for errors and major adverse events.1–16 However, the impact of complexity on safety and quality has been difficult to study because it has not been quantifiable. We hypothesized that the data stored in the medical record (what we have designated the CCMR) would be a useful surrogate for true patient complexity and would therefore predict the risk of major adverse events.
Cognitive complexity is not the same as medical acuity. Although very sick patients will often be very complex, there are patients with cognitively straightforward problems (eg, carbon monoxide poisoning) who are critically ill, and patients with very complex illnesses (long-standing diabetes) who are—at the moment—physically well.
Robust Prediction of Risk
Using a number of assumptions, we developed a tool to measure CCMR.18,19 We demonstrate in this project that both 24-hour and lifetime CCMR are risk factors for major adverse events. Naturally, our tool is preliminary and is subject to revision and improvement. However, even in its first iteration, it was very robust at predicting the risk of major adverse events. Most importantly, use of either 24-hour or lifetime CCMR alone was essentially equal in predictive power as multivariate models built on more than 30 clinical factors. Also important, use of 24-hour or lifetime CCMR alone was nearly as useful in predicting the risk of major adverse events as was a multivariate analysis starting with more than 35 clinical variables.
To implement risk reduction strategies, high-risk patients must be identified in real time, and risk stratification systems that incorporate multiple clinical variables are inherently difficult to implement in real time. On the other hand, CCMR could be readily “built into” the electronic medical record and be available continuously in real time.
If patients with high CCMR were identified in real time, potential risk-reduction strategies could be tested. Since cognitive capacity increases with experience,21 one hypothesis is that very complex patients should be managed by physicians with greater seniority. More complex patients might be provided with additional physician or nurse staffing. Determining which risk reduction strategies might be effective is far beyond the scope of this study, but it is impossible to test strategies scientifically unless one can identify the patients at high risk.
Limitations
The Complexity Ruler and this project both have many important limitations. Since there is no gold standard for cognitive complexity, we had to construct the Complexity Ruler de novo using multiple assumptions. Although the first version of the Complexity Ruler is predictive of adverse events, we are certain that it can be modified and improved. For example, different weightings of English-language text and digits might more accurately predict the likelihood of a major adverse event.
The Complexity Ruler measures the total CCMR of the chart, including physician notes, laboratory tests, radiographs, nursing notes, flowsheets, and all other items. We did measure the different sections of the medical record separately but did not find any increased predictive value from looking at these as separate variables (data not shown). This could simply reflect the small size of our sample. Future studies will examine whether there are specific sections of the record whose CCMR is more predictive of major adverse events.
Because the Complexity Ruler measures the amount that clinicians write in the chart, it is possible that it overestimates the CCMR of patients whose clinicians are more verbose. This is a particular concern because clinicians can “cut and paste” sections of previous notes into current notes in the electronic medical record, increasing the amount of text without adding new information. There are computational tools to reduce or eliminate the redundancy from text (“compression algorithms”); therefore, it will be possible in future studies to determine whether verbosity and repetition are significant issues when measuring CCMR.
These data demonstrate that high CCMR is a risk factor for major adverse events but do not establish a causative link. We believe that the mechanism is cognitive overload of the physician, but additional studies will be needed to test that hypothesis.
Because we studied only very serious adverse events, we do not know if CCMR as measured by our tool is a risk factor for less serious adverse events.
… use of either 24-hour or lifetime CCMR alone was essentially equal in predictive power as multivariate models built on more than 30 clinical factors.
Finally, our datasets were relatively small, and thus it was not possible to determine whether 24-hour CCMR and lifetime CCMR are independent risk factors for major adverse events.
Conclusion
A new instrument, the Complexity Ruler, was designed to measure the cognitive complexity of the medical record. In this study, the measured CCMR correlated with complexity as assessed by experienced clinicians, and it was a risk factor for major adverse events. Although unproven, it may be that the mechanism linking CCMR to major adverse events is cognitive overload.
An automated version of the Complexity Ruler is being developed. Automated, real-time information about complexity may have value in identifying patients at high risk of adverse events.
Acknowledgments
The authors gratefully acknowledge the support of numerous clinicians at Boston Children’s Hospital, who helped us understand the many sources of information accumulating in the medical record.
Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.
Footnotes
Disclosure Statement
Dr Connell is Chief Executive Officer of Knowledge Design Inc; Boston, MA. Boston Children’s Hospital, (Boston, MA) has applied for a patent for the method of determining patient complexity reported in this article. Draper Laboratory (Cambridge, MA) has sponsored this study with an unrestricted research grant. The authors have no other competing interests to disclose.
Both studies were approved by the Boston Children’s Hospital institutional review board.
Besieged City
The physician should look upon the patient as a besieged city and try to rescue him with every means that art and science place at his command.
—Alexander of Tralles, c525–c605, one of the most eminent of ancient physicians
References
- 1.Burns KH, Casey PH, Lyle RE, Bird TM, Fussell JJ, Robbins JM. Increasing prevalence of medically complex children in US hospitals. Pediatrics. 2010 Oct;126(4):638–46. doi: 10.1542/peds.2009-1658. DOI: http://dx.doi.org/10.1542/peds.2009-1658. [DOI] [PubMed] [Google Scholar]
- 2.Simon TD, Berry J, Feudtner C, et al. Children with complex chronic conditions in inpatient hospital settings in the United States. Pediatrics. 2010 Oct;126(4):647–55. doi: 10.1542/peds.2009-3266. DOI: http://dx.doi.org/10.1542/peds.2009-3266. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Cullen DJ, Sweitzer BJ, Bates DW, Burdick E, Edmondson A, Leape LL. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med. 1997 Aug;25(8):1289–97. doi: 10.1097/00003246-199708000-00014. DOI: http://dx.doi.org/10.1097/00003246-199708000-00014. [DOI] [PubMed] [Google Scholar]
- 4.Rothschild JM, Landrigan CP, Cronin JW, et al. The Critical Care Safety Study: the incidence and nature of adverse events and serious medical errors in intensive care. Crit Care Med. 2005 Aug;33(8):1694–700. doi: 10.1097/01.ccm.0000171609.91035.bd. DOI: http://dx.doi.org/10.1097/01.CCM.0000171609.91035.BD. [DOI] [PubMed] [Google Scholar]
- 5.Pronovost P, Wu AW, Dorman T, Morlock L. Building safety into ICU care. J Crit Care. 2002 Jun;17(2):78–85. doi: 10.1053/jcrc.2002.34363. DOI: http://dx.doi.org/10.1053/jcrc.2002.34363. [DOI] [PubMed] [Google Scholar]
- 6.Satish U, Streufert S. Value of a cognitive simulation in medicine: towards optimizing decision making performance of healthcare personnel. Qual Saf Health Care. 2002 Jun;11(2):163–7. doi: 10.1136/qhc.11.2.163. DOI: http://dx.doi.org/10.1136/qhc.11.2.163. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Chervany NL, Dickson GW. An experimental evaluation of information overload in a production environment. Manage Sci. 1974 Jun;20(10):1335–44. DOI: http://dx.doi.org/10.1287/mnsc.20.10.1335. [Google Scholar]
- 8.Edmunds A, Morris A. The problem of information overload in business organizations: a review of the literature. Int J Inf Manage. 2000 Feb;20(1):17–28. [Google Scholar]
- 9.Hunt RE, Newman RG. Medical knowledge overload: a disturbing trend for physicians. Health Care Manage Rev. 1997 Winter;22(1):70–5. DOI: http://dx.doi.org/10.1097/00004010-199701000-00009. [PubMed] [Google Scholar]
- 10.Payne JW. Task complexity and contingent processing in decision making: an information search and protocol analysis. Organ Behav Hum Perform. 1976 Aug;16(2):366–87. DOI: http://dx.doi.org/10.1016/0030-5073(76)90022-2. [Google Scholar]
- 11.Schneider SC. Information overload: causes and consequences. Human Systems Management. 1987;7(2):143–53. DOI: http://dx.doi.org/10.3233/HSM-1987-7207. [Google Scholar]
- 12.Speier C, Valacich JS, Vessey I. The influence of task interruption on individual decision making: an information overload perspective. Decision Sciences. 1999 Mar;30(2):337–60. DOI: http://dx.doi.org/10.1111/j.1540-5915.1999.tb01613.x. [Google Scholar]
- 13.Treisman AM, Davies A. Divided attention to ear and eye. In: Kornblum S, editor. Attention and performance IV. London, England: Academic Press; 1973. Jan 1, pp. 101–17. [Google Scholar]
- 14.Baddeley A. Working memory. Science. 1992 Jan 31;255(5044):556–9. doi: 10.1126/science.1736359. DOI: http://dx.doi.org/10.1126/science.1736359. [DOI] [PubMed] [Google Scholar]
- 15.Ericsson KA, Kintsch W. Long-term working memory. Psychol Rev. 1995 Apr;102(2):211–45. doi: 10.1037/0033-295x.102.2.211. DOI: http://dx.doi.org/10.1037/0033-295X.102.2.211. [DOI] [PubMed] [Google Scholar]
- 16.Miller GA. The magical number seven, plus or minus two: some limits on our capacity for processing information. 1956. Psychol Rev. 1994 Apr;101(2):343–52. doi: 10.1037/0033-295x.101.2.343. DOI: http://dx.doi.org/10.1037/0033-295X.101.2.343. [DOI] [PubMed] [Google Scholar]
- 17.Thomas JC, Richard JT. Achieving psychological simplicity: measures and methods to reduce cognitive complexity. In: Sears A, Jacko JA, editors. The human computer interaction handbook: fundamentals, evolving technology and applications. 2nd ed. New York, NY: Taylor & Francis Group, LLC; 2008. pp. 489–508. [Google Scholar]
- 18.Bar-Yam Y. Dynamics of complex systems. Reading, MA: Addison-Wesley; 1997. [Google Scholar]
- 19.Shannon CE. A mathematical theory of communication. The Bell System Technical Journal. 1948 Jul;27(3):623–56. [Google Scholar]
- 21.Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med. 2008 Nov;15(11):988–94. doi: 10.1111/j.1553-2712.2008.00227.x. DOI: http://dx.doi.org/10.1111/j.1553-2712.2008.00227.x. [DOI] [PubMed] [Google Scholar]