Confirmatory Factor Analysis of the Delirium Rating Scale Revised-98 (DRS-R98)

No test of a psychological or psychiatric construct is ever completely established in that the psychometric properties of any instrument need to be monitored for its performance to ascertain if the mathematical aspects of the tool continue to attain appropriate standards. Those properties may change as populations where the instrument is being applied evolve over time. Further, the technical aspects of tests may not generalize across cultures or racial and socioeconomic levels. Nevertheless, in the measurement of the construct of the disorder delirium, one tool has seemingly emerged as ascendant among several competing instruments¹ that purport to be adequate, appropriate assessments: the Delirium Rating Scale-Revised-98 (DRS-R98).²

The DRS-R98 measures pathology by clinicians who will make decisions that potentially affect the lives of humans in significant ways. Consequently, much of the psychometric research on the scale has dealt with investigations involving sources of measurement error (interrater reliability, internal consistency reliability) and methods for establishing optimal cutoff scores in relation to sensitivity and specificity considerations. Validity studies to date indicate elevated concurrent and discriminant coefficients. The results across nearly 15 years of investigation have yielded very impressive data indicating homogeneous (i.e., intercorrelated) items that reflect congruence among independent raters and results that can be used with confidence in clinical decision-making.^2–8

Construct Validity

Whereas most of the prior psychometric investigations of the DRS-R98 have focused on information important in practice, the more theoretical and scientific import of the construct of delirium has not been neglected in the research. A construct is a conceptual term to describe a phenomenon of theoretical interest in a scientific discipline.⁹ It is literally an abstract word that is denotatively inexact. One cannot point to an object or a specific behavior of person and say this is what the word represents. A construct is abstract rather than concrete. It is indeed a construction regarding what the word conceptually subsumes in the world of reality; it often involves a theoretical construction regarding how more concrete observed variables interrelate.¹⁰

In brief, a clinician cannot take the word “delirium” and use it in the same symbolic manner as for instance, the word “table.” The latter is an explicit verbal symbol that denotes a concrete object in the environment. The verbal symbol “delirium,” on the other hand, denotes a variety of mental and behavioral events that in theory converge in the same patient. What those specific mental and behavioral phenomena may also be a source of theoretical debate.

The DRS-R98 purports to measure something that is abstract, a construct. The 16 items come from a content domain formulated as the result of theory and clinical experiences regarding how certain acute medical conditions affect human consciousness, brain functions, and resultant specific cognitive abilities and behaviors.

Factor Analysis

Factor analysis is a statistical tool designed, in part, to investigate the structure of a measuring instrument to evaluate if the items correspond to complex, theoretical notions about the nature of the construct in the real world. It is the statistical method applied for the explication of constructs, to render them more denotatively exact, and to provide an operational definition acceptable to other scientists. With respect to the DRS-R98, the goal is to ascertain mathematically if test items reflect the theoretical nature of delirium with respect to consciousness, neurological functioning, and deportment.

Exploratory factor analysis has been used to explore the internal structure of the DRS-R98 in consecutive delirious and nondelirious general hospital inpatients from Colombia.¹¹ The findings supported a two-factor structure with theoretical import. Factor (F)1 reflected deficits in or diminished capacities in normal cognitive functioning (e.g., memory, language, attentiveness, and information processing). F2 included items reflecting distortions in normal perceptual and emotional functioning (delusions and mood fluctuations). In addition, F1 included reduced motor activity, whereas F2 was associated with motor agitation. This study suggested that there may be two presentations of delirium: one with cognitive deficits and reduced activity levels and the other with psychotic symptoms and excessive motor agitation. That two-factor structure has been replicated by another exploratory factor analysis of the DRS-R98 in patients with delirium referred to a consultation-liaison service from India, where items evaluating higher-order thinking loaded together with cognitive items onto F1 and motor items loaded together with sleep-wake cycle, psychotic symptoms, and emotional functioning onto F2.¹² By contrast, two other exploratory factor analyses of consultation-liaison referrals with delirium from India found three factors but not two, labeled cognitive, sleep/motor symptoms, and thought/language/fluctuation factors in one¹³ and cognitive, circadian/psychosis, and higher-order thinking factors in the other.¹⁴ Sometimes cognition and higher level thinking loaded together (when two factors) and sometimes separately (with three-factor structure). Circadian factors, sleep-wake cycle, and motor items often loaded together and sometimes with psychosis. Those exploratory factor analyses informed not only the scale structure but also the nature of delirium itself.

Confirmatory factor analysis (CFA) is an approach that uses structural equations to verify a factor structure of a test. It involves more exacting hypothesis testing than used in exploratory techniques. With confirmatory analyses, a fit between empirical data (test scores) and a structural model is rigorously tested.

The aim of the current investigation was to evaluate whether a two-factor structure can be verified with the DRS-R98 by using a large multicultural multisite source of data (i.e., to verify whether homogeneity/intercorrelations among DRS-R98 items represented a single factor or two correlated-oblique factors).

Methods

Data Set

De-identified cross-sectional data for adult patients from 14 studies (11 of them published) conducted by expert delirium researchers in seven countries (United States, Brazil, Colombia, Ireland, Taiwan, Korea, and Japan) from four continents comprised the pooled database.^{2–8,15–18} Investigators were invited by Dr. Trzepacz, the developer of the DRS-R98, to participate in this study because they had previously received permission to use the scale for work related to its validation. Their work, published or not yet published, needed to meet the following criteria: (1) all site investigators were delirium experts and well trained in the use of the DRS-R98; (2) delirium and other psychiatric diagnoses in the original projects were made according to DSM-IV/DSM-IV-TR using all sources of clinical information; (3) DRS-R98 ratings were independent (blinded) from DSM-IV diagnosis and covered the previous 24-hour period; and (4) all subjects (patients with delirium and control subjects) were evaluated cross-sectionally using DSM-IV criteria and the DRS-R98.

All studies were approved by the appropriate human ethics committees at participating sites and informed or proxy consent was obtained as required.

Participants

The participants (N=859) were adult patients from a variety of inpatient and outpatient clinical settings corresponding to different centers that originally did the assessment as part of the separate studies described previously, where 516 had delirium and 343 were nondelirious control subjects according to DSM-IV criteria. Demographic data and comorbid psychiatric diagnoses according to DSM-IV were collected, and demented cases were excluded from the study to avoid contamination from overlapping symptoms. Active medical-surgical diagnoses from the original datasets were categorized according to the Delirium Etiology Checklist that allows classification of diverse medical-surgical diagnoses into 13 categories.¹⁹

Measure

The DRS-R98 is a widely used, well-validated, specific, and sensitive rating scale with phenomenological descriptive anchors for rating the severity levels for each item (0 is normal to a maximum of 3) with a maximum scale score of 46 points. It was designed to evaluate the breadth and severity of delirium symptoms for phenomenological and treatment studies, in addition to diagnosing delirium. The DRS-R98 is not derived from any particular diagnostic system and instead was developed based on known delirium characteristics. Its 16 items include three diagnostic items for the total score, where 13 items constitute the severity scale score. It was originally validated using raters blind to diagnoses and results compared against four other diagnostic groups of inpatients. It has been subsequently translated and revalidated in countries outside of the United States. Translations were used, as appropriate, and raters had been trained as part of individual projects.^3–8 The DRS-R98 versions used had very high interrater reliability (intraclass correlation coefficient>0.9), and excellent validity as shown by the area under the curve>0.9 (receiver-operating characteristic analyses) when comparing DSM delirium with nondelirium patients for all DRS-R98 versions used in this study. The DRS-R98 broadly measures delirium phenomenology using anchored item descriptors and has been validated against other neuropsychiatric disorders, making it an ideal instrument to assess phenomenology.

The optimum cutoff scores between delirium and nondelirium based on receiver-operating characteristic analysis were reported across countries.^2–7 All studies have shown clinically useful high sensitivity and specificity at each optimum cutoff score: the original English version (cutoff score=17.8, sensitivity 92%, specificity 95%),² the Spanish version (cutoff score=14.0, sensitivity 82%, specificity 98%),³ the Portuguese version (cutoff score=20.1, sensitivity 93%, specificity 95%),⁴ the Chinese version (cutoff score=15.5, sensitivity 89%, specificity 97%),⁵ the Japanese version (cutoff score=14.5, sensitivity 98%, specificity 94%),⁶ and the Korean version (cutoff score=18.5, sensitivity 94%, specificity 85%).⁷ The DRS-R98 scales could also distinguish delirium from dementia: the original English version (using either a cutoff score=22.5, sensitivity 91%, specificity 100% or 17.8 with sensitivity of 100%, specificity 85%),² the Portuguese version (cutoff score=22.5, sensitivity 89%, specificity 100%),⁴ the Japanese version (with the same cutoff=14.5 and the same sensitivity of 98% as for distinguishing delirium from nondelirium, but with a lower specificity of 75%),⁶ and the Korean version (cutoff score=20.5, sensitivity 86%, specificity 72%).⁷ Several clinical epidemiologists have recently criticized the use of the fixed cutoff thresholds and have advocated multilevel or stratum-specific likelihood ratios.²⁰ One study showed that less than 5% of study populations had noninformative stratum-specific likelihood ratios, suggesting that the DRS-R98 scale is a very useful clinical diagnostic tool.⁶

Statistical Procedure

Age and DRS-R98 score differences among delirium and nondelirium groups were compared using the t test. Sex and other diagnosis differences between groups were compared with the chi-square test.

The correlation matrix of the 16 DRS-R98 items was initially scrutinized for factor analytic suitability using the Kaiser-Meyer-Olkin measure of sampling adequacy; item distributions and assessment of multivariate normality were then conducted. Next, the data were explored, and both orthogonal and oblique rotations were computed. This was followed by CFA.

Analyses for violations of multivariate normality included principal axis extraction, scale-free estimation in CFA, and the bootstrapping technique afforded by AMOS 16.0.²¹ The latter allows for the assessment of stability and accuracy of parameter estimates via equating the non-normal empirical distribution to the theoretical distributional assumptions of the statistical analysis.²²

In CFA, the standardized root-mean-residual and Tucker-Lewis index were selected as measures of model-data fit, based on the work of Hu and Bentler.²³ Many of the traditional and often-used indices of fit (e.g., goodness of git index; normed fit Index) have been discounted because, for example, they have not been sensitive to model misspecification or because they behave erratically with different sample sizes or methods of estimation. Hu and Bentler²³ suggested that a good fit index should have a large model misspecification effect, together with trivial effects of sample size, distribution, and estimation method. The standardized root-mean-residual tends to meet such standards. In their comprehensive investigation of various fit measures, Hu and Bentler²³ reported strong support for use of standardized root-mean-residual as a generally unbiased measure. Further, they recommend the use of standardized root-mean-residual with on other fit measure to evaluate the adequacy of a structural equation model. One ancillary recommended measure is the Tucker-Lewis index. The standardized root-mean-residual relates to the extent to which the sample variance and covariance differ from their estimates, whereas the Tucker-Lewis index compares the fit of the model being tested to a baseline or independence model; it also penalizes the model being tested in relation to excessive number of parameters.²¹ Values of standardized root-mean-residual>0.08 correspond to a well-fitted model; values of Tucker-Lewis index beyond 0.90 suggest an acceptable fit.²⁴ Finally, the chi-square difference test was used to test whether the two-factor model is better to represent the data than the one-factor model.²⁵

Results

Characteristics of this sample of 859 patients are described in Table 1. As expected, the delirium group was older than the nondelirium group, and there was a trend for active medical-surgical diagnosis being more frequent (four of seven Delirium Etiology Checklist categories) in the delirium group, with none of the seven more frequent in the nondelirium subjects. There was a significantly higher proportion of men in the delirium cases compared with the nondelirium patients. The most frequent psychiatric comorbidity was depressive disorder (63 or 12.2% for the delirium group and 45 or 13.1% for the nondelirium group, χ² =0.155; df=1; p=0.694). Most cases came from Korea (334, 38.9%) and Ireland (161, 18.7%), followed by Colombia (113, 13.2%), the United States (91, 10.6%), Japan (64, 7.4%), Brazil, and Taiwan (48, 5.5% each one).

The Kaiser-Mayer-Olkin measure was 0.947, a “marvelous” value (according to Kaiser), indicating that the distribution of values is adequate, even meritorious, for factor analysis. The Mardia coefficient of multivariate kurtosis was elevated (45.35, z = 27.69, p<0.01), resulting in the use of methods for dealing with multivariate non-normal data.

Findings from scale-free estimation for unifactor and two-factor models are presented in Table 2. Both models met standards for model-data congruence, with a chi-square difference test supporting the superiority of an oblique two-factor model. Items and factor loadings of the DRS-R98 are presented in Table 3.

Discussion

To continue to evaluate its scale characteristics, we performed CFA of the DRS-R98 using a large international pooled data set of prospectively collected ratings on 516 patients with delirium and 343 patients without delirium and subjected the data to further testing not heretofore performed. Our confirmed two-factor structure and standardized factor loadings are supportive of previous clinical descriptions and of factor analytic studies performed in nondelirium and delirium consecutively assessed inpatients during their first 24 hours of hospitalization¹¹ or in delirium referrals to a consultation-liaison service,¹² but are not supportive of two other exploratory factor analyses on delirium referrals to consultation-liaison services, where three factors were proposed.^13,14 In those studies, motor retardation showed a complex (negative sign) loading on studies of patients referred to consultation liaison,^12–14 suggesting a bias toward hyperactive case detection. Differences in sampling design from those exploratory factor analyses may account for diversity of their results with all three studies on referred cases showing complex loading for motor characteristics and two leading to three factors. Studies performed on consultation-liaison referral patients with delirium may be more representative of presentations detected by nonpsychiatric clinicians and may be the most behaviorally compelling cases, whereas studies on the continuum of severity from nondelirium to delirium may better reflect the correlation (from very mild to severe scores) between the diverse characteristics (items) studied without a priori assumptions about the construct of delirium.

Delirium has been shown^17,18,26,27 to encompass three core domains of symptoms: Cognitive (attention and other cognitive deficits, including disorientation, memory and visuospatial impairment), higher level thinking (semantic language, thought process, and executive function), and circadian (sleep-wake and motor activity alterations), whereas less common, “noncore” symptoms (affective lability, delusions, and perceptual disturbances and hallucinations) are felt to not be common or representative of the construct of delirium across patients and etiologies. Franco et al²⁷ used CFA to validate this hypothesis and address the relationships among the core domain symptoms of delirium and found that those three domains were supported by the items comprising each of those three factors (with high correlations for items within each factor) that all loaded onto one core factor with high values as well. However, three noncore symptoms and diagnostic characteristics (temporal onset, fluctuation, physical disorder) were purposefully excluded from their CFA, and the two motor presentations were combined into one motor activity item that could reflect either hyperactivity, hypoactivity, or a mixed state to reflect any motor activity disturbance (i.e. they used only 10 of 16 DRS-R98 items and collapsed two of them into one). In that analysis, both sleep-wake cycle and motor activity disturbances loaded onto the circadian factor together. The purpose of that report was not to evaluate all symptoms a clinician might encounter irrespective of its frequency, but rather to verify the three core domains of delirium as suggested by other literature. Nor was it an attempt to understand the scale itself. Therefore, it differed in its constituents for the two factors from the current report findings.

Our analysis, using all DRS-R98 items, found F1 to be comprised of all core domain symptoms (cognitive, language, thought process, and sleep-wake cycle) except motor agitation, whereas F2 represented noncore symptoms of delirium. Motor presentation items separated onto the factors with retardation on F1 and agitation on F2 with the noncore symptoms. This is similar to that reported by Franco et al¹¹ in their exploratory factor analysis where a cognition factor including motor retardation and higher-order thinking alterations and an agitated/psychotic factor including affective lability were found, and to a lesser degree with that delineated by Jain et al,¹² with a cognitive factor including higher-order thinking items and a behavioral factor including motor items (negative sign for retardation), psychotic symptoms, affective lability, and sleep-wake cycle alterations comprising a circadian/noncore symptoms factor. Interestingly, the two reports from Grover et al¹³ and Mattoo et al¹⁴ that do not support our two-factor structure delineated three factors congruent with the three core domains of delirium.

Our CFA therefore lends credence to the two separate types of delirium manifestations, core and noncore, although we might have combined the motor items and discerned whether the three noncore symptoms would still have populated F2 by themselves. Because hyperactivity often loads with psychosis and affective lability when not combined or loading with hypoactivity, it suggests it is a component of circadian rhythm but also possibly affected by neurophysiological mechanisms underlying those noncore symptoms, such as excessive dopaminergic activity.

On F1 attention had the highest loading, consistent with its being the cardinal feature of delirium in diagnostic criteria and its high prevalence as a symptom ubiquitous in studies. Attentional impairment was found in 100%,¹⁵ or almost 100%,^28,29 of patients with delirium in a blinded assessment with the DRS-R98. Furthermore, the degree of inattention correlated with levels of severity of other core delirium symptoms.¹⁵ Disorientation, the symptom with the second higher loading on this report, also occurs often in delirium.^15,28,29

Our report adds to the prior work using both exploratory factor analysis and CFA that renders the construct of delirium more denotatively explicit through systematic evaluation of its symptoms and their interrelationships. It sets the stage for research into the endophenotype of the delirium construct using biological markers such as recent work reporting altered medial antero-posterior neural network connectivity during the resting state (default mode network) in delirium using functional MRI, where greater severity of delirium on the DRS-R98 correlated with greater alteration of connectivity.³⁰

It is also important to note that the two factors are oblique (correlated). This implies a superordinate or general factor accounting for the confirmed correlated two-factor structure such that delirium is a single condition/construct with some symptoms conferring subtler differences in presentation (noncore). From a practical standpoint, the findings support the continued use of a DRS-R98 Total score together with a cut score for determination of a delirium diagnosis. Moreover, computation of separate factor scores might prove useful in the future in relation to treatment and prognostic ramifications as there is speculation that the noncore symptoms may represent effects of certain underlying etiologies, where larger study samples are needed to further explore this.

Previous studies have reported little difference in symptom profile when delirium occurs alone or with concurrent dementia, where delirium symptoms overshadow those of dementia.^31–33 In this report, we focused on delirium phenomenology, so subjects with dementia were excluded to avoid misattributing symptoms of dementia to delirium. Given that in real-life practice with older patients, dementia is one of the major risk factors and comorbid conditions of delirium, we are conducting new studies to better decipher how clinical characteristics of delirium assessed with the DRS-R98 are modified by comorbid dementia.

Limitations of this study include not assessing probable etiologies whereby we might decipher the role of noncore symptoms and their relationship to core symptoms, especially circadian features. The latter probably requires a larger sample size. Another limitation is the cross-sectional nature of our data, which does not allow for assessment of the temporal evolution of symptoms. Nonetheless, our CFA was performed on subjects along spectrum from nondelirium to severe delirium, and provides a breadth of scores for analysis. Additionally, longitudinal reports found that delirium symptoms remain consistent in severity throughout episodes so long as diagnostic criteria are still met,^34,35 suggesting our cross-sectional sampling would be representative of delirium cases throughout their episode. Dementia was excluded clinically and not with an instrument, although many of our subjects were not elderly. Finally, further studies are needed to confirm our two-factor model of delirium assessed by the DRS-R98 in other populations such as pediatric and geriatric populations, where effects of neurodevelopmental or neurodegenerative comorbidities may influence delirium phenomenology.