Diagnostic Utility of the International HIV Dementia Scale for HIV-Associated Neurocognitive Impairment and Disorder in South Africa

HIV-associated neurocognitive impairment and disorder occurs in 25%−60% of HIV-infected patients internationally.¹ Overall, studies from North America and Western Europe have focused on the most severe form of HIV-associated neurocognitive disorder (HAND): HIV-associated dementia (HAD).² However, the cumulative risk for this disorder now is estimated to be only 2% of HIV-infected persons in the United States.³ A milder form of HAND, mild neurocognitive disorder (MND), is estimated at a prevalence of 12%. MND, in turn, is bordered by the less severe, subclinical condition of HIV-associated asymptomatic neurocognitive impairment (ANI)—currently estimated at a prevalence of 33%. With the reduced prevalence of HAD, MND has become a more important focus for clinical intervention—analogous to minor cognitive impairment versus Alzheimer’s disease in the general population. Reports from sub-Saharan Africa^4–8 suggest that a variety of factors may cause a divergence from HAND rates observed in North America and Western Europe, including a greater divergence in HIV clades (particularly a higher frequency of clade C), a greater frequency and divergence of HIV-associated and central nervous system-specific comorbidities, more patients at an advanced HIV disease stage, and lower access to effective antiretroviral therapy. In kind, HAND may be similarly improved by current antiretroviral regimens.⁸ Despite the established benefit of making HAND diagnoses worldwide, the process of establishing the reliability and validity of HAND diagnoses in sub-Saharan Africa remains challenging.

The diagnosis of HAND by the Frascati conference criteria⁹ prioritizes the administration of a neuropsychological test battery. This is not feasible in Voluntary Counseling and Testing (VCT) clinics in South Africa because it requires a neuropsychologist, a trained psychometrist, specialized neuropsychological testing materials, and as much as 2–3 hours of assessment time per patient. Screening tests for HAND have been recommended by the Frascati conference⁹ in such limited-resource settings, and several screening tests for HAND have been used.^10–13 Ganasen et al reported that one of these screening tests, the HIV Dementia Scale (HDS), showed a sensitivity and specificity of approximately 80% in a South African sample.¹⁴ However, this may well represent an overestimate because the comparator was another screening test more sensitive to cortical dysfunction—the Mini-Mental Status Examination—as opposed to a comprehensive HIV-specific neuropsychological test battery focused more on subcortical dysfunction. The HDS has been widely studied as a screening test for HAND in numerous HIV-infected samples.^15–18 One issue has been the scoring of the antisaccadic eye movement error item.¹⁸ In addition, the timed alphabet and cube copy items are susceptible to cultural bias. Hence, the International HDS (IHDS) was developed to address this issue of cultural bias. It is a brief, culturally neutral test not requiring English language knowledge, high educational level, or special instrumentation and can be administered by trained, nonclinical personnel.¹¹

The use of the IHDS has been supported by several studies. Per the South African antiretroviral roll-out program, patients with AIDS-defining illnesses, including HAD,^19,20 are eligible for antiretrovirals—regardless of CD4 cell count. The study’s aim was to determine the potential utility of the IHDS in an antiretroviral-naïve sample expected to benefit from HAND screening. At the time of this study, patients with a CD4 cell count <200 cells/mm³ were considered treatment eligible in South Africa. Hence, we chose a CD4 cell count range that was the next highest (200–350 cells/mm³) to administer the IHDS as a HAND screening test (together with a brief neuropsychological test assessment for HAND) in an antiretroviral-naïve sample. The IHDS had been initially standardized against a larger neuropsychological battery as well as neurological examination and a functional status assessment. The association of the IHDS with neuropsychological test performance was consistent across neuropsychological tests in that study. Thus, we anticipated that this would justify our selection of a smaller neuropsychological battery from that group of tests. We focused on two tests given there, the Digit Span and the Trial Making Test, both of which showed significant associations with the IHDS score in a similar population (the Ugandan population). Moreover, the selection of these two neuropsychological tests is very similar to that of the Trail Making Test (Parts A and B) and the WAIS-R Digit Symbol sub-test by the AIDS Clinical Trials Group (ACTG) for its ALLRT (ACTG Longitudinal Linked Randomized Trials) study merging cognitive data across multiple ACTG trials to generate data on cognitive performance among HIV-infected individuals.²¹ We predicted that the IHDS would be a valid HAND screening test in this South African VCT clinic setting compared with the combination of the Digit Span and Trial Making tests.

Measures

International HIV Dementia Scale

The IHDS consists of three items: a timed finger tapping test, a timed alternating hand sequence test, and a short-term verbal memory test of four items at 2 minutes.¹¹ Each item contributes four points to the total score of 12. The IHDS was originally validated in both a US and sub-Saharan African population (from Uganda). Administration requires approximately 10 minutes. At the cut-off score of ≤10, the IHDS demonstrated 80% sensitivity and 57% specificity.

Validation of HAND by Neuropsychological Test Battery

We used a brief neuropsychological battery as the standard for validation of HAND diagnosis. Attention was assessed using the Digit Span Forward and working memory by the Digit Span Backward subtests of the WAIS.²² Information processing speed was assessed by the Trail Making Test Part A and executive function by the Trail Making Test Part B.²³ The latter two tests also reflect psychomotor function. We recently published the norms for the above tests derived from HIV-seronegative patients attending the VCT clinic at this hospital.²⁴

Statistical Analysis

The data were analyzed on JMP 8 software.²⁵ We constructed two nonparametric ROC curves to determine the ability of the IHDS to discriminate HAD and the milder neurocognitive conditions of MND merged with ANI.²⁶ Using the local neuropsychological test norms we developed,²⁴ we classified patients into ANI/MND or HAD. If a subject had a 1–2 SD difference on two or more of the four neuropsychological tests (Digit Span Forward, Digit Span Backward, Trail Making Test Part A, or Trail Making Test Part B), they were classified as ANI/MND. Subjects who had a ≥2 SD difference on two or more neuropsychological tests were classified as HAD. These tests represent only the neuropsychological criteria for HAND diagnoses; functional status was not assessed, and an exclusionary medical workup for other causes was not conducted.⁹ For the IHDS, sensitivity (the true positive rate) and “1 − specificity” (the false-positive rate) were computed for each possible score to determine the optimal IHDS cut-off scores for ANI/MND and HAD. We also present the positive predictive value (PPV), the negative predictive value (NPV), and the overall accuracy.

Results

Results of group demographic comparisons are presented in Table 1. Our final analytic sample (N=70, with loss of three from point of recruitment) was comprised predominantly by women (81%). The mean age was 31.5 years (SD = 7.99), with no significant differences by HAND grouping. The average level of education was 9.0 years (SD = 2.88); the HAD group showed fewer years of education than the neurocognitively normal group (p=0.044), in line with previously published work.²⁷ The mean CD4 cell count was 267 cells/mm³ (SD = 41.7), with no significant differences by HAND grouping.

Using our normative data for the Trail Making Test and the Digit Span Test together with the diagnostic criteria promulgated at the Frascati Conference,⁹ 29% (N=20) were categorized as ANI/MND and 14% (N=10) as HAD. The IHDS item and total scores, as well as the neuropsychological performance on the Digit Span Forward, Digit Span Backward, Trail Making Test Part A, or Trail Making Test Part B, are summarized by HAND disorder groupings (with omnibus analysis of variance p values) in Table 1. For all overall group comparisons on the foregoing measures, all but two were statistically significant. Those two scores were the IHDS motor item score (finger tapping) and the Digit Span Backward score, with the latter showing a trend toward statistical significance.

Specific group contrast comparisons (data not shown) showed a significant difference between the normal group and the ANI/MND group (p=0.018) on the IHDS total score. Likewise, there was a significant difference between the normal group and the HAD group (p=0.01). There was no statistically significant difference between the ANI/MND group and the HAD group (p=0.72). In kind, there were no statistically significant contrast comparisons between those with ANI/MND and those with HAD on any of the IHDS item scores. Although the Digit Span Forward and Digit Span Backward test contrasts were not statistically significantly different between the ANI/MND and HAD groups, the Trail Making Test Part A and Trail Making Test Part B did distinguish between the ANI/MND and the HAD groups (p=0.014 and 0.001, respectively).

At the generally recommended cut-off score of 10.0, the ROC curve model was not statistically significant for ANI/MND in this sample (Table 2). However, a trend toward significance (p=0.06) was shown, yielding a sensitivity of 60% and a specificity of 68%. Overall accuracy was 70% for any level of HAND (ANI/MND merged with HAD; Table 3), 66% for ANI/MND, and 66% for HAD (Table 2). The PPV of the IHDS for ANI/MND was 36%−66%, with the cut-off score ranging from 11.5 to 6.5. The PPV for HAD was fairly low (16%−50%) for the same IHDS range. However, at the cut-off scores of 10.0 and 10.5, the NPVs were 93% and 95%, respectively, for HAD.

To determine the optimal cut-off score for the IHDS to maximize its sensitivity and specificity, additional ROC model analyses were performed at the cut-off score of 10.5. For any level of HAND, a ROC model analysis showed that the IHDS correctly determined presence versus absence in 69% of cases (p=0.003), with a sensitivity of 69% and a specificity of 74% (Table 3). The ROC measure of the area under the curve (also referred to as A′), equal to the probability that a classification will rank a randomly chosen positive instance higher than a randomly chosen negative one, was 0.73 for this model, confirming the value of this cut-off score using a separate ROC model outcome measure. A significant model was also obtained at this cut-off score for ANI/MND, yielding a sensitivity of 65% and a specificity of 62% (Table 2; Figure 1) and for HAD yielding a sensitivity of 80% and a specificity of 60% (Table 2; Figure 2).

Discussion

The established prevalence rates of HIV-associated neurocognitive impairment and disorder in sub-Saharan Africa highlight the urgency to develop sensitive and specific screening tests. Based on our abbreviated neuropsychological test battery, 43% of this sample showed some degree of neurocognitive impairment, reflecting either ANI/MND or HAD. This finding is quite consistent with findings from Lawler et al, who reported a prevalence of neurocognitive impairment of 38% in Botswana.⁷ Thus, we suspect a high prevalence of HAND in sub-Saharan Africa may exist more generally but that HAND nonetheless remains largely undetected, despite the availability of current screening tools.

Results from the ROC model analyses suggest that the IHDS is a suitable and efficient screening test for both ANI/MND and HAD in resource-limited settings, such as HIV VCT clinics in sub-Saharan Africa. Although the sensitivity was less for ANI/MND than for HAD, our data suggest that the IHDS is useful across the spectrum of HAND. Further, as aforementioned, the IHDS can be administered within approximately 10 minutes. The PPV reflects the percentage of true positive calls by the test being evaluated to the total number of positive calls (i.e. including the false positives)²⁸ and was fairly low for the IHDS on HAD. The PPV was likely impacted by a number of competing causes of cognitive impairment other than HIV itself in this patient population, including a history of alcohol and substance use, major depressive disorder, central nervous system opportunistic infections, and the effects of some of the antiretroviral medications themselves, among others. The test sensitivity reflects the percentage of true positive calls by the test being evaluated to the total number of testees with the disease being screened (i.e., including the false negatives). The sensitivity is higher than the PPV, as a false-negative test is less likely on the IHDS than a false positive.

Based on our ROC analyses, we recommend that a cut-off score of 10.5 be used in South Africa rather than 10.0, which is the generally recommended cut-off score on this scale.¹¹ Of note, the standardization article on the IHDS indicated that a cut-off of 10.5 provided even greater sensitivity with a minimal loss of specificity. The fact that the specificity of the IHDS declines rapidly with higher scores suggests the need for the exclusionary medical workup, as do the low PPVs. In line with these results, the high NPVs across the cut-off scores for HAND indicate that when a person is negative on the IHDS total score, there is a very high probability they do not have HAND (as the confound of other medical conditions causing neurocognitive impairment does not impact the negative PPVs). Regarding the lower level of sensitivity we observed for ANI/MND, the scale might be modified to detect milder forms of HAND. One possibility would be to use a longer delay for the word-recall item. Another issue is educational level; possibly, the lower IHDS scores in the HAD group are driven by lower educational level and decreased cognitive reserve. We deem this unlikely, however, because additional analyses indicate that significant group differences remain when educational level is included as a covariate (data not shown). In fact, results from the analyses of covariance show no relationships, in general, between educational level and the IHDS total score.

To assess the clinical utility of the IHDS, we discussed several test properties (sensitivity, specificity, PPV, and NPV) by different cut-off scores and severity of HAND. In terms of overall accuracy, it is high in this sample. This is, in part, related to the relatively high prevalence of the disease (HAND) in our sample; because we evaluated the IHDS in patients restricted to an upper limit of CD4 cell count of 350 cells/mm³, HAND prevalence was relatively high. This allowed us to surmount the potential limitation of a high CD4 cell count on test accuracy: the higher the disease prevalence, the higher the accuracy.²⁹ We conclude that our results support the use of the IHDS total score for identifying HAND in South Africa, at least at this range of CD4 cell counts. It should be cautioned that these results require confirmation by studies of other HIV-infected samples in South Africa, as well as other parts of sub-Saharan Africa before they might be considered generalizable to the region.

This study has several additional strengths. We used focused, brief, and culturally neutral neuropsychological tests that do not require special instrumentation, and we applied locally validated, published norms. We also accrued a sample representative of the HIV seropositive population in South Africa. The two tests we used (the Trail Making Test and the Digit Span Test) can be recommended for research studies across a variety of HIV clinical care settings in South Africa to further document their generalizability to the South African population. We selected a sample of patients who were expected to most clearly benefit from effective antiretroviral therapy. With a HAND prevalence of 43% in patients within our stipulated CD4 cell count range from 200 to 350 cells/mm³, we have demonstrated a clear benefit of screening for neurocognitive impairment.

Regarding antiretroviral treatment initiation in the region, the World Health Organization recommended that the CD4 cell count threshold for initiation should be 350 cells/mm³ in 2010,³⁰ which is an increase from the previous recommended threshold of 200 cells/mm³. This recommendation for earlier antiretroviral initiation was adopted in the South African National Guidelines in 2013.³¹ It is also recommended by both sources that clinically symptomatic HIV infection in patients with World Health Organization stage 3 and 4 disease (which would include HIV-induced cognitive impairment) should initiate antiretroviral treatment. Thus, the IHDS would contribute to the decision to initiate antiretroviral therapy in this region, directly impacting patient care in this resource-limited setting.

At higher CD4 cell counts, there may be too many false-positive results generated to warrant screening with the IHDS. Yet, longitudinal study of screening of patients in the higher CD4 cell count ranges (>500 cells/mm³) may prove justified, especially as the current recommendation in the United States is for universal treatment for HIV-infected individuals, regardless of CD4 cell count,³² and as this is being advocated more frequently in high-resource countries. Moreover, the associated risk of a false-positive screen is not typically clinically significant, if the diagnosis is ruled out on a later visit. Given the apparent limits of the IHDS in detecting ANI/MND in this sample, the IHDS and the brief neuropsychological validation tests we used (the Digit Span Test and the Trail Making Test) may be studied for use together in a two-part screening process. In fact, a recently published study from Thailand has demonstrated increased performance when the IHDS was combined with the Trail Making Test Part A.³³ Regarding the IHDS, our data suggest that the finger tapping item could be considered for deletion. A focused approach to add neuropsychological tests to the IHDS might prove particularly relevant to increasing the yield of positively screened cases from the South African patient population in the higher CD4 cell count ranges.

This study has several limitations worthy of note. In terms of HAND characterization, we did not include a functional status measure to confirm HAND diagnosis (cf. with ref. 9). In addition and more specifically, we therefore could not distinguish between ANI and MND, resulting in their merged consideration here. Future research should examine the sensitivity and specificity of the IHDS for each diagnostic category of HAND (ANI, MND, and HAD) versus HIV-seropositive and -seronegative age- and education-matched controls and include a measure of functional status, as well as a medical workup to exclude other medical conditions that might account for neurocognitive impairment. We also did not use a full neuropsychological test battery recommended by the Frascati conference criteria.⁹ The two neuropsychological tests used here were selected to assess the domains of attention, working memory, information processing speed, and executive function. However, this is not equivalent to a comprehensive screening of all neurocognitive domains. Thus, patients with equivocal screening results and/or relevant deficits in functional status in activities of daily living should be referred for more extensive testing. In addition, our sample is relatively small, has a limited age range, and is predominantly female. However, this sample is representative of the patient population presenting at HIV VCT clinics in South Africa. With a minor modification in the cut-off score, we conclude that the IHDS is a valid and feasible diagnostic screening test for HAND in clinical settings in South Africa.