Man Versus Machine Part 2: Comparison of Radiologists’ Interpretations and NeuroQuant Measures of Brain Asymmetry and Progressive Atrophy in Patients With Traumatic Brain Injury

Previously, we reported that NeuroQuant, an FDA-cleared method for measuring brain MRI volume, was more sensitive for detecting atrophy in patients with traumatic brain injury (TBI) than was the radiologist’s traditional approach, which is based on simple visual inspection.¹ Our previous study examined only cross-sectional (i.e., one point in time) atrophy.

In addition to simple or straightforward measures of brain MRI volume, other volume measures also are important in patients with TBI. Clinical experience has provided extensive evidence that traumatic injury to one side of the head or brain can cause greater brain atrophy on that side, particularly in the case of penetrating brain injuries. Therefore, measures of brain asymmetry are important in TBI. In addition, previous research has shown that the longitudinal design (obtaining data at more than one point in time) is more sensitive for detecting atrophy than is the cross-sectional approach (for review, see reference 2 ²; for more recent, see reference 3³).

The purpose of this study was to expand the previous one and test the hypotheses that, in comparison with the radiologist’s traditional approach, NeuroQuant would be more sensitive for detecting abnormal asymmetry and progressive atrophy in patients with TBI. An additional aim was to test the hypothesis that the NeuroQuant extended analysis (which measured 15 brain regions), in comparison with the NeuroQuant standard analysis (which measured three brain regions), would be more sensitive for detecting atrophy.

Methods

Subjects

Patients.

Included in this study were outpatients consecutively admitted to the Virginia Institute of Neuropsychiatry who met the selection criteria. Selection criteria required that each patient 1) was diagnosed with traumatic brain injury by a board-certified neuropsychiatrist (D.E.R.) according to the criteria of Menon et al⁴; 2) had a mild or moderate level of brain injury according to the criteria of Rao and Lyketsos⁵; 3) agreed to be in the study and signed the informed consent form; 4) had no contraindications to obtaining an MRI, such as having magnetic metal in the head or being pregnant; and 5) had an MRI without artifacts (such as motion artifacts), which would preclude accurate identification of brain structures by the NeuroQuant software. In addition, each patient was matched with a normal control to have a similar level of education (within 3 years), in order to minimize the potentially confounding effect of education on brain volume; three patients were excluded using this method because they had very low levels of education. This study was approved by the New England Institutional Review Board and satisfied the requirements of the Code of Ethics of the World Medical Association (Declaration of Helsinki) for human research.

Twenty-four patients met the selection criteria. Twenty-three had mild TBI and one had moderate TBI. Demographic characteristics were as follows: 10 men and 14 women; mean age in years was 46.8 (SD 13.0; range 19–66); mean number of years of education was 14.4 (SD 2.5; range 11–19).

Normal control subjects.

The NeuroQuant computer-automated analysis routinely provides volume data on 15 brain regions, left and right sides, for a total of 30 volume measurements (referred to herein as the “NeuroQuant automated analysis”) (http://www.cortechs.net/products/neuroquant.php).⁶ However, it provides comparisons to a normal control group for only three brain regions (averaged across left and right sides) (referred to herein as the “NeuroQuant standard analysis”). In order to assess NeuroQuant’s ability to detect atrophy in all 30 brain regions, this study used a group of normal controls different from the NeuroQuant normal controls. For these extended analyses (referred to herein as the “NeuroQuant extended analysis”), normal control data were obtained from a larger group previously studied as part of the Alzheimer’s Disease Neuroimaging Initiative (ADNI).^7–9 The ADNI normal control data were made publicly available (http://adni.loni.ucla.edu).

The ADNI was launched in 2003 by the National Institute on Aging (NIA), the National Institute of Biomedical Imaging and Bioengineering (NIBIB), the Food and Drug Administration (FDA), private pharmaceutical companies and nonprofit organizations, as a $60 million, 5-year public-private partnership. The primary goal of ADNI has been to test whether serial MRI, positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer's disease (AD). The Principal Investigator of this initiative is Michael W. Weiner, M.D., VA Medical Center and University of California - San Francisco. ADNI is the result of efforts of many coinvestigators from a broad range of academic institutions and private corporations, and subjects have been recruited from over 50 sites across the US and Canada.

For the NeuroQuant extended analyses reported herein, a subgroup of 20 normal control subjects (10 men, 10 women) were chosen from the ADNI database. The mean age was 68.3 years (SD 3.6 years; range 60.0–71.5), and the mean number of years of education was 16.0 (SD 3.1; range 9–20).

The groups of patients and ADNI normal controls did not differ significantly with respect to sex (chi square likelihood ratio=0.31, df=1,42, p=NS). In order to compare the two groups with respect to age, a nonparametric test (Wilcoxon rank sum test) was chosen because the normal control group’s data were not normally distributed. The two groups differed significantly with respect to age (chi square=29.4, p <0.01), with ADNI normal controls older than the patient group. The two groups did not differ significantly with respect to years of education (independent t test, t=1.81, df=1,42, p=NS).

Brain Imaging

Magnetic resonance imaging.

Each patient had a 3.0 Tesla MRI of the brain performed at one of five local radiology centers using the scanning protocol recommended for allowing later NeuroQuant analysis; this protocol is described in detail on the NeuroQuant website (http://www.cortechs.net/products/neuroquant.php) and was the same protocol used for the ADNI subjects. In addition to the general requirements for having an MRI (e.g., having no magnetic metal in the head), the NeuroQuant protocol required, at a minimum, the following:

As part of the standard clinical procedure at the Virginia Institute of Neuropsychiatry and nearby radiology centers, several other MRI sequences were obtained on each patient in order to allow a thorough evaluation of the effects of traumatic brain injury on brain structure. Accordingly, each MRI evaluated by the radiologists included the following sequences: 1) T1-weighted 3D sagittal sequence without contrast (which also was used for the NeuroQuant-based volumetric analyses); 2) coronal T2-weighted sequence; 3) axial FLAIR sequence; 4) susceptibility-weighted imaging (SWI) (preferably) or gradient-recall echo (if SWI unavailable); and 5) diffusion tensor imaging.

Longitudinal MRI data collection.

In order to examine possible progressive atrophy, brain MRI data were collected more than once per subject. In order to avoid interscanner differences, repeat MRI scans were performed on the same scanner as the initial MRI for each subject. For the ADNI normal controls, MRI data were collected initially and 1 year later. For the patients, MRI data were collected initially, about 6 months later, and about 14 months later. Some patients did not return for follow-up and, therefore, did not have a second or third MRI scan. In summary, 24 patients had an initial MRI scan, 21 (87.5%) had a second MRI scan (at a median of 6.3 months), and 9 (37.5%) had a third MRI scan (at a median of 13.7 months).

NeuroQuant automated brain MRI segmentation.

The brain MRI data for each patient or ADNI normal control was uploaded to the NeuroQuant server, which processed and analyzed the brain imaging data. This computer-automated analysis involved several steps, including stripping the brain of scalp, skull, and meninges; inflating the brain to a spherical shape; mapping the spherical brain to a common spherical space shared with the Talairach atlas brain¹⁰; identification of brain segments (that is, regions); and deflation of the patient’s brain back to its original shape while retaining the identifying information for brain segments. The output of the NeuroQuant computer-automated analysis included a report, which contained volumetric information, and a set of DICOM-formatted brain images, which were segmented, with each region identified by a distinctive color.

Segmentation errors.

The NeuroQuant segmented DICOM images were inspected for errors, a step recommended by the makers of NeuroQuant in order to ensure accurate identification of brain regions by the software. The left and right counterparts for each of the 15 brain regions were segmented. Therefore, for each subject, there were 30 brain regions segmented. The segmentation results for each region were visually inspected by one of the authors (D.E.R. and A.L.O). For brain regions associated with NeuroQuant segmentation errors, those volume measures were considered invalid and were not included in further analyses.

NeuroQuant standard versus extended analyses.

Two types of NeuroQuant analyses were used to test the hypotheses in this study: standard and extended. The NeuroQuant standard analysis was automatically performed by the server and presented in a report called the “Age-Related Atrophy” report by CorTechs Labs, manufacturer of NeuroQuant. The NeuroQuant extended analysis was performed by the authors at the Virginia Institute of Neuropsychiatry using the ADNI normal controls subjects described above.

For the NeuroQuant standard and extended analyses, each brain region volume was corrected for interindividual differences in head size by dividing by intracranial volume, with the result being expressed as a percentage.

NeuroQuant standard analysis of atrophy.

The NeuroQuant standard analysis reported results for cross-sectional measurements of brain volume for three regions (hippocampus, lateral ventricle, and inferior lateral ventricle), compared with the CorTechs Labs normal control subjects.

NeuroQuant extended analysis of atrophy.

For the NeuroQuant extended analysis, the results from the automated NeuroQuant analyses of the ADNI normal control data were used to determine means and standard deviations for each of the 15 brain regions (left and right sides analyzed separately). Each patient’s data were compared with the data from the normal controls in order to calculate z-scores which were converted to normative percentile ranks.

Results were considered to be consistent with parenchymal atrophy if they met one of the following criteria: 1) parenchymal volume ≤5th normative percentile; or 2) ventricular volume ≥95th normative percentile, consistent with atrophy of the surrounding parenchyma.

NeuroQuant extended analysis of asymmetry.

Asymmetry was defined using a standard definition adopted by the NeuroQuant software: asymmetry index=difference between left and right volumes, divided by their mean, expressed as a percentage. Expressed simplistically, the asymmetry index was a measure of how much bigger (or smaller) the left side of the brain was compared with the right.

With asymmetry, in contrast to atrophy, in general there was no a priori hypothesis regarding directionality; in other words, rightward asymmetry was expected to occur about as often as leftward asymmetry. Therefore, with respect to defining cutoff points for abnormality, in order to maintain the rate of false positive findings at 5%, a 2.5% cutoff was used at both ends of the normal distribution, adding up to an overall rate of 5%. This approach was analogous to using a two-tailed test of statistical significance. Therefore, results were considered to be consistent with abnormal asymmetry if they met one of the following criteria: 1) an asymmetry value ≤2.5 normative percentile; or 2) an asymmetry value ≥2.5 normative percentile. However, if there was an a priori rationale for expecting directionality, 5% (for leftward asymmetry) and 95% (for rightward asymmetry) cutoffs were used. A priori rationales for expecting directionality were based on ipsilateral signs consistent with head or brain injury, including the following:

NeuroQuant longitudinal analysis.

Longitudinal analysis was conducted using previously published methods.¹¹ This analysis was conducted by subtracting, for each region, the volume at the follow-up scan from the volume at the initial scan, dividing the result by the volume at the initial scan, and expressing the resulting proportion as a percentage change. For follow-up scan data collected more than 1 year after the initial scan, the data were annualized [i.e., the annual rate of volume change was calculated by dividing the percentage change by the duration between scans (measured in years)]. For follow-up scan data collected less than 1 year after the initial scan, the data were not annualized in order to avoid amplifying noise and extrapolating beyond the time period of the data collected.

Results were considered to be consistent with progressive parenchymal atrophy if they met one of the following criteria: 1) a decrease in parenchymal volume ≤5th normative percentile; or 2) an increase in ventricular volume ≥95th normative percentile, consistent with progressive atrophy of the surrounding parenchyma.

Radiologist’s traditional interpretation.

For each patient, the MRI was interpreted by one of eight local, board-certified radiologists based on simple visual inspection per the usual clinical practice. For patients who had follow-up MRI scans, the radiologist compared the later images to the earlier images in all cases. The radiologists were blind to the NeuroQuant results. The radiologist’s interpretation was examined to determine if any of the following were present:

Statistics

Two-tailed paired sign tests were used to test the hypotheses that NeuroQuant standard findings differed from NeuroQuant extended findings, and that the NeuroQuant findings differed from the radiologist’s interpretations. JMP software was used to perform the statistical analyses.

Results

A two-tailed paired sign test showed that a significantly higher percentage of patients had atrophy identified by the NeuroQuant extended analysis (12 of 24 patients; 50.0%) than by the NeuroQuant standard analysis (four of 24 patients; 16.7%) (test statistic M, two-tailed=4.00, p=0.04).

Comparisons of detection rates for the NeuroQuant methods versus the radiologists are shown in Table 1.

Figures 1a and 1b show an example of a patient with abnormal brain volume identified by NeuroQuant, which was not identified by the radiologist.

Discussion

Main Findings

The main finding of this study was that NeuroQuant was more sensitive for detecting atrophy, abnormal asymmetry and progressive atrophy of the brain than the method of simple visual inspection traditionally used by radiologists. Furthermore, the NeuroQuant extended analysis was more sensitive for detecting atrophy than was the NeuroQuant standard analysis (the latter is called the “Age Related Atrophy” analysis by CorTechs Labs). The results of this study expanded on those of our previous related study¹ in several ways.

The NeuroQuant standard analysis found at least one sign of atrophy in 16.7% of patients; in contrast, the NeuroQuant extended analysis found at least one sign of atrophy in 50.0% of patients. This finding was not surprising, given the fact that the NeuroQuant standard analysis compared only three brain regions to normal controls, with left and right brain regions added together, for a total of three volumes; whereas the NeuroQuant extended analysis compared 15 brain regions to normal controls, with left and right brain regions analyzed separately, for a total of 30 volumes.

The radiologist’s traditional approach found at least one sign of atrophy in 12.5% of patients; this rate was similar to that of the NeuroQuant standard analysis.

The radiologist’s traditional approach found no signs of abnormal asymmetry. In contrast, the NeuroQuant extended analysis found this abnormality in a majority (83.3%) of patients. Surprisingly, although there is a vast amount of clinical experience that TBI causes greater brain atrophy on the side of the impact (especially with penetrating injuries), there is little published literature on brain volume asymmetry in patients with TBI. In fact, a PubMed search by the authors using the terms “asymmetry” and “traumatic brain injury” found no studies in human subjects. The high rate of abnormal asymmetry found in the current study suggests that measurement of brain volume asymmetry could be an important area for future research.

Regarding the failure of the radiologists to identify abnormal asymmetry, there appear to be two explanations: 1) it is difficult to see subtle asymmetry, perhaps e.g. asymmetry indices<10%−15%; and 2) when asymmetry can be seen, it usually is interpreted as being within the normal range. Regarding the latter point, in our experience, radiologists occasionally notice asymmetry in brain structures, but in most cases, they believe it is within the normal range. However, when asked what the cutoff is between normal and abnormal asymmetry, usually they do not know, because they were not taught it and do not routinely measure brain volume. For example, the cutoff between normal and abnormal asymmetry of the lateral ventricles, based on our sample of ADNI normal controls, is about 25%. This degree of asymmetry is easily visible. So while it is true that some normal brain regions can be clearly asymmetric, it does not follow that, therefore, most of the obviously asymmetric regions in a subject’s brain are normal. The results of the current study showed that the large majority of the patients had abnormal asymmetry of the brain, and the radiologists failed to identify any sign of asymmetry. (Note that our sample consisted of patients with mild to moderate TBI whose symptoms persisted for months to years after the injury; therefore, these findings may not apply to patients whose symptoms completely abate within days to weeks after the injury.) The good news is that, now that computer-automated volumetric MRI techniques are readily available, radiologists and other brain specialists can learn what the cutoffs are between normal and abnormal asymmetry, and what the related brain images look like. In other words, not only has the machine beat man in this contest, the machine has begun teaching man how to do the job better.

Similar to the findings discussed above for asymmetry, the radiologist’s traditional approach found no signs of progressive atrophy, and the NeuroQuant extended analysis found this abnormality in a majority (70.0%) of patients. In our experience, unlike the case with asymmetry, most cases in which progressive atrophy occurred were difficult to appreciate visually, even using simultaneous visual inspection of both sets (time 1 and time 2) of NeuroQuant segmented brain images, which were coregistered in three-dimensional space. The reason for this seemed to be that changes of less than about 10%−15% of volume, typically spread throughout the three-dimensional brain structure, were difficult to see. And most of the longitudinal abnormal changes in brain volume were less than 10%−15%. In contrast, many of the abnormal asymmetries were greater than 15% and therefore were easier to appreciate visually. The fact that longitudinal abnormal changes are relatively difficult to see makes it more important that volumetric techniques be used to assess for these changes.

Limitations

Although brain atrophy or asymmetry commonly is caused by TBI, it is not always true in a given patient that atrophy or asymmetry found months or years after the injury was caused by the injury. Patients with persistent symptoms from TBI (like the patients in this study) often have pre-accident neurological or psychiatric disorders which can cause abnormal brain structure.

In comparison with the group of patients, the ADNI normal controls used in this study for the NeuroQuant extended analyses were significantly older. It is well-known that increasing age, especially over 50 years old, is associated with brain atrophy.^12–14 Because the ADNI normal controls were older than the patients in the current study, it was quite possible that the patients had a higher rate of atrophy than was revealed by the NeuroQuant analyses. Therefore, this was a conservative limitation. It was unlikely that NeuroQuant would have found atrophy in the patients when it did not actually exist, or that the results would have been biased in favor of NeuroQuant finding more atrophy than that found by the radiologists, who knew each patient’s age and could factor that information into their decision about presence of atrophy.

Furthermore, it was unlikely that the older ages of the normal controls affected the rates of asymmetry found by NeuroQuant because it is unlikely that normal aging in adults causes brain asymmetry. The primary effect of aging on brain volume is generalized atrophy, probably because the left side of the brain ages at the same rate as the right. This conclusion is supported by normal control data provided by CorTechs Labs, which shows that for multiple brain regions, volumes plotted versus age over the range of 18- to 90-years old showed no change of asymmetry with age.

There were some reasons why the radiologists involved may have underreported brain atrophy or asymmetry. They might have noticed some mild atrophy or asymmetry but not reported it because it did not seem clinically important. On the other hand, while it is true that there is no specific treatment for atrophy in patients with chronic TBI, atrophy does have prognostic value^2,11 and awareness of the existence of atrophy may be the first step toward developing treatments to halt its progress. In support of the radiologists, they were not informed that they were being checked against a machine, and if they were asked to detect the mildest asymmetry or atrophy, perhaps they would have been more sensitive. But the purpose of this study was to assess how radiologists would perform in a typical clinical setting, and tipping them off would have unnecessarily unblinded that aspect of the study. Finally, one could argue that, if the cutoffs for abnormality used in this study (5% and 95%) had been made more stringent (1% and 99%, for example), the radiologists might have fared better because it was more likely that they noticed severe atrophy. Although the latter point is true, the problem with using stricter criteria for the cutoffs for abnormality would be the failure to identify patients with mildly abnormal volume.

The radiologists had a potential advantage over NeuroQuant (which was based only on a T1 MRI sequence) because they had a greater range and amount of brain imaging data available (including T2, FLAIR, etc., in addition to the T1 sequence). Therefore, at least with respect to this issue, the results could have been biased in the favor of the radiologists finding atrophy. The fact that the results actually showed the opposite make it somewhat more remarkable that NeuroQuant was more sensitive for detecting atrophy or asymmetry than was the traditional approach used by the radiologists.

Conclusions

NeuroQuant was more sensitive for finding atrophy, abnormal asymmetry, and progressive atrophy than the traditional radiologist’s approach. However, the radiologist’s approach is better for finding nonvolume related abnormalities. Therefore, the two approaches are complementary.