
J Neuropsychiatry Clin Neurosci 13:5-21, February 2001
© 2001 American Psychiatric Press, Inc.
Role of Serotonin in the Behavioral and Psychological Symptoms of Dementia
Krista L. Lanctôt, Ph.D.,
Nathan Herrmann, M.D., F.R.C.P.C. and
Paolo Mazzotta, M.Sc.
Received July 28, 1999; revised March 23, 2000; accepted July 7, 2000. From the Division of Clinical Pharmacology and Department of Geriatric Psychiatry, Sunnybrook and Women's College Health Sciences Centre and Departments of Psychiatry and Pharmacology, University of Toronto, Toronto, Ontario, Canada. Address correspondence to Dr. Lanctôt, Clinical Pharmacology and Psychiatry, Sunnybrook and Women's College Health Sciences Centre, 2075 Bayview Avenue, Room FG-20, Toronto, Ontario M4N 3M5 Canada. E-mail: krista.lanctot{at}swchsc.on.ca

|
ABSTRACT
|
The behavioral and psychological symptoms of dementia (BPSD) can have serious debilitating effects on the patient and increase caregiver burden. Investigations into the underlying neuropathology indicate that the serotonergic system may contribute to BPSD. In addition, serotonergic pathways are known to interact extensively with the cholinergic, noradrenergic, GABAergic, and dopaminergic systems. Hence, serotonergic therapies may be used to manipulate other neurotransmitters systems to alleviate BPSD or in combination with agents specific for the other neurotransmitter receptor sites. Neurotransmitter-modulated behaviors and evidence provided by pharmacological interventions are reviewed, focusing primarily on the serotonergic system.
Key Words: Dementia Serotonin

|
INTRODUCTION
|
Behavioral disorders have long been recognized as a common feature in Alzheimer's disease (AD) and related dementias. The nature and frequency of the behavioral and psychological symptoms of dementia (BPSD) have been outlined in numerous papers.1,2 BPSD can include psychotic symptoms such as hallucinations and paranoid/delusional ideation, nonpsychotic symptoms such as aggression and wandering, and affective disturbances and anxieties/phobias.1 BPSD occur throughout the course of dementia, irrespective of level of cognitive impairment.3 It has been estimated that up to 90% of patients display BPSD.1,4 Frequently identified BPSD include delusions (10% to 73%), wandering (10% to 70%), aggression (18% to 65%), motor restlessness (21% to 60%), hallucinations (3% to 49%), and depression (30% to 40%).1,2,5 These BPSD are distressing to the caregivers and are factors stimulating the decision to institutionalize.5,6 Furthermore, the presence of BPSD may have a negative impact on the course of the disease.79 Knowledge of the importance of BPSD for both caregivers and patients has stimulated research into their underlying pathological and neurochemical substrates.
Strong, consistent evidence supports the presence of significant disruptions in global serotonergic neurotransmission in dementia.10 Serotonergic neurons originating from the dorsal and median raphe nuclei innervate many structures in the cortex and limbic system.11,12 These widespread projections enable the serotonergic system to regulate aggression, mood, feeding, sleep, temperature, sexual activity, and motor activity.11,13 Therefore, alterations in the functioning of the central serotonergic system can be expected to have a clinically discernible impact on behavior.
The serotonergic system is heterogeneous in terms of structure and function, making it a complex system to study. Moreover, the dysfunction of the serotonergic system studied in isolation cannot be expected to explain changes in behavior, given that many neurotransmitters work in conjunction with each other. For example, although serotonin (5-hydroxytryptamine; 5-HT) generally plays an inhibitory role in the human cortex, overall excitability is mediated by acetylcholine, -aminobutyric acid (GABA), noradrenaline, histamine, and purines.14 There is some evidence that each of these neurotransmitters is altered in AD and related dementias,1520 and each has a role in controlling human behavior. Hence, optimum treatment efficacy may be achieved by correcting multiple neurotransmitters or the balance between neurotransmitters.
We will review evidence supporting neurochemical alterations and putative links with BPSD for the serotonergic system and four neurotransmitters that interact extensively with serotonin: acetylcholine, noradrenaline, dopamine, and GABA. For each section, the functional role of the neurotransmitter, the evidence supporting a disruption in AD, and any studies indicating a link with BPSD will be reviewed. Any evidence for other dementias will immediately follow. Neurotrophic factors also control behavior. The neuropeptides somatostatin, neuropeptide Y, and brain-derived neurotrophic factor also interact extensively with serotonin and will be briefly discussed.
Articles were retrieved from various sources, including electronic databases (MEDLINE, EMBASE) and cross-references from relevant articles. The following keywords were used: serotonin, acetylcholine, dopamine, norepinephrine, gamma-aminobutyric acid, GABA, neuropeptides, neurotrophic factors, neuropeptide Y, somatostatin, brain derived neurotrophic factor, Alzheimer's disease, dementia, behavior, behavioral disorders, delusions, hallucinations, agitation, aggression, apathy, disinhibition, depression, cognition, and non-cognitive.

|
SEROTONERGIC SYSTEM
|
The evidence supporting the role of serotonin disruption contributing to BPSD consists of biologic plausibility, demonstration of disruptions in dementia, studies linking disruptions with BPSD, and clinical evidence from serotonergic medications. Most of the studies have been done in patients with Alzheimer's disease (Figure 1). This evidence is reviewed below.

View larger version (33K):
[in this window]
[in a new window]
|
FIGURE 1. Observations supporting the putative role of 5-HT in the behavioral and psychological symptoms associated with Alzheimer's disease
|
In Vitro Evidence of Serotonergic Disruption in Dementia
Neurochemical and neuropathological disruptions in the serotonergic system have been established in AD. Loss of neurons in the raphe nuclei has been demonstrated,21,22 as has loss of serotonergic nerve terminals in the neocortex.15 Decreased concentrations of 5-HT and its major metabolite 5-hydroxyindoleacetic acid (5-HIAA) have been demonstrated in the central nervous system by use of postmortem brain studies, particularly in the temporal cortex16,17,19,2327 and in the cerebrospinal fluid (CSF).28 Although indirect, taken as a whole these data provide evidence that there are changes in the actual functional status of central serotonergic transmission. The presence of expected adaptive consequences due to serotonergic lesions, such as supersensitivity, tolerance, trophic changes, or other changes in 5-HT-modulated central functions,29 remains to be elucidated.
The actions of 5-HT are mediated through at least seven major receptor classes that have differing placements in the synapse, utilize different second messenger systems, and have different locations in the brain.12,29 5-HT1A receptors are located both presynaptically (somatodendritic in the raphe nuclei) and postsynaptically (terminals in the limbic system, hypothalamus, hippocampus, frontal cortex, and brainstem) and can have contradictory actions on intracellular signaling.12,29 This receptor subtype is thought to be involved with anxiety, depression, sexual behavior, and aggression12 (Table 1), as well as appetite. The functions influenced by 5-HT1D , 5-HT1Dß, 5-HT1E, and 5-HT1F are still being elucidated, and functions identified so far are not related to behavior.12 Deficits involving 5-HT1 receptors in cortex, hippocampus, and amygdala have been shown to exist in AD by ligand binding studies.15,23,25,3032 However, loss of 5-HT1 receptors is also age-related in people with AD.23 The 5-HT1A selective agonist 8-hydroxy-2-(di-n-propylamino) tetralin (abbreviated 8-OH-DPAT) has been used to suggest that 5-HT1 decreases are in the 5-HT1A receptor subtype.31,32 Given the actions of 5-HT1A, alterations in this subtype may be highly relevant to BPSD in AD.
View this table:
[in this window]
[in a new window]
|
TABLE 1. Potential role for serotonin (5-HT) receptor subtypes in behavioral and psychological symptoms of dementia
|
There are three subtypes of 5-HT2 receptors, all of which have some involvement in behaviors (Table 1). 5-HT2A is located in the cortex and caudate and is involved in anxiety (as well as vasoconstriction and migraine); 5-HT2B is a postsynaptic receptor found in the limbic system and hypothalamus that has been associated with depression, sleep, and hallucinations; and 5-HT2C is a postsynaptic receptor that has been associated with anxiety, depression, learning, and psychosis (as well as appetite and aversion).12 Deficits involving 5-HT2 receptors in AD frontal and temporal cortex, cingulate, hippocampus, and amygdala have been demonstrated with ligand binding studies,23,30,33,34 autoradiographic studies,35 and positron emission tomography (PET) studies.36 Loss of 5-HT2 receptors is much greater than loss of 5-HT1 receptors in AD23,34,37 and is not seen in other dementing illnesses.11,34 One study, which found 5-HT2 receptor binding was unchanged in the frontal cortex and hippocampus, may have been confounded by use of psychotropic medications at the time of death.38 Given the magnitude of the deficit in 5-HT2 receptors and the involvement of these receptors in behavior, this serotonergic system may also be important in BPSD associated with AD. Polymorphic variations have been identified for 5-HT2A and 5-HT2B receptors that may be risk factors for BPSD such as visual hallucinations.39
Relationships between the remaining 5-HT receptors and behavioral disorders seen in AD are not as well supported. The 5-HT3 receptor is postsynaptic and found in the cortex, midbrain, limbic system, hippocampus, and vomiting centers.12 It has very little homology with other 5-HT receptors and uses a difference second messenger system.12,29 Behaviors modulated by 5-HT3 are thought to include anxiety and psychosis,12 and thus this receptor subtype has a potential role to play. However, loss of 5-HT3 receptors was not demonstrated in the amygdala and hippocampus of AD patients compared with age-matched control subjects.40 The 5-HT4 receptor is found postsynaptically in the hippocampus, frontal cortex, limbic system, superior colliculi, and basal ganglia and has been implicated in learning, cognition, emotion, anxiety and sleep.12 Alterations in this subtype have not yet been identified in AD. The functions and locations of 5-HT5, 5-HT6, and 5-HT7 remain unclear, and their dysregulation in AD and related dementia has not been studied (Table 1). Receptor-specific ligands are currently being developed for 5-HT5,6,7.29
Preliminary studies show that AD may also be associated with changes in 5-HT signal transduction. Alterations in 5-HT1A receptor coupling to guanine nucleotide binding (G) proteins have been demonstrated in the superior frontal cortex,41 although another group found coupling to be preserved in the cerebral cortex.42 Uncoupling of phosphoinositide (PPI) hydrolysis from 5-HT stimulation in frontal cortex tissue of AD brains compared with age-matched controls has also been demonstrated.43 Study of second messenger systems is important because uncoupling implies that neurotransmitter replacement strategies would be ineffective.
Disruptions in serotonergic neurotransmission have also been studied in other dementing illnesses. Serotonin binding was reduced by 50% in the putamen of patients with the multi-infarct type of vascular dementia (VaD).44 Serotonin deficits were also found in a non multi-infarct category of VaD in cortical and subcortical gray matter.45 However, a more recent study was unable to demonstrate any serotonin deficits.46 Radioligand binding showed an intact brain serotonin system both presynaptically (uptake with [3H]paroxetine) and postsynaptically (5-HT1A with [3H]8-OH-DPAT and 5-HT2 with [3H]ketanserin) in the frontal cortex, temporal cortex, and caudate nucleus (presynaptic only) in vascular dementia.46 As expected, serotonin deficits may depend upon the subtype of VaD and the extent and location of lesions. There is also preliminary evidence for serotonergic changes in Pick's disease47 and Lewy body dementia.48,49
In Vivo Studies Examining Linkage With BPSD
Neuroendocrine studies have been performed in AD patients that support the role of 5-HT in behavior control. In one study using the serotonin agonist m-chlorophenylpiperazine (m-CPP), patients with AD (n=12) showed increased behavioral responsivity (psychomotor activation, restlessness, and perceptual abnormalities) compared with 10 age-matched control subjects.50,51 This may reflect damage to the serotonergic system with subsequent up-regulation of the remaining postsynaptic receptors.11 Unexpectedly, the neuroendocrine response to m-CPP did not differ between the two groups. However, since m-CPP binds to 5-HT1A, 5-HT2, and 5-HT3, as well as 1-, 2-, and ß-adrenergic sites, dopaminergic and cholinergic sites, discrepancies between the neuroendocrine response and the behavioral response of m-CPP may be due to actions on nonserotonergic neuronal systems. Although this study indicates a possible link between 5-HT and behavior, unfortunately there was no assessment for presence of BPSD in these patients. McLoughlin et al.52 performed a neuroendocrine study that confirmed the presence of a 5-HT hyperresponsivity in AD. Unfortunately, no attempt was made to relate this to the presence or absence of behavioral disturbances.
The relationship between decreased serotonin and specific behaviors in patients with AD and other dementing disorders has been studied (Table 2). Postmortem studies have found decreased 5-HT levels in AD patients with a history of psychotic behaviors, compared with nonpsychotic AD patients, in some areas of the brain.53 However, another study looking at the temporal cortex and both psychotic and nonpsychotic symptoms in patients with AD was unable to confirm these findings.54 Chen et al.26 found that concentrations of serotonin in the frontal cortex and 5-HIAA in the temporal cortex were significantly lower in patients on chronic neuroleptic treatment compared with patients not receiving neuroleptics. Because previous studies did not control for use of neuroleptics, it is unclear whether decreased serotonin is related to psychotic behaviors or to treatment with neuroleptics in AD. In Lewy body dementia, there is some evidence that preservation of 5-HT2 receptors in the temporal cortex may be associated with hallucinations.48
View this table:
[in this window]
[in a new window]
|
TABLE 2. Summary of literature linking serotonergic abnormalities with behavioral and psychological symptoms of dementia in Alzheimer's disease patients
|
Serotonin has also been related to nonpsychotic BPSD in both postmortem and clinical studies. A clinical study using CSF levels of the 5-HT metabolite 5-HIAA found that levels of 5-HIAA in the CSF were positively correlated with anxiety and fear/panic.55 Decreased cortical levels of 5-HT were found post mortem in patients with a history of aggression compared with nonagitated patients,19 whereas normal numbers of 5-HT2 receptors were found in nonaggressive patients.56 Clinical studies looking at putative peripheral markers of serotonergic activity have also been performed. Schneider et al.57 evaluated the relationship between binding to the platelet serotonin transporter system (a marker of serotonergic activity) and BPSD in patients with AD. The BPSD were predominantly agitation and delusions. The agitated/delusional groups showed significantly lower peripheral serotonergic activity than AD subjects without BPSD.57 However, a previous report by Suranyi-Cadotte et al.58 found no differences in the density of platelet serotonin binding sites between 9 AD patients and 11 age-appropriate controls when subjects with depressive symptoms had been excluded. Although that study58 demonstrated a link with depression, a recent postmortem study was unable to confirm this link.27 Because depressed patients were not explicitly excluded in the Schneider et al. study57 and the nonagitated group had apathy, it is unclear whether the association with agitation and delusions can be partially or wholly attributed to comorbid depression in some of these patients. Furthermore, platelet 5-HT2 function may not be adequate as a peripheral marker for the central serotonergic system. For example, a negative correlation has been found between CSF 5-HIAA and receptor indexes in other neuropsychiatric patients.59
Recently, two neuroendocrine studies found that agitated aggressive patients had an increased response to the 5-HT releasing agent fenfluramine compared with nonagitated aggressive patients.60,61 Both neuroendocrine BPSD studies excluded AD patients with significant depression. One study also found a significant gender effect.62 Gender has not been controlled for in other studies looking at the serotonergic system and may be an important contributor to variations in the serotonergic system throughout the life span.
In summary, in vitro and in vivo studies provide inconsistent evidence to link serotonin dysfunction with psychotic symptoms and with anxiety and depression in AD. Failure to control for concomitant medication use and comorbidity are largely responsible for these inconsistencies. Current clinical evidence does support earlier work by Palmer et al.19 and Procter et al.56 linking aggressive agitation with the serotonergic system.
Evidence From Pharmacotherapy Studies
The neurochemical and neurophysiological evidence that the serotonergic system may contribute to BPSD has been supported by a number of studies involving pharmacologic agents specific for the serotonergic system. Clinical manipulation of the serotonergic system through pharmacotherapy provides only indirect evidence of the involvement of that system with the target behavior. However, because the goal of understanding BPSD is to improve treatment, this information is worth briefly reviewing. (More extensive reviews are available.1,63) Considering there are 15 different serotonergic receptor subtypes, the number of serotonergic selective medications is limited. Agents reviewed here include the selective serotonin reuptake inhibitors (SSRIs), atypical antipsychotics, buspirone, ondansetron, and the serotonin and noradrenaline reuptake inhibitors (SNRIs).
The SSRIs alaproclate,64 zimeldine,65 fluoxetine,66,67 citalopram,68 fluvoxamine,69 paroxetine,70 and sertraline71 have been given to patients with BPSD in randomized controlled trials. Many patients have also been treated with open-label SSRIs, with positive results. For example, in an open-label study by Pollock et al.,72 9 of 16 patients receiving citalopram showed significant improvement in disinhibition, agitation, hostility, and suspicion. However, when SSRIs have been administered under double-blind, placebo-controlled conditions, results have been equivocal with respect to efficacy for BPSD. Randomized controlled trials in AD patients with alaproclate64 and fluoxetine66 have shown little efficacy for the SSRIs in treating BPSD, except for depression.67 Placebo-controlled trials with mixed dementia populations have shown nonsignificant improvement in BPSD such as irritability, anxiety, fear/panic, mood, and restlessness with fluvoxamine,69 and significant improvements in depression with paroxetine.70 Fluoxetine was reported to slightly improve disinhibition, depressive symptoms, carbohydrate craving, and compulsions in about one-half of 11 patients with frontotemporal dementia.73 The strongest efficacy was demonstrated by citalopram, which, among the SSRIs, has the greatest in vitro selectivity ratio for the serotonergic versus the noradrenergic system.74 Nyth and Gottfries68 administered citalopram for 4 weeks to patients with mild to moderate AD (n=65) or multi-infarct dementia (n=24) in a multicenter, placebo-controlled, parallel group study. In AD patients, significant improvements were noted on a geriatric rating scale in emotional bluntness and in all six BPSD (restlessness, confusion, irritability, anxiety, fear/panic, and depressed mood) for baseline versus citalopram scores. Irritability and depression improved significantly for citalopram versus placebo groups. There was no significant treatment effect in the multi-infarct group. The patients in this study had very mild BPSD at baseline, with mean scores of less than 2 out of 6 on each symptom.
The atypical antipsychotics (e.g., risperidone, clozapine, olanzapine) are being used more frequently in patients with BPSD because of their efficacy in treating psychotic symptoms and their favorable side effect profiles compared with their typical antipsychotic counterparts (i.e., reduced extrapyramidal effects). Most of the new antipsychotics have strong antagonistic affinity for the 5-HT2 receptor. Because the hyperresponsivity of the serotonergic system noted in agitated and aggressive AD patients60,61 suggests up-regulation of 5-HT2 receptors, the atypicals may be of benefit in the management of BPSD. To date, risperidone, clozapine, olanzapine, and quetiapine have been assessed to treat behaviors related to dementia.
Risperidone, in a total of 534 patients from case reports,7581 case series,82,83 retrospective chart reviews,8487 and open-label cohort studies,8890 was reported to produce significant reductions in behaviors such as psychosis and agitation in patients with various types of dementia. Further evidence of risperidone efficacy for BPSD was reported in two recent randomized controlled trials. Katz et al.91 reported on 625 patients, diagnosed with AD and/or vascular dementia, who received either placebo or 0.5 mg, 1 mg, or 2 mg of risperidone for 12 weeks. The authors found a dose response relationship with risperidone for reducing all behaviors and aggressiveness, and psychotic symptoms were significantly reduced with 1 mg, but not 2 mg, of risperidone compared with placebo. De Deyn et al.,92 in a 12-week trial of patients with AD and/or vascular dementia, found risperidone to significantly reduce aggressiveness compared with haloperidol and placebo.
Clozapine has not been evaluated for the treatment of BPSD in any randomized controlled trials. In an open-label dechallenge and rechallenge case series by Oberholzer et al.,93 clozapine was found to significantly improve "antisocial behavior," "social competence," "social interest," and "irritability" in 7 patients with AD. In two retrospective chart reviews,94,95 clozapine (doses of 100300 mg per day) was reported to have minimal to moderate effects on behaviors (e.g., psychosis, agitation) in 5 patients with a variety of dementia diagnoses. Frankenburg et al.96 reported a marked response (reduced hallucinations and verbal aggression) to low-dose clozapine (i.e., 18.5 mg/day) in a patient with parkinsonian dementia. However, 2 other patients with unspecified dementia were reported to have no changes or minimal change in behaviors with clozapine.96 Finally, in a case report of 4 patients at varied stages of dementia, Pitner et al.97 reported marked improvement in psychosis and other behaviors in 2 patients, and worsening of behavioral symptomatology in the remaining 2 patients.
In a large open-label trial of olanzapine, Kinon et al.98 reported that patients diagnosed with AD and/or vascular dementia showed significant improvements in psychosis and/or agitation. A recent randomized controlled trial by Street et al.99 reported olanzapine (5 mg and 10 mg doses) to significantly improve symptoms of psychosis and other behaviors compared with placebo in 206 AD patients over a 6-week period. The authors also reported that fewer patients without psychosis treated with olanzapine developed delusions and/or hallucinations by the end of the study compared with those receiving a placebo. Walker et al.100 reported that 6 of 8 patients diagnosed with Lewy body dementia responded to open-label olanzapine with significant reductions in BPSD.
Quetiapine, in a single published open-label study by McManus et al.,101 was found to significantly improve overall BPSD and psychotic symptoms in 106 patients with a variety of organic diseases over a 12-week period.
Although the atypical antipsychotics appear to improve BPSD, it is unclear whether this result is secondary to serotonergic or dopaminergic antagonism, since the typical antipsychotics (e.g., haloperidol, perphenazine, thioridazine) have also been demonstrated to have significant benefit.102 Future research might help to determine if typical and atypical antipsychotics have different patterns of symptom improvement.
Buspirone (a 5-HT1A partial agonist), in two randomized controlled double-blind trials, was compared with placebo10 and haloperidol103 to determine efficacy in treating BPSD. There was no significant overall reduction in behavioral symptoms with buspirone compared with placebo in the small number of patients tested (N=10). However, in the other study, buspirone was as effective as haloperidol in treating 26 patients with probable AD and superior to haloperidol in decreasing anxiety and tension.103 This latter study did not have a placebo control. Thus, the evidence for the efficacy of buspirone for BPSD is not strong.
Ondansetron (a 5-HT3 selective antagonist), used primarily for the management of chemotherapy-induced nausea and vomiting, has been used in patients with Parkinson's disease accompanied by dementia.104,105 Both open-label trials showed some benefit in alleviating visual hallucinations and delusions. Further evidence linking ondansetron to a reduction in behavioral problems awaits controlled clinical trials.
The SNRIs trazodone and clomipramine have also been tested for BPSD. Trazodone, in a number of open-label studies, was found to improve behavioral symptoms in 80% of demented elderly patients enrolled.106112 In a double-blind trial, trazodone showed a small but significant reduction in behavioral symptoms compared with buspirone and placebo.10 In another double-blind study controlled with haloperidol, trends suggested that specific symptoms, including aggression, may respond preferentially to trazodone.113 Clomipramine, another SNRI, produced significant overall improvement in depression in a small double-blind crossover study of AD patients.114 Although evidence to date shows some benefit with SNRIs, these drugs have more affinity for the noradrenaline than the serotonergic receptors and cannot be considered serotonin-selective medications.
In summary, the evidence supporting the efficacy of SSRIs, atypical antipsychotics, 5-HT agonists and antagonists, and SNRIs in the management of BPSD is as yet inconclusive. However, it must be noted that, as with other trials for BPSD,102 methodological issues with these studies included failure to blind or control,7173,7590,9398,100,101,106111,115130 insufficient power,10,66,73,9497,100,101,106111,113 inclusion of a heterogeneous population (patients had varying etiologies for their dementias or were at varying stages of the illness or had different subtypes of BPSD),64,6870,7779,8183,85,86,88,89,9198,113,117,129,130 and failure to use standardized and validated scales to identify and track target behaviors.64,66,77,78,82,85,86,88,9496,103,113115,119,120,122124,130 Furthermore, improper dosing and duration of treatment could contribute to some negative studies, since the effective dose and duration for treatment of different BPSD are unknown. Thus far, there is some evidence to support the use of serotonergic medications. The preliminary literature on SSRIs supports the use of the most serotonin-selective of these medications in patients with Alzheimer-type dementia, and this is corroborated by positive results with atypical antipsychotics. However, there are no predictors of response.

|
OTHER NEUROTRANSMITTER SYSTEMS
|
Acetylcholine (ACh)
Loss of cholinergic neurons is an early and consistent finding in AD and is thought to be essential to the pathophysiology. Numerous studies have demonstrated profound changes in the cholinergic system in AD, including deficits in the major cholinergic system arising in the basal forebrain and projecting to the cortex;131134 decreases in the cholinergic markers choline acetyltransferase (ChAT) and acetylcholinesterase (AChE) in the cortex, particularly the temporal cortex;24,38,135137 significant losses of neurons in the nucleus basalis of Meynert;131,138 and reductions in the muscarinic type 2 (M2) presynaptic receptor density.132,137 Cholinergic neurons arise in the basal forebrain (nucleus basalis of Meynert, diagonal band of Broca, and medial septum) and innervate widespread areas of the cortex, including the hippocampus and a variety of subcortical structures. Therefore, alterations in the central cholinergic system can be expected to cause significant disruptions in the functioning of the neocortex.
Although the role of the central cholinergic system in cognition is well recognized,139145 there is only preliminary evidence suggesting that this neurotransmitter plays an important role in the noncognitive disorders associated with dementia. The cholinergic hypothesis of noncognitive symptoms of AD put forth by Cummings and Back146 suggests that cholinergic deficits may contribute to symptoms such as psychosis-agitation, apathy-indifference, disinhibition, and aberrant motor behavior. Since virtually all AD patients have loss of cholinergic neurons but some do not have BPSD, other neurotransmitters may be involved.
Serotonin and acetylcholine interact extensively in the human brain. For example, 5-HT inhibits release of ACh from cortical and hippocampal cholinergic nerve terminals, possibly via 5-HT1B receptors in the hippocampus.11 The 5-HT3 receptors may also inhibit the release of ACh, whereas 5-HT1A receptors may mediate an increase in ACh release.29 Thus, disruptions in 5-HT have the potential to influence an already compensated cholinergic system.
A single postmortem study by Procter et al.56 examined 5-HT receptor binding (with [3H]ketanserin) and ChAT activity in 17 dementia patients with or without aggressive symptomatology (4 aggressive patients, 13 nonaggressive patients) compared with 18 healthy elderly control subjects. Except for the superior parietal lobe, [3H]ketanserin binding was decreased in 12 cortical areas in aggressive compared with nonaggressive patients. However, ChAT activity throughout the cortex was similar in the two patient groups and was significantly lower in these groups compared with healthy control subjects. Thus, current evidence supports 5-HT deficits being more closely related than acetylcholine to aggression in AD.
Clinical experience with cholinergic and anticholinergic medications provides an indirect link between BPSD and the cholinergic system. In AD, cholinergic therapies have been reported to affect psychosis and agitation, apathy/indifference, disinhibition, and aberrant motor behavior.146 Postmortem studies provide some evidence that neuronal losses in the nucleus basalis, and also decreases in cholinergic transmission, are correlated with psychotic symptoms in Lewy body dementia.147,148 Choline acetyltransferase was also shown to be decreased in Lewy body dementia patients with hallucinations.49 Overall, there is some preliminary evidence linking psychotic behavioral changes with the cholinergic system, especially the balance between this system and the monoamines.49 In summary, evidence supports the suggestion that the addition of serotonergic and cholinergic deficits may be responsible for aggression and psychotic symptoms.
The question of whether serotonergic changes are secondary to cholinergic changes or independent of them cannot be answered as yet. To date, there are no longitudinal studies identifying parallel progression in the degeneration of the 5-HT and cholinergic systems. There are some studies that have identified concurrent deficits25,149152 or no change15,153,154 in 5-HT (e.g., 5-HT and 5-HIAA concentrations, [3H]5-HT binding) with respect to loss of cholinergic markers (e.g., ChAT, ACh synthesis) in AD patients compared with control subjects. These findings suggest that serotonergic deficits are either a variable component of AD or occur only in advanced AD.
Noradrenaline (NA)
The central serotonergic and noradrenergic systems interact in many areas. Serotonin is a co-transmitter with NA, and uptake of 5-HT and NA can be accomplished by either 5-HT or NA neurons.155 The serotonergic system also inhibits the release of NA via 5-HT1 receptors.29 Thus, there is reason to suspect that dysfunction in the serotonergic system will be accompanied by changes in the NA system. Animal studies have shown that NA neurons from the locus ceruleus (LC) are involved in behaviors such as the sleep-wake cycle, level of vigilance, and emotion.156,157 Dysfunction in the NA system may also directly influence BPSD.
Postmortem studies suggest AD is associated with loss of neurons in the locus ceruleus,134,158 the major source of NA in the brain, and with decreased levels of NA in many brain areas innervated by the LC.17,20,24,27,53,159,160 Loss of NA neurons is correlated with the severity of dementia.161 In contrast, levels of the NA metabolite 4-methoxy-5-hydroxy-phenylglycol (MHPG) are increased,162165 indicating that NA metabolism is increased. This increased MHPG/NA ratio likely reflects compensatory increases in the activity of remaining LC neurons.27 These changes are not reflected in the CSF of patients.160,166 CSF levels of MHPG are thought to increase with advancing disease,167,168 complicating interpretation of CSF studies.
Evidence supports a link between depressive symptoms in AD patients and NA dysfunction. The evidence includes neuronal losses in the LC,156,169171 lower levels of cortical NA,172 and increased NA dysfunction.173 However, a well-designed postmortem study with prospective patient evaluations before death was unable to confirm this relationship.27
Preservation of NA may also be associated with BPSD. A postmortem study in AD patients indicated that increased levels of NA,53 comparable to those found in elderly control subjects,172 were associated with psychotic behaviors, but this relationship was not reflected in CSF MHPG concentrations.168 There is evidence that restlessness is weakly related to CSF levels of MHPG in patients with AD,55 although agitation, in another study, was not.168 A recent study also showed markedly increased concentrations of 2 receptors (by 70%) and ß1 and ß2 adrenergic receptors (by 25%) in the cerebellar cortex of aggressive patients compared with nonaggressive patients with similar levels of cognitive deficit. Levels of 2 receptors in aggressive patients were slightly above those in the healthy elderly control subjects, whereas increases in ß receptors were higher than in both nonaggressive AD patients and control subjects.174 Thus, aggressive behavior may be associated with cerebellar noradrenergic preservation.
Dopamine (DA)
Serotonergic neurons interact closely with dopaminergic neurons. For example, serotonergic neurons from the raphe nuclei are known to synapse with dopaminergic neurons and to control dopamine release in the midbrain, striatum, and nucleus accumbens.175 Serotonergic neurons may either inhibit (via 5-HT1A) or increase release of DA.29 Thus, loss of serotonergic neurons will affect the DA system.
The dopaminergic system has been implicated in depression, agitation, and psychotic behaviors in nondemented patients, and thus this system has the potential to directly influence BPSD. Although disruptions in the dopaminergic system have been demonstrated post mortem in AD patients compared with control subjects,20,159 DA is less affected than other biogenic amines.17,27 Dopaminergic disruptions may primarily involve presynaptic disruptions in the DA transporter and postsynaptic loss of striatal dopamine type 2 (D2) receptors.176 Furthermore, loss of striatal D2 receptors may be more common in the 30% of AD patients who also show extrapyramidal features.176,177 Interestingly, a neuroimaging study demonstrated that disruptions in DA metabolism became increasingly severe as the cognitive impairment progressed.178 As a result, level of cognitive impairment will be a confounder for any studies attempting to relate DA dysfunction to behavior.
Studies in patients with AD that controlled for level of cognitive impairment have not been able to demonstrate the expected links between psychotic behaviors and abnormalities in the DA system53,179,180 or depression and the DA system.27 The same may not be true in other dementias. In Lewy body dementia, dopamine metabolites were significantly decreased in nonhallucinating patients in conjunction with serotonergic abnormalities (i.e., decreased serotonergic 5-HT2 receptor binding and decreased 5-HT metabolites).49 When combined with the finding that ChAT was decreased in hallucinating patients, these results suggest that an imbalance between monoaminergic and cholinergic transmitters is involved in hallucinations seen in Lewy body dementia.
There is some evidence that agitated and aggressive behaviors may be associated with a relative preservation of DA function in AD patients. A postmortem study181 demonstrated that preservation of substantia nigra neurons was related to aggression and interpersonal violence. Studies performed in vivo have supported this. Lopez et al.182 found that whereas aggressive patients showed similar CSF homovanillic acid (HVA) concentrations to their age-matched controls, nonaggressive patients showed decreased levels. Similarly, Sweet et al.183 found that improvement in hostility during perphenazine treatment correlated with change in plasma HVA levels in 21 patients (89% AD), further supporting the link between aggression and DA. Thus, DA preservation may be associated with the presence of aggressive behaviors. It is plausible that cholinergic and serotonergic dysfunction may also be present and necessary for this behavior. There are no studies answering this question as yet.
Gamma-aminobutyric Acid (GABA)
Serotonin is a co-transmitter with GABA, and GABA agonists can alter the function of several 5-HT receptors.184 GABA is the primary inhibitory neurotransmitter in the CNS.185 It is a local inhibitory interneuron for other neurotransmitters that are key in controlling behavior, including serotonin and dopamine.186 In addition to these local circuit actions, GABAergic neurons project from the striatum to the lateral globus pallidus and from the cerebellar inferior olivary nucleus to the vestibular nucleus.186 Thus, GABA is well positioned to influence many psychobiological functions such as behavior through interactions with serotonin.
Involvement of the neurotransmitter GABA has been shown in behaviors such as aggression, increased GABA being associated with decreased aggression.187 Many lines of evidence have been pursued to establish the presence or absence of a GABAergic abnormality in AD, including postmortem studies of neurotransmitter concentrations150,188198 and binding,18,35,37,199207 antemortem studies of CSF levels of neurotransmitter and metabolites,208216 and PET217,218 and SPECT219 studies. Each of these avenues has generated inconclusive results; many studies showed decreased GABA,18,37,189195,197,199201,203,205,206,209,210,215 but some studies showed no significant changes.150,188,196,207,208,211214,216 Although the discrepant results could reflect the varying methodologies used, each with inherent limitations, these results may also reflect reality. This evidence suggests that GABA disruption is not linked to cognitive degeneration to the same extent as cholinergic dysfunction. GABA dysfunction is a variable component of AD, and, as such, may be linked to the presence of certain behavioral disorders. As yet, there are no clinical studies linking changes in the GABA system with particular behaviors seen in AD.
Neuropeptides
Neurotrophic factors, such as brain-derived neurotrophic factor, somatostatin, and neuropeptide Y, have all been reported to be co-localized within serotonergic neurons in the CNS in both animals and humans.220222 Thus, loss of serotonergic neurons implies at least some disruption in these neuropeptides. Although a number of authors have studied the relationship between neuropeptide abnormalities and AD pathology (see review by Gottfries et al.223), very little work has focused on the putative effects of altered neuropeptides and BPSD. Minthon and colleagues, in two separate studies,224,225 attempted to correlate somatostatin and neuropeptide Y levels in the CSF with BPSD. The authors reported a significant negative correlation of neuropeptide Y levels with clinical features of AD, including anxiousness, restlessness-agitation, and irritability. The authors failed to find a significant correlation between somatostatin and specific behaviors related to dementia. Whether a reduction in neuropeptide Y correlates with serotonergic deficits in patients with BPSD has yet to be determined. However, given that many of these neurotrophic factors have been associated with behaviors (e.g., aggression, anxiety, depression) in nondemented populations,226 it is plausible that reductions in specific neuropeptides concurrently with serotonin can contribute behavioral phenotypes in dementia.

|
CONCLUSIONS
|
The serotonergic system is complex, and much remains to be learned about its function in normal states and dementia. Current evidence suggests that AD and related dementias are associated with changes in serotonergic neurotransmission. In addition, many neurotransmitters and neuropeptides that interact extensively with the serotonergic system are dysfunctional in these dementias, but it is unknown whether this is secondary to cholinergic or serotonergic changes. Attempts to link these changes with specific behavioral and psychological symptoms of dementia have been made. Although there is no clear evidence linking alterations of the serotonergic system with a single behavior or cluster of behaviors, it seems relatively certain that such alterations do contribute to noncognitive symptoms. The most supportive evidence is for a contribution of serotonergic system changes to aggressive behaviors and psychotic behaviors.
Much of the evidence to date consists of postmortem studies. For these studies to be rigorous, it is necessary to have prospective, antemortem data collection, including data on drug use and on well-defined behaviors. Neuroimaging techniques (PET, SPECT), for example, represent novel approaches to determining neurotransmitter changes in antemortem brain. Radiolabeled agonists or antagonists specific for each receptor system can help to elucidate altered binding potential of certain receptor subtypes. For example, the 5-HT1Aspecific antagonist [11C]WAY-100635 has been used successfully to determine 5-HT1A binding potential in patients with depression.227,228 In the case of dementia, these radiolabeled tracers may lead to the identification of specific receptor deficits that may correlate with noncognitive aspects of the diseasewhich in turn may lead to receptor-specific drug therapies. Other potential antemortem investigations of central neurotransmitter function include neuroendocrine challenge studies to predict response to drug treatment (e.g., prolactin response to serotonergic challenge, as mentioned previously), prospective intervention studies of multiple drug therapies to manage the spectrum of behaviors (e.g., neuroleptics and antidepressants), and longitudinal studies with drug prophylaxis in early dementia (e.g., SSRI administration, combined with longitudinal scanning, in dementia patients with a family history of mood disorders).
Unfortunately, much of the work done to date suffers from lack of standardized definitions for behaviors. Patients were frequently defined clinically by the presence of a given behavior. Because many of the behaviors overlap, selecting on the basis of the presence of a single behavior will not provide homogeneity in the patient population. A neurotransmitter-based approach will offer information that can be more closely linked with treatment options. Another limitation is that in postmortem studies linking behaviors to neurotransmitter abnormalities, the behaviors may or may not be expressed at the time of death. Because behavioral changes associated with degenerative neurologic changes are likely to be predominantly a state rather than a trait, it is important that the behaviors are expressed near the time of death. Also, both neurotransmitter disruptions and behaviors are known to change with level of cognitive impairment.229 Findings of changes in behaviors may arise both from actual changes in the behaviors and difficulty in measuring some behaviors as the level of cognitive impairment increases (e.g., auditory hallucinations as the patient becomes less verbal). Thus, it is important to control for level of cognitive impairment as well as to study BPSD through the course of the disease. This latter goal will be more easily achieved with the application of in vivo measures of receptor function; these can include peripheral markers such as neuroendocrine challenges and CSF levels, as well as functional imaging when feasible.
Studying the neurochemical etiology of BPSD presents many challenges. The activity of a single neurotransmitter such as serotonin cannot be separated from those of other neurotransmitters and neuropeptides. Hence, it is essential that future work focus on identifying concurrent neurotransmitter and neuropeptide changes through the progression of the disease. Careful study of clusters of BPSD and their corresponding imbalances may eventually provide enough information to make possible evidence-based, patient-specific treatments for behavioral disorders in dementia.

|
ACKNOWLEDGMENTS
|
Dr. Lanctôt's work was supported by the Alzheimer Society of Canada Research Program and is currently partially supported by the Kunin-Lunenfeld Applied Research Unit of Baycrest Centre for Geriatric Care, Toronto, Ontario.

|
REFERENCES
|
-
Stoppe G, Brandt CA, Staedt JH: Behavioural problems associated with dementia: the role of newer antipsychotics. Drugs Aging 1999; 14:4154[CrossRef][Medline]
-
Mega MS, Cummings JL, Fiorello T, et al: The spectrum of behavioral changes in Alzheimer's disease. Neurology 1996; 46:130135[Abstract/Free Full Text]
-
Hope T, Keene J, Fairburn CJ, et al: Natural history of behavioural changes and psychiatric symptoms in Alzheimer's disease: a longitudinal study. Br J Psychiatry 1999; 174:3944[Abstract/Free Full Text]
-
Reisberg B, Borenstein J, Salob SP, et al: Behavioral symptoms in Alzheimer's disease: phenomenology and treatment. J Clin Psychiatry 1987; 48(suppl 5):915
-
Nagaratnam N, Lewis-Jones M, Scott D, et al: Behavioral and psychiatric manifestations in dementia patients in a community: caregiver burden and outcome. Alzheimer Dis Assoc Disord 1998; 12:330334[Medline]
-
Cohen CA, Gold DP, Shulman KI, et al: Factors determining the decision to institutionalize dementing individuals: a prospective study. Gerontologist 1993; 33:714720[Abstract]
-
Merriam AE, Aronson MK, Gaston P, et al: The psychiatric symptoms of Alzheimer's disease. J Am Geriatr Soc 1988; 36:712[Medline]
-
Walsh JS, Welch HG, Larson EB: Survival of outpatients with Alzheimer-type dementia. Ann Intern Med 1990; 113:429434
-
Rubin EH, Drevets WC, Burke WJ: The nature of psychotic symptoms in senile dementia of the Alzheimer type. J Geriatr Psychiatry Neurol 1988; 1:1620
-
Lawlor BA, Radcliffe J, Molchan SE, et al: A pilot placebo-controlled study of trazodone and buspirone in Alzheimer's disease. Int J Geriatr Psychiatry 1994; 9:5559[CrossRef]
-
Siever LJ, Kahn RS, Lawlor BA, et al: Critical issues in defining the role of serotonin in psychiatric disorders. Pharmacol Rev 1991; 43:509525[Medline]
-
Dubovsky SL, Thomas M: Serotonergic mechanisms and current and future psychiatric practice. J Clin Psychiatry 1995; 56(suppl 2):3848
-
Fuller RW: Role of serotonin in therapy of depression and related disorders. J Clin Psychiatry 1991; 52(suppl 5):5257
-
McCormick DA, Williamson A: Convergence and divergence of neurotransmitter action in human cerebral cortex. Proc Natl Acad Sci USA 1989; 86:80988102[Abstract/Free Full Text]
-
Bowen DM, Allen SJ, Benton JS, et al: Biochemical assessment of serotonergic and cholinergic dysfunction and cerebral atrophy in Alzheimer's disease. J Neurochem 1983; 41:266272[Medline]
-
Adolffson R, Gottfries CG, Roos BE, et al: Changes in brain catecholamines in patients with dementia of Alzheimer type. Br J Psychiatry 1979; 135:216233[Abstract/Free Full Text]
-
Arai H, Kosaka K, Iiziku R: Changes of biogenic amines and their metabolites in postmortem brains from patients with Alzheimer-type dementia. J Neurochem 1984; 43:388393[CrossRef][Medline]
-
Hardy J, Cowburn R, Barton A, et al: A disorder of cortical GABAergic innervation in Alzheimer's disease. Neurosci Lett 1987; 73:192196[CrossRef][Medline]
-
Palmer AM, Stratman GC, Procter AW, et al: Possible neurotransmitter basis of behavioral changes in Alzheimer's disease. Ann Neurol 1988; 23:616620[CrossRef][Medline]
-
Nazarali AJ, Reynolds GP: Monoamine neurotransmitters and their metabolites in brain regions in Alzheimer's disease: a postmortem study. Cell Mol Neurobiol 1992; 12:581587[CrossRef][Medline]
-
Mann DMA, Yates PO: Serotonin nerve cells in Alzheimer's disease. J Neurol Neurosurg Psychiatry 1983; 46:9698
-
Yamamoto T, Hirano A: Nucleus raphe dorsalis in Alzheimer's disease: neurofibrillary tangles and loss of large neurons. Ann Neurol 1985; 17:5737[CrossRef][Medline]
-
Cross AJ, Crow TJ, Ferrier IN, et al: Serotonin receptor changes in dementia of the Alzheimer type. J Neurochem 1984; 43:15741581
-
Francis PT, Palmer AM, Sims NR, et al: Neurochemical studies of early onset Alzheimer's disease. N Engl J Med 1985; 313:711[Abstract]
-
Palmer AM, Francis PT, Benton JS, et al: Presynaptic serotonergic dysfunction in patients with Alzheimer's disease. J Neurochem 1987; 48:815[Medline]
-
Chen CPL-H, Alder JT, Bowen DM, et al: Presynaptic serotonergic markers in community-acquired cases of Alzheimer's disease: correlations with depression and neuroleptic medication. J Neurochem 1996; 66:15921598
-
Hoogendijk WJG, Feenstra MGP, Botterblom MHA, et al: Increased activity of surviving locus ceruleus neurons in Alzheimer's disease. Ann Neurol 1999; 45:8291[CrossRef][Medline]
-
Volicer L, Direnfeld LK, Freedman M, et al: Serotonin and 5-hydroxyindoleacetic acid in CSF: difference in Parkinson's disease and dementia of the Alzheimer's type. Arch Neurol 1985; 42:127129[Abstract]
-
Murphy DL, Andrews AM, Wichems CH, et al: Brain serotonin neurotransmission: an overview and update with an emphasis on serotonin subsystem heterogeneity, multiple receptors, interactions with other neurotransmitter systems, and consequent implications for understanding the actions of serotonergic drugs. J Clin Psychiatry 1998; 59(suppl 15):412
-
Crow TJ, Cross A, Cooper S, et al: Neurotransmitter receptors and monoamine metabolites in the brains of patients with Alzheimer-type dementia and depression, and suicides. Neuropharmacology 1984; 23:15611569
-
Middlemiss DN, Palmer AM, Edel N, et al: Binding of the novel serotonin agonist 8-hydroxy-2-(di-n-propylamino) tetralin in normal and Alzheimer brain. J Neurochem 1986; 46:993996[CrossRef][Medline]
-
Proctor AW, Middlemiss DN, Bowen DM: Selective loss of serotonin recognition sites in the parietal cortex in Alzheimer's disease. Int J Geriatr Psychiatry 1988; 3:3744
-
Brown GL, Ebert MH, Goyer PE, et al: Aggression, suicide, and serotonin: relationships to CSF amine metabolites. Am J Psychiatry 1982; 139:741746[Abstract/Free Full Text]
-
Perry EK, Perry RH, Candy JM, et al: Cortical serotonin-S2 receptor binding abnormalities in patients with Alzheimer's disease: comparisons with Parkinson's disease. Neurosci Lett 1984; 51:353357[CrossRef][Medline]
-
Jansen KL, Faull RL, Dragunow M, et al: Alzheimer's disease: changes in hippocampal N-methyl-D-aspartate, quisqualate, neurotensin, adenosine, benzodiazepine, serotonin and opioid receptors: an autoradiographic study. Neuroscience 1990; 39:61327[CrossRef][Medline]
-
Blin J, Baron JC, Dubois B, et al: Loss of brain 5-HT2 receptors in Alzheimer's disease: in vivo assessment with positron emission tomography and [18F] setoperone. Brain 1993; 166(pt 3):497510
-
Cross AJ, Crow TJ, Ferrier IN, et al: The selectivity of the reduction of serotonin S2 receptors in Alzheimer-type dementia. Neurobiol Aging 1986; 7:37[CrossRef][Medline]
-
Dewar D, Braham DI, McCulloch J:5-HT2 receptors in dementia of Alzheimer type: a quantitative autoradiographic study of frontal cortex and hippocampus. J Neural TransmParkinson's Disease and Dementia 1990; 2:129137
-
Holmes C, Arranz MJ, Powell JF, et al:5-HT2A and 5-HT2C receptor polymorphisms and psychopathology in late onset Alzheimer's disease. Hum Mol Genet 1998; 7:15071509
-
Barnes NM, Costall B, Naylor RJ, et al: Normal densities of 5-HT3 receptor recognition sites in Alzheimer's disease. Neuroreport 1990; 1:253254[Medline]
-
Weinstein D, Magnus
|