Role of Serotonin in the Behavioral and Psychological Symptoms of Dementia

Behavioral disorders have long been recognized as a common feature in Alzheimer's disease (AD) and related dementias. The nature and frequency of the behavioral and psychological symptoms of dementia (BPSD) have been outlined in numerous papers.^1,2 BPSD can include psychotic symptoms such as hallucinations and paranoid/delusional ideation, nonpsychotic symptoms such as aggression and wandering, and affective disturbances and anxieties/phobias.¹ BPSD occur throughout the course of dementia, irrespective of level of cognitive impairment.³ It has been estimated that up to 90% of patients display BPSD.^1,4 Frequently identified BPSD include delusions (10% to 73%), wandering (10% to 70%), aggression (18% to 65%), motor restlessness (21% to 60%), hallucinations (3% to 49%), and depression (30% to 40%).^1,2,5 These BPSD are distressing to the caregivers and are factors stimulating the decision to institutionalize.^5,6 Furthermore, the presence of BPSD may have a negative impact on the course of the disease.^7–9 Knowledge of the importance of BPSD for both caregivers and patients has stimulated research into their underlying pathological and neurochemical substrates.

Strong, consistent evidence supports the presence of significant disruptions in global serotonergic neurotransmission in dementia.¹⁰ Serotonergic neurons originating from the dorsal and median raphe nuclei innervate many structures in the cortex and limbic system.^11,12 These widespread projections enable the serotonergic system to regulate aggression, mood, feeding, sleep, temperature, sexual activity, and motor activity.^11,13 Therefore, alterations in the functioning of the central serotonergic system can be expected to have a clinically discernible impact on behavior.

The serotonergic system is heterogeneous in terms of structure and function, making it a complex system to study. Moreover, the dysfunction of the serotonergic system studied in isolation cannot be expected to explain changes in behavior, given that many neurotransmitters work in conjunction with each other. For example, although serotonin (5-hydroxytryptamine; 5-HT) generally plays an inhibitory role in the human cortex, overall excitability is mediated by acetylcholine, γ-aminobutyric acid (GABA), noradrenaline, histamine, and purines.¹⁴ There is some evidence that each of these neurotransmitters is altered in AD and related dementias,^15–20 and each has a role in controlling human behavior. Hence, optimum treatment efficacy may be achieved by correcting multiple neurotransmitters or the balance between neurotransmitters.

We will review evidence supporting neurochemical alterations and putative links with BPSD for the serotonergic system and four neurotransmitters that interact extensively with serotonin: acetylcholine, noradrenaline, dopamine, and GABA. For each section, the functional role of the neurotransmitter, the evidence supporting a disruption in AD, and any studies indicating a link with BPSD will be reviewed. Any evidence for other dementias will immediately follow. Neurotrophic factors also control behavior. The neuropeptides somatostatin, neuropeptide Y, and brain-derived neurotrophic factor also interact extensively with serotonin and will be briefly discussed.

Articles were retrieved from various sources, including electronic databases (MEDLINE, EMBASE) and cross-references from relevant articles. The following keywords were used: serotonin, acetylcholine, dopamine, norepinephrine, gamma-aminobutyric acid, GABA, neuropeptides, neurotrophic factors, neuropeptide Y, somatostatin, brain derived neurotrophic factor, Alzheimer's disease, dementia, behavior, behavioral disorders, delusions, hallucinations, agitation, aggression, apathy, disinhibition, depression, cognition, and non-cognitive.

SEROTONERGIC SYSTEM

The evidence supporting the role of serotonin disruption contributing to BPSD consists of biologic plausibility, demonstration of disruptions in dementia, studies linking disruptions with BPSD, and clinical evidence from serotonergic medications. Most of the studies have been done in patients with Alzheimer's disease (Figure 1). This evidence is reviewed below.

In Vitro Evidence of Serotonergic Disruption in Dementia

Neurochemical and neuropathological disruptions in the serotonergic system have been established in AD. Loss of neurons in the raphe nuclei has been demonstrated,^21,22 as has loss of serotonergic nerve terminals in the neocortex.¹⁵ Decreased concentrations of 5-HT and its major metabolite 5-hydroxyindoleacetic acid (5-HIAA) have been demonstrated in the central nervous system by use of postmortem brain studies, particularly in the temporal cortex^{16,17,19,23–27} and in the cerebrospinal fluid (CSF).²⁸ Although indirect, taken as a whole these data provide evidence that there are changes in the actual functional status of central serotonergic transmission. The presence of expected adaptive consequences due to serotonergic lesions, such as supersensitivity, tolerance, trophic changes, or other changes in 5-HT-modulated central functions,²⁹ remains to be elucidated.

The actions of 5-HT are mediated through at least seven major receptor classes that have differing placements in the synapse, utilize different second messenger systems, and have different locations in the brain.^12,29 5-HT_1A receptors are located both presynaptically (somatodendritic in the raphe nuclei) and postsynaptically (terminals in the limbic system, hypothalamus, hippocampus, frontal cortex, and brainstem) and can have contradictory actions on intracellular signaling.^12,29 This receptor subtype is thought to be involved with anxiety, depression, sexual behavior, and aggression¹² (Table 1), as well as appetite. The functions influenced by 5-HT_1Dα, 5-HT_1Dβ, 5-HT_1E, and 5-HT_1F are still being elucidated, and functions identified so far are not related to behavior.¹² Deficits involving 5-HT₁ receptors in cortex, hippocampus, and amygdala have been shown to exist in AD by ligand binding studies.^{15,23,25,30–32} However, loss of 5-HT₁ receptors is also age-related in people with AD.²³ The 5-HT_1A selective agonist 8-hydroxy-2-(di-n-propylamino) tetralin (abbreviated 8-OH-DPAT) has been used to suggest that 5-HT₁ decreases are in the 5-HT_1A receptor subtype.^31,32 Given the actions of 5-HT_1A, alterations in this subtype may be highly relevant to BPSD in AD.

There are three subtypes of 5-HT₂ receptors, all of which have some involvement in behaviors (Table 1). 5-HT_2A is located in the cortex and caudate and is involved in anxiety (as well as vasoconstriction and migraine); 5-HT_2B is a postsynaptic receptor found in the limbic system and hypothalamus that has been associated with depression, sleep, and hallucinations; and 5-HT_2C is a postsynaptic receptor that has been associated with anxiety, depression, learning, and psychosis (as well as appetite and aversion).¹² Deficits involving 5-HT₂ receptors in AD frontal and temporal cortex, cingulate, hippocampus, and amygdala have been demonstrated with ligand binding studies,^23,30,33,34 autoradiographic studies,³⁵ and positron emission tomography (PET) studies.³⁶ Loss of 5-HT₂ receptors is much greater than loss of 5-HT₁ receptors in AD^23,34,37 and is not seen in other dementing illnesses.^11,34 One study, which found 5-HT₂ receptor binding was unchanged in the frontal cortex and hippocampus, may have been confounded by use of psychotropic medications at the time of death.³⁸ Given the magnitude of the deficit in 5-HT₂ receptors and the involvement of these receptors in behavior, this serotonergic system may also be important in BPSD associated with AD. Polymorphic variations have been identified for 5-HT_2A and 5-HT_2B receptors that may be risk factors for BPSD such as visual hallucinations.³⁹

Relationships between the remaining 5-HT receptors and behavioral disorders seen in AD are not as well supported. The 5-HT₃ receptor is postsynaptic and found in the cortex, midbrain, limbic system, hippocampus, and vomiting centers.¹² It has very little homology with other 5-HT receptors and uses a difference second messenger system.^12,29 Behaviors modulated by 5-HT₃ are thought to include anxiety and psychosis,¹² and thus this receptor subtype has a potential role to play. However, loss of 5-HT₃ receptors was not demonstrated in the amygdala and hippocampus of AD patients compared with age-matched control subjects.⁴⁰ The 5-HT₄ receptor is found postsynaptically in the hippocampus, frontal cortex, limbic system, superior colliculi, and basal ganglia and has been implicated in learning, cognition, emotion, anxiety and sleep.¹² Alterations in this subtype have not yet been identified in AD. The functions and locations of 5-HT₅, 5-HT₆, and 5-HT₇ remain unclear, and their dysregulation in AD and related dementia has not been studied (Table 1). Receptor-specific ligands are currently being developed for 5-HT_5,6,7.²⁹

Preliminary studies show that AD may also be associated with changes in 5-HT signal transduction. Alterations in 5-HT_1A receptor coupling to guanine nucleotide binding (G) proteins have been demonstrated in the superior frontal cortex,⁴¹ although another group found coupling to be preserved in the cerebral cortex.⁴² Uncoupling of phosphoinositide (PPI) hydrolysis from 5-HT stimulation in frontal cortex tissue of AD brains compared with age-matched controls has also been demonstrated.⁴³ Study of second messenger systems is important because uncoupling implies that neurotransmitter replacement strategies would be ineffective.

Disruptions in serotonergic neurotransmission have also been studied in other dementing illnesses. Serotonin binding was reduced by 50% in the putamen of patients with the multi-infarct type of vascular dementia (VaD).⁴⁴ Serotonin deficits were also found in a non– multi-infarct category of VaD in cortical and subcortical gray matter.⁴⁵ However, a more recent study was unable to demonstrate any serotonin deficits.⁴⁶ Radioligand binding showed an intact brain serotonin system both presynaptically (uptake with [³H]paroxetine) and postsynaptically (5-HT_1A with [³H]8-OH-DPAT and 5-HT₂ with [³H]ketanserin) in the frontal cortex, temporal cortex, and caudate nucleus (presynaptic only) in vascular dementia.⁴⁶ As expected, serotonin deficits may depend upon the subtype of VaD and the extent and location of lesions. There is also preliminary evidence for serotonergic changes in Pick's disease⁴⁷ and Lewy body dementia.^48,49

In Vivo Studies Examining Linkage With BPSD

Neuroendocrine studies have been performed in AD patients that support the role of 5-HT in behavior control. In one study using the serotonin agonist m-chlorophenylpiperazine (m-CPP), patients with AD (n=12) showed increased behavioral responsivity (psychomotor activation, restlessness, and perceptual abnormalities) compared with 10 age-matched control subjects.^50,51 This may reflect damage to the serotonergic system with subsequent up-regulation of the remaining postsynaptic receptors.¹¹ Unexpectedly, the neuroendocrine response to m-CPP did not differ between the two groups. However, since m-CPP binds to 5-HT_1A, 5-HT₂, and 5-HT₃, as well as α1-, α2-, and β-adrenergic sites, dopaminergic and cholinergic sites, discrepancies between the neuroendocrine response and the behavioral response of m-CPP may be due to actions on nonserotonergic neuronal systems. Although this study indicates a possible link between 5-HT and behavior, unfortunately there was no assessment for presence of BPSD in these patients. McLoughlin et al.⁵² performed a neuroendocrine study that confirmed the presence of a 5-HT hyperresponsivity in AD. Unfortunately, no attempt was made to relate this to the presence or absence of behavioral disturbances.

The relationship between decreased serotonin and specific behaviors in patients with AD and other dementing disorders has been studied (Table 2). Postmortem studies have found decreased 5-HT levels in AD patients with a history of psychotic behaviors, compared with nonpsychotic AD patients, in some areas of the brain.⁵³ However, another study looking at the temporal cortex and both psychotic and nonpsychotic symptoms in patients with AD was unable to confirm these findings.⁵⁴ Chen et al.²⁶ found that concentrations of serotonin in the frontal cortex and 5-HIAA in the temporal cortex were significantly lower in patients on chronic neuroleptic treatment compared with patients not receiving neuroleptics. Because previous studies did not control for use of neuroleptics, it is unclear whether decreased serotonin is related to psychotic behaviors or to treatment with neuroleptics in AD. In Lewy body dementia, there is some evidence that preservation of 5-HT₂ receptors in the temporal cortex may be associated with hallucinations.⁴⁸

Serotonin has also been related to nonpsychotic BPSD in both postmortem and clinical studies. A clinical study using CSF levels of the 5-HT metabolite 5-HIAA found that levels of 5-HIAA in the CSF were positively correlated with anxiety and fear/panic.⁵⁵ Decreased cortical levels of 5-HT were found post mortem in patients with a history of aggression compared with nonagitated patients,¹⁹ whereas normal numbers of 5-HT₂ receptors were found in nonaggressive patients.⁵⁶ Clinical studies looking at putative peripheral markers of serotonergic activity have also been performed. Schneider et al.⁵⁷ evaluated the relationship between binding to the platelet serotonin transporter system (a marker of serotonergic activity) and BPSD in patients with AD. The BPSD were predominantly agitation and delusions. The agitated/delusional groups showed significantly lower peripheral serotonergic activity than AD subjects without BPSD.⁵⁷ However, a previous report by Suranyi-Cadotte et al.⁵⁸ found no differences in the density of platelet serotonin binding sites between 9 AD patients and 11 age-appropriate controls when subjects with depressive symptoms had been excluded. Although that study⁵⁸ demonstrated a link with depression, a recent postmortem study was unable to confirm this link.²⁷ Because depressed patients were not explicitly excluded in the Schneider et al. study⁵⁷ and the nonagitated group had apathy, it is unclear whether the association with agitation and delusions can be partially or wholly attributed to comorbid depression in some of these patients. Furthermore, platelet 5-HT₂ function may not be adequate as a peripheral marker for the central serotonergic system. For example, a negative correlation has been found between CSF 5-HIAA and receptor indexes in other neuropsychiatric patients.⁵⁹

Recently, two neuroendocrine studies found that agitated aggressive patients had an increased response to the 5-HT releasing agent fenfluramine compared with nonagitated aggressive patients.^60,61 Both neuroendocrine BPSD studies excluded AD patients with significant depression. One study also found a significant gender effect.⁶² Gender has not been controlled for in other studies looking at the serotonergic system and may be an important contributor to variations in the serotonergic system throughout the life span.

In summary, in vitro and in vivo studies provide inconsistent evidence to link serotonin dysfunction with psychotic symptoms and with anxiety and depression in AD. Failure to control for concomitant medication use and comorbidity are largely responsible for these inconsistencies. Current clinical evidence does support earlier work by Palmer et al.¹⁹ and Procter et al.⁵⁶ linking aggressive agitation with the serotonergic system.

Evidence From Pharmacotherapy Studies

The neurochemical and neurophysiological evidence that the serotonergic system may contribute to BPSD has been supported by a number of studies involving pharmacologic agents specific for the serotonergic system. Clinical manipulation of the serotonergic system through pharmacotherapy provides only indirect evidence of the involvement of that system with the target behavior. However, because the goal of understanding BPSD is to improve treatment, this information is worth briefly reviewing. (More extensive reviews are available.^1,63) Considering there are 15 different serotonergic receptor subtypes, the number of serotonergic selective medications is limited. Agents reviewed here include the selective serotonin reuptake inhibitors (SSRIs), atypical antipsychotics, buspirone, ondansetron, and the serotonin and noradrenaline reuptake inhibitors (SNRIs).

The SSRIs alaproclate,⁶⁴ zimeldine,⁶⁵ fluoxetine,^66,67 citalopram,⁶⁸ fluvoxamine,⁶⁹ paroxetine,⁷⁰ and sertraline⁷¹ have been given to patients with BPSD in randomized controlled trials. Many patients have also been treated with open-label SSRIs, with positive results. For example, in an open-label study by Pollock et al.,⁷² 9 of 16 patients receiving citalopram showed significant improvement in disinhibition, agitation, hostility, and suspicion. However, when SSRIs have been administered under double-blind, placebo-controlled conditions, results have been equivocal with respect to efficacy for BPSD. Randomized controlled trials in AD patients with alaproclate⁶⁴ and fluoxetine⁶⁶ have shown little efficacy for the SSRIs in treating BPSD, except for depression.⁶⁷ Placebo-controlled trials with mixed dementia populations have shown nonsignificant improvement in BPSD such as irritability, anxiety, fear/panic, mood, and restlessness with fluvoxamine,⁶⁹ and significant improvements in depression with paroxetine.⁷⁰ Fluoxetine was reported to slightly improve disinhibition, depressive symptoms, carbohydrate craving, and compulsions in about one-half of 11 patients with frontotemporal dementia.⁷³ The strongest efficacy was demonstrated by citalopram, which, among the SSRIs, has the greatest in vitro selectivity ratio for the serotonergic versus the noradrenergic system.⁷⁴ Nyth and Gottfries⁶⁸ administered citalopram for 4 weeks to patients with mild to moderate AD (n=65) or multi-infarct dementia (n=24) in a multicenter, placebo-controlled, parallel group study. In AD patients, significant improvements were noted on a geriatric rating scale in emotional bluntness and in all six BPSD (restlessness, confusion, irritability, anxiety, fear/panic, and depressed mood) for baseline versus citalopram scores. Irritability and depression improved significantly for citalopram versus placebo groups. There was no significant treatment effect in the multi-infarct group. The patients in this study had very mild BPSD at baseline, with mean scores of less than 2 out of 6 on each symptom.

The atypical antipsychotics (e.g., risperidone, clozapine, olanzapine) are being used more frequently in patients with BPSD because of their efficacy in treating psychotic symptoms and their favorable side effect profiles compared with their typical antipsychotic counterparts (i.e., reduced extrapyramidal effects). Most of the new antipsychotics have strong antagonistic affinity for the 5-HT₂ receptor. Because the hyperresponsivity of the serotonergic system noted in agitated and aggressive AD patients^60,61 suggests up-regulation of 5-HT₂ receptors, the atypicals may be of benefit in the management of BPSD. To date, risperidone, clozapine, olanzapine, and quetiapine have been assessed to treat behaviors related to dementia.

Risperidone, in a total of 534 patients from case reports,^75–81 case series,^82,83 retrospective chart reviews,^84–87 and open-label cohort studies,^88–90 was reported to produce significant reductions in behaviors such as psychosis and agitation in patients with various types of dementia. Further evidence of risperidone efficacy for BPSD was reported in two recent randomized controlled trials. Katz et al.⁹¹ reported on 625 patients, diagnosed with AD and/or vascular dementia, who received either placebo or 0.5 mg, 1 mg, or 2 mg of risperidone for 12 weeks. The authors found a dose– response relationship with risperidone for reducing all behaviors and aggressiveness, and psychotic symptoms were significantly reduced with 1 mg, but not 2 mg, of risperidone compared with placebo. De Deyn et al.,⁹² in a 12-week trial of patients with AD and/or vascular dementia, found risperidone to significantly reduce aggressiveness compared with haloperidol and placebo.

Clozapine has not been evaluated for the treatment of BPSD in any randomized controlled trials. In an open-label dechallenge and rechallenge case series by Oberholzer et al.,⁹³ clozapine was found to significantly improve “antisocial behavior,” “social competence,” “social interest,” and “irritability” in 7 patients with AD. In two retrospective chart reviews,^94,95 clozapine (doses of 100–300 mg per day) was reported to have minimal to moderate effects on behaviors (e.g., psychosis, agitation) in 5 patients with a variety of dementia diagnoses. Frankenburg et al.⁹⁶ reported a marked response (reduced hallucinations and verbal aggression) to low-dose clozapine (i.e., 18.5 mg/day) in a patient with parkinsonian dementia. However, 2 other patients with unspecified dementia were reported to have no changes or minimal change in behaviors with clozapine.⁹⁶ Finally, in a case report of 4 patients at varied stages of dementia, Pitner et al.⁹⁷ reported marked improvement in psychosis and other behaviors in 2 patients, and worsening of behavioral symptomatology in the remaining 2 patients.

In a large open-label trial of olanzapine, Kinon et al.⁹⁸ reported that patients diagnosed with AD and/or vascular dementia showed significant improvements in psychosis and/or agitation. A recent randomized controlled trial by Street et al.⁹⁹ reported olanzapine (5 mg and 10 mg doses) to significantly improve symptoms of psychosis and other behaviors compared with placebo in 206 AD patients over a 6-week period. The authors also reported that fewer patients without psychosis treated with olanzapine developed delusions and/or hallucinations by the end of the study compared with those receiving a placebo. Walker et al.¹⁰⁰ reported that 6 of 8 patients diagnosed with Lewy body dementia responded to open-label olanzapine with significant reductions in BPSD.

Quetiapine, in a single published open-label study by McManus et al.,¹⁰¹ was found to significantly improve overall BPSD and psychotic symptoms in 106 patients with a variety of organic diseases over a 12-week period.

Although the atypical antipsychotics appear to improve BPSD, it is unclear whether this result is secondary to serotonergic or dopaminergic antagonism, since the typical antipsychotics (e.g., haloperidol, perphenazine, thioridazine) have also been demonstrated to have significant benefit.¹⁰² Future research might help to determine if typical and atypical antipsychotics have different patterns of symptom improvement.

Buspirone (a 5-HT_1A partial agonist), in two randomized controlled double-blind trials, was compared with placebo¹⁰ and haloperidol¹⁰³ to determine efficacy in treating BPSD. There was no significant overall reduction in behavioral symptoms with buspirone compared with placebo in the small number of patients tested (N=10). However, in the other study, buspirone was as effective as haloperidol in treating 26 patients with probable AD and superior to haloperidol in decreasing anxiety and tension.¹⁰³ This latter study did not have a placebo control. Thus, the evidence for the efficacy of buspirone for BPSD is not strong.

Ondansetron (a 5-HT₃ selective antagonist), used primarily for the management of chemotherapy-induced nausea and vomiting, has been used in patients with Parkinson's disease accompanied by dementia.^104,105 Both open-label trials showed some benefit in alleviating visual hallucinations and delusions. Further evidence linking ondansetron to a reduction in behavioral problems awaits controlled clinical trials.

The SNRIs trazodone and clomipramine have also been tested for BPSD. Trazodone, in a number of open-label studies, was found to improve behavioral symptoms in 80% of demented elderly patients enrolled.^106–112 In a double-blind trial, trazodone showed a small but significant reduction in behavioral symptoms compared with buspirone and placebo.¹⁰ In another double-blind study controlled with haloperidol, trends suggested that specific symptoms, including aggression, may respond preferentially to trazodone.¹¹³ Clomipramine, another SNRI, produced significant overall improvement in depression in a small double-blind crossover study of AD patients.¹¹⁴ Although evidence to date shows some benefit with SNRIs, these drugs have more affinity for the noradrenaline than the serotonergic receptors and cannot be considered serotonin-selective medications.

In summary, the evidence supporting the efficacy of SSRIs, atypical antipsychotics, 5-HT agonists and antagonists, and SNRIs in the management of BPSD is as yet inconclusive. However, it must be noted that, as with other trials for BPSD,¹⁰² methodological issues with these studies included failure to blind or control,^{71–73,75–90,93–98,100,101,106–111,115–130} insufficient power,^{10,66,73,94–97,100,101,106–111,113} inclusion of a heterogeneous population (patients had varying etiologies for their dementias or were at varying stages of the illness or had different subtypes of BPSD),^{64,68–70,77–79,81–83,85,86,88,89,91–98,113,117,129,130} and failure to use standardized and validated scales to identify and track target behaviors.^{64,66,77,78,82,85,86,88,94–96,103,113–115,119,120,122–124,130} Furthermore, improper dosing and duration of treatment could contribute to some negative studies, since the effective dose and duration for treatment of different BPSD are unknown. Thus far, there is some evidence to support the use of serotonergic medications. The preliminary literature on SSRIs supports the use of the most serotonin-selective of these medications in patients with Alzheimer-type dementia, and this is corroborated by positive results with atypical antipsychotics. However, there are no predictors of response.

OTHER NEUROTRANSMITTER SYSTEMS

Acetylcholine (ACh)

Loss of cholinergic neurons is an early and consistent finding in AD and is thought to be essential to the pathophysiology. Numerous studies have demonstrated profound changes in the cholinergic system in AD, including deficits in the major cholinergic system arising in the basal forebrain and projecting to the cortex;^131–134 decreases in the cholinergic markers choline acetyltransferase (ChAT) and acetylcholinesterase (AChE) in the cortex, particularly the temporal cortex;^{24,38,135–137} significant losses of neurons in the nucleus basalis of Meynert;^131,138 and reductions in the muscarinic type 2 (M₂) presynaptic receptor density.^132,137 Cholinergic neurons arise in the basal forebrain (nucleus basalis of Meynert, diagonal band of Broca, and medial septum) and innervate widespread areas of the cortex, including the hippocampus and a variety of subcortical structures. Therefore, alterations in the central cholinergic system can be expected to cause significant disruptions in the functioning of the neocortex.

Although the role of the central cholinergic system in cognition is well recognized,^139–145 there is only preliminary evidence suggesting that this neurotransmitter plays an important role in the noncognitive disorders associated with dementia. The cholinergic hypothesis of noncognitive symptoms of AD put forth by Cummings and Back¹⁴⁶ suggests that cholinergic deficits may contribute to symptoms such as psychosis-agitation, apathy-indifference, disinhibition, and aberrant motor behavior. Since virtually all AD patients have loss of cholinergic neurons but some do not have BPSD, other neurotransmitters may be involved.

Serotonin and acetylcholine interact extensively in the human brain. For example, 5-HT inhibits release of ACh from cortical and hippocampal cholinergic nerve terminals, possibly via 5-HT_1B receptors in the hippocampus.¹¹ The 5-HT₃ receptors may also inhibit the release of ACh, whereas 5-HT_1A receptors may mediate an increase in ACh release.²⁹ Thus, disruptions in 5-HT have the potential to influence an already compensated cholinergic system.

A single postmortem study by Procter et al.⁵⁶ examined 5-HT receptor binding (with [³H]ketanserin) and ChAT activity in 17 dementia patients with or without aggressive symptomatology (4 aggressive patients, 13 nonaggressive patients) compared with 18 healthy elderly control subjects. Except for the superior parietal lobe, [³H]ketanserin binding was decreased in 12 cortical areas in aggressive compared with nonaggressive patients. However, ChAT activity throughout the cortex was similar in the two patient groups and was significantly lower in these groups compared with healthy control subjects. Thus, current evidence supports 5-HT deficits being more closely related than acetylcholine to aggression in AD.

Clinical experience with cholinergic and anticholinergic medications provides an indirect link between BPSD and the cholinergic system. In AD, cholinergic therapies have been reported to affect psychosis and agitation, apathy/indifference, disinhibition, and aberrant motor behavior.¹⁴⁶ Postmortem studies provide some evidence that neuronal losses in the nucleus basalis, and also decreases in cholinergic transmission, are correlated with psychotic symptoms in Lewy body dementia.^147,148 Choline acetyltransferase was also shown to be decreased in Lewy body dementia patients with hallucinations.⁴⁹ Overall, there is some preliminary evidence linking psychotic behavioral changes with the cholinergic system, especially the balance between this system and the monoamines.⁴⁹ In summary, evidence supports the suggestion that the addition of serotonergic and cholinergic deficits may be responsible for aggression and psychotic symptoms.

The question of whether serotonergic changes are secondary to cholinergic changes or independent of them cannot be answered as yet. To date, there are no longitudinal studies identifying parallel progression in the degeneration of the 5-HT and cholinergic systems. There are some studies that have identified concurrent deficits^25,149–152 or no change^15,153,154 in 5-HT (e.g., 5-HT and 5-HIAA concentrations, [³H]5-HT binding) with respect to loss of cholinergic markers (e.g., ChAT, ACh synthesis) in AD patients compared with control subjects. These findings suggest that serotonergic deficits are either a variable component of AD or occur only in advanced AD.

Noradrenaline (NA)

The central serotonergic and noradrenergic systems interact in many areas. Serotonin is a co-transmitter with NA, and uptake of 5-HT and NA can be accomplished by either 5-HT or NA neurons.¹⁵⁵ The serotonergic system also inhibits the release of NA via 5-HT₁ receptors.²⁹ Thus, there is reason to suspect that dysfunction in the serotonergic system will be accompanied by changes in the NA system. Animal studies have shown that NA neurons from the locus ceruleus (LC) are involved in behaviors such as the sleep-wake cycle, level of vigilance, and emotion.^156,157 Dysfunction in the NA system may also directly influence BPSD.

Postmortem studies suggest AD is associated with loss of neurons in the locus ceruleus,^134,158 the major source of NA in the brain, and with decreased levels of NA in many brain areas innervated by the LC.^{17,20,24,27,53,159,160} Loss of NA neurons is correlated with the severity of dementia.¹⁶¹ In contrast, levels of the NA metabolite 4-methoxy-5-hydroxy-phenylglycol (MHPG) are increased,^162–165 indicating that NA metabolism is increased. This increased MHPG/NA ratio likely reflects compensatory increases in the activity of remaining LC neurons.²⁷ These changes are not reflected in the CSF of patients.^160,166 CSF levels of MHPG are thought to increase with advancing disease,^167,168 complicating interpretation of CSF studies.

Evidence supports a link between depressive symptoms in AD patients and NA dysfunction. The evidence includes neuronal losses in the LC,^{156,169–171} lower levels of cortical NA,¹⁷² and increased NA dysfunction.¹⁷³ However, a well-designed postmortem study with prospective patient evaluations before death was unable to confirm this relationship.²⁷

Preservation of NA may also be associated with BPSD. A postmortem study in AD patients indicated that increased levels of NA,⁵³ comparable to those found in elderly control subjects,¹⁷² were associated with psychotic behaviors, but this relationship was not reflected in CSF MHPG concentrations.¹⁶⁸ There is evidence that restlessness is weakly related to CSF levels of MHPG in patients with AD,⁵⁵ although agitation, in another study, was not.¹⁶⁸ A recent study also showed markedly increased concentrations of α₂ receptors (by ∼70%) and β₁ and β₂ adrenergic receptors (by ∼25%) in the cerebellar cortex of aggressive patients compared with nonaggressive patients with similar levels of cognitive deficit. Levels of α₂ receptors in aggressive patients were slightly above those in the healthy elderly control subjects, whereas increases in β receptors were higher than in both nonaggressive AD patients and control subjects.¹⁷⁴ Thus, aggressive behavior may be associated with cerebellar noradrenergic preservation.

Dopamine (DA)

Serotonergic neurons interact closely with dopaminergic neurons. For example, serotonergic neurons from the raphe nuclei are known to synapse with dopaminergic neurons and to control dopamine release in the midbrain, striatum, and nucleus accumbens.¹⁷⁵ Serotonergic neurons may either inhibit (via 5-HT_1A) or increase release of DA.²⁹ Thus, loss of serotonergic neurons will affect the DA system.

The dopaminergic system has been implicated in depression, agitation, and psychotic behaviors in nondemented patients, and thus this system has the potential to directly influence BPSD. Although disruptions in the dopaminergic system have been demonstrated post mortem in AD patients compared with control subjects,^20,159 DA is less affected than other biogenic amines.^17,27 Dopaminergic disruptions may primarily involve presynaptic disruptions in the DA transporter and postsynaptic loss of striatal dopamine type 2 (D₂) receptors.¹⁷⁶ Furthermore, loss of striatal D₂ receptors may be more common in the 30% of AD patients who also show extrapyramidal features.^176,177 Interestingly, a neuroimaging study demonstrated that disruptions in DA metabolism became increasingly severe as the cognitive impairment progressed.¹⁷⁸ As a result, level of cognitive impairment will be a confounder for any studies attempting to relate DA dysfunction to behavior.

Studies in patients with AD that controlled for level of cognitive impairment have not been able to demonstrate the expected links between psychotic behaviors and abnormalities in the DA system^53,179,180 or depression and the DA system.²⁷ The same may not be true in other dementias. In Lewy body dementia, dopamine metabolites were significantly decreased in nonhallucinating patients in conjunction with serotonergic abnormalities (i.e., decreased serotonergic 5-HT₂ receptor binding and decreased 5-HT metabolites).⁴⁹ When combined with the finding that ChAT was decreased in hallucinating patients, these results suggest that an imbalance between monoaminergic and cholinergic transmitters is involved in hallucinations seen in Lewy body dementia.

There is some evidence that agitated and aggressive behaviors may be associated with a relative preservation of DA function in AD patients. A postmortem study¹⁸¹ demonstrated that preservation of substantia nigra neurons was related to aggression and interpersonal violence. Studies performed in vivo have supported this. Lopez et al.¹⁸² found that whereas aggressive patients showed similar CSF homovanillic acid (HVA) concentrations to their age-matched controls, nonaggressive patients showed decreased levels. Similarly, Sweet et al.¹⁸³ found that improvement in hostility during perphenazine treatment correlated with change in plasma HVA levels in 21 patients (89% AD), further supporting the link between aggression and DA. Thus, DA preservation may be associated with the presence of aggressive behaviors. It is plausible that cholinergic and serotonergic dysfunction may also be present and necessary for this behavior. There are no studies answering this question as yet.

Gamma-aminobutyric Acid (GABA)

Serotonin is a co-transmitter with GABA, and GABA agonists can alter the function of several 5-HT receptors.¹⁸⁴ GABA is the primary inhibitory neurotransmitter in the CNS.¹⁸⁵ It is a local inhibitory interneuron for other neurotransmitters that are key in controlling behavior, including serotonin and dopamine.¹⁸⁶ In addition to these local circuit actions, GABAergic neurons project from the striatum to the lateral globus pallidus and from the cerebellar inferior olivary nucleus to the vestibular nucleus.¹⁸⁶ Thus, GABA is well positioned to influence many psychobiological functions such as behavior through interactions with serotonin.

Involvement of the neurotransmitter GABA has been shown in behaviors such as aggression, increased GABA being associated with decreased aggression.¹⁸⁷ Many lines of evidence have been pursued to establish the presence or absence of a GABAergic abnormality in AD, including postmortem studies of neurotransmitter concentrations^{150,188–198} and binding,^{18,35,37,199–207} antemortem studies of CSF levels of neurotransmitter and metabolites,^208–216 and PET^217,218 and SPECT²¹⁹ studies. Each of these avenues has generated inconclusive results; many studies showed decreased GABA,^{18,37,189–195,197,199–201,203,205,206,209,210,215} but some studies showed no significant changes.^{150,188,196,207,208,211–214,216} Although the discrepant results could reflect the varying methodologies used, each with inherent limitations, these results may also reflect reality. This evidence suggests that GABA disruption is not linked to cognitive degeneration to the same extent as cholinergic dysfunction. GABA dysfunction is a variable component of AD, and, as such, may be linked to the presence of certain behavioral disorders. As yet, there are no clinical studies linking changes in the GABA system with particular behaviors seen in AD.

Neuropeptides

Neurotrophic factors, such as brain-derived neurotrophic factor, somatostatin, and neuropeptide Y, have all been reported to be co-localized within serotonergic neurons in the CNS in both animals and humans.^220–222 Thus, loss of serotonergic neurons implies at least some disruption in these neuropeptides. Although a number of authors have studied the relationship between neuropeptide abnormalities and AD pathology (see review by Gottfries et al.²²³), very little work has focused on the putative effects of altered neuropeptides and BPSD. Minthon and colleagues, in two separate studies,^224,225 attempted to correlate somatostatin and neuropeptide Y levels in the CSF with BPSD. The authors reported a significant negative correlation of neuropeptide Y levels with clinical features of AD, including anxiousness, restlessness-agitation, and irritability. The authors failed to find a significant correlation between somatostatin and specific behaviors related to dementia. Whether a reduction in neuropeptide Y correlates with serotonergic deficits in patients with BPSD has yet to be determined. However, given that many of these neurotrophic factors have been associated with behaviors (e.g., aggression, anxiety, depression) in nondemented populations,²²⁶ it is plausible that reductions in specific neuropeptides concurrently with serotonin can contribute behavioral phenotypes in dementia.

CONCLUSIONS

The serotonergic system is complex, and much remains to be learned about its function in normal states and dementia. Current evidence suggests that AD and related dementias are associated with changes in serotonergic neurotransmission. In addition, many neurotransmitters and neuropeptides that interact extensively with the serotonergic system are dysfunctional in these dementias, but it is unknown whether this is secondary to cholinergic or serotonergic changes. Attempts to link these changes with specific behavioral and psychological symptoms of dementia have been made. Although there is no clear evidence linking alterations of the serotonergic system with a single behavior or cluster of behaviors, it seems relatively certain that such alterations do contribute to noncognitive symptoms. The most supportive evidence is for a contribution of serotonergic system changes to aggressive behaviors and psychotic behaviors.

Much of the evidence to date consists of postmortem studies. For these studies to be rigorous, it is necessary to have prospective, antemortem data collection, including data on drug use and on well-defined behaviors. Neuroimaging techniques (PET, SPECT), for example, represent novel approaches to determining neurotransmitter changes in antemortem brain. Radiolabeled agonists or antagonists specific for each receptor system can help to elucidate altered binding potential of certain receptor subtypes. For example, the 5-HT_1A–specific antagonist [¹¹C]WAY-100635 has been used successfully to determine 5-HT_1A binding potential in patients with depression.^227,228 In the case of dementia, these radiolabeled tracers may lead to the identification of specific receptor deficits that may correlate with noncognitive aspects of the disease—which in turn may lead to receptor-specific drug therapies. Other potential antemortem investigations of central neurotransmitter function include neuroendocrine challenge studies to predict response to drug treatment (e.g., prolactin response to serotonergic challenge, as mentioned previously), prospective intervention studies of multiple drug therapies to manage the spectrum of behaviors (e.g., neuroleptics and antidepressants), and longitudinal studies with drug prophylaxis in early dementia (e.g., SSRI administration, combined with longitudinal scanning, in dementia patients with a family history of mood disorders).

Unfortunately, much of the work done to date suffers from lack of standardized definitions for behaviors. Patients were frequently defined clinically by the presence of a given behavior. Because many of the behaviors overlap, selecting on the basis of the presence of a single behavior will not provide homogeneity in the patient population. A neurotransmitter-based approach will offer information that can be more closely linked with treatment options. Another limitation is that in postmortem studies linking behaviors to neurotransmitter abnormalities, the behaviors may or may not be expressed at the time of death. Because behavioral changes associated with degenerative neurologic changes are likely to be predominantly a state rather than a trait, it is important that the behaviors are expressed near the time of death. Also, both neurotransmitter disruptions and behaviors are known to change with level of cognitive impairment.²²⁹ Findings of changes in behaviors may arise both from actual changes in the behaviors and difficulty in measuring some behaviors as the level of cognitive impairment increases (e.g., auditory hallucinations as the patient becomes less verbal). Thus, it is important to control for level of cognitive impairment as well as to study BPSD through the course of the disease. This latter goal will be more easily achieved with the application of in vivo measures of receptor function; these can include peripheral markers such as neuroendocrine challenges and CSF levels, as well as functional imaging when feasible.

Studying the neurochemical etiology of BPSD presents many challenges. The activity of a single neurotransmitter such as serotonin cannot be separated from those of other neurotransmitters and neuropeptides. Hence, it is essential that future work focus on identifying concurrent neurotransmitter and neuropeptide changes through the progression of the disease. Careful study of clusters of BPSD and their corresponding imbalances may eventually provide enough information to make possible evidence-based, patient-specific treatments for behavioral disorders in dementia.

ACKNOWLEDGMENTS

Dr. Lanctôt's work was supported by the Alzheimer Society of Canada Research Program and is currently partially supported by the Kunin-Lunenfeld Applied Research Unit of Baycrest Centre for Geriatric Care, Toronto, Ontario.

TABLE 1. Potential role for serotonin (5-HT) receptor subtypes in behavioral and psychological symptoms of dementia

Enlarge table

TABLE 2. Summary of literature linking serotonergic abnormalities with behavioral and psychological symptoms of dementia in Alzheimer's disease patients

Enlarge table