Neurophysiological Aspects of Synesthetic Experience

Integration of information from multiple sensory channels is a necessary prerequisite for many tasks with which the nervous system is entrusted. Most notably, the visual and vestibular systems are deeply intertwined so as to induce the sensation of movement when the vestibular system is activated. Also, intersensory bias can occur in a number of instances—for example, vision can influence perception of auditory or proprioceptive input and vice versa.¹ Most notably, this effect has been exploited by ventriloquists who fool the audience about the source of their dummy's voice.²

Besides these common examples of intersensory integration, there is a rare phenomenon termed synesthesia, in which a commingling of sensory experiences from different modalities occurs in otherwise neurologically and psychologically healthy individuals. For example, those individuals might experience the intense feeling of touching shapes upon tasting something (taste–shape synesthesia) or they might see colors upon hearing words or sounds (colored hearing synesthesia). Synesthetic phenomena can also be found in normal individuals, albeit to a much lesser degree—for instance, some people have intense somesthetic sensations upon hearing music.^3,4

Several pieces of evidence support the notion that indeed synesthetic experience has a neural basis:

With regard to possible mechanisms through which brain circuits might subserve synesthetic experience, a wealth of data has accumulated showing the presence of multisensory neurons at several levels of the central nervous system.¹² However, although a number of studies address the phenomenology of synesthetic experience^{6,7,9,13–17} experiments trying to pinpoint the neural basis of synesthesia have been scarce. Cytowic and Wood¹⁸ have used the nontomographic xenon-133 method to assess cerebral blood flow in a single subject with taste–shape synesthesia. They concluded that the sensory integration leading to the synesthetic sensation takes place in the limbic system (a region not actually assessed in their study) and that neocortical structures appeared to be inhibited. In a more recent study using the superior PET activation technique in 6 synesthetic (colored hearing) and 6 control subjects, Paulesu et al.¹¹ found activation in visual association areas (posterior inferior temporal cortex, parieto-occipital junction) as well as activation of right prefrontal cortex, insula, and superior temporal gyrus. Notably, primary visual areas did not show blood flow changes during activation.

In the present study we sought to investigate the neurophysiology of synesthesia by the recording of event-related brain potentials (ERPs). These are small voltage fluctuations occurring in response to a stimulus or a cognitive event and can be extracted from the ongoing EEG by averaging techniques.¹⁹ The main advantages of ERPs are their high temporal resolution, noninvasive nature, and repeatability. Their applications in neurology and psychology have been manifold and include the investigation of visual perception,^20,21 memory,²² and language processes.²³ A study by Hillyard et al.²⁴ had subjects attend to visual or auditory stimuli at a particular location and examined the effects on the ERPs to stimuli of the unattended (visual or auditory) modality. It was found that ERPs to stimuli of the unattended modality were enhanced in amplitude at the attended location, suggesting a mapping of all stimuli onto a common auditory-visual space. These results attest to the utility of ERPs in the investigation of sensory processing and suggest a preferred allocation of resources to stimuli at the attended location, regardless of modality.

For the current study we adapted the well-known oddball paradigm,²⁵ which requires the subjects to respond to a rare target stimulus in a series of distractor stimuli. Subjects were confronted with stimuli (numbers and letters) that were known to elicit synesthetic experiences in the selected synesthetes. We hypothesized that differences in the processing of the stimuli between synesthetic and control subjects would show up in the ERPs. Specifically, in light of the PET data presented by Paulesu et al.¹¹ on synesthetic subjects and single-cell data from primates and other species indicating that the frontal and prefrontal areas are richly endowed with multisensory neurons,^26–30 we predicted differences between synesthetic and control subjects over frontal regions.

METHODS

Subjects

Seventeen synesthetic (SYN) subjects (14 women, 3 men; age range 20–52 years, mean 32.4) were selected from a much larger sample of more than a hundred persons claiming to have synesthetic experiences. The original sample was collected after a feature article on synesthesia had appeared in a popular journal. Those subjects were carefully screened for psychiatric, neurological, and ophthalmological diseases by one of the authors (K.T.) and were found to be healthy. The subjects were selected for color-graphemic synesthesia. The original sample also included subjects with colored hearing, colored music, and more complex synesthesias. Thus, all SYN subjects reported continuous synesthetic perceptions when they read letters and/or numbers. Nine subjects reported that the perceived color depended on the vowels that a given word contains, 6 said it depended on both the vowels and the consonants, and 2 reported that the first letter exclusively determined the color. The perceptions were described as unstructured patches or blobs of color. Subjects in which two or more letters determined the synesthesia reported several patches of different colors next to each other rather than one patch of a mixed color. All subjects reported that the synesthesias “automatically” accompanied reading and that they made no efforts to elicit the synesthetic experiences.

Eleven SYN subjects had obtained a university diploma, 4 were expert workers, and 2 were homemakers. Four of the female subjects reported at least one other case of synesthesia in their families. None of the male subjects had any relatives with known synesthesia. Most SYN subjects dated their synesthesias back to the preschool age (latest onset 10 years).

A group of age- and sex-matched control (CON) subjects was obtained in which no subject had ever experienced any synesthesia.

All subjects were right-handed and gave their informed consent to participate. The study protocol was approved by the local ethics committee.

Stimuli

Only visual stimuli were used in the present study. In Experiment 1 the subjects viewed a sequence of letters appearing in white against a dark background in the middle of a video monitor. The viewing distance was 80 cm, and height (width) of stimuli was 4 (3) cm. The duration of the letters was 300 ms, with an interstimulus interval varying randomly between 1,500 and 2,500 ms. Before each experimental run, subjects were informed as to which of the letters would serve as a target during that particular run and had to be responded to by pressing a button held in the right hand. Each run contained 90 stimuli, and 6 runs were administered to each subject. Fifteen letters in each run served as targets, and the others served as distractors. The letters A, E, I, O, U, and M were used as distractors and targets.

The mode of presentation, timing, and task of Experiment 2 was very similar to Experiment 1 except that the numbers 1, 4, 5, and 18 served as distractors and targets.

Recording and Data Quantification

The EEG was recorded from 29 scalp sites, including all standard sites of the international 10-20 system, by using tin electrodes mounted in an elastic cap (Electro-Cap, Inc., Eaton, OH). The reference electrode was placed on the right mastoid process, and additional electrodes were used to monitor eye movements for artifact rejection purposes. Biosignals were amplified (low-pass filter settings 100 Hz, time constants 10 seconds), digitized with a 4 ms resolution, and stored for later offline analysis. ERPs were averaged according to the different stimulus categories (e.g., targets vs. nontargets) for 1,024-ms epochs, starting 100 ms before stimulus onset.

The resulting waveforms were quantified by using mean amplitude measures relative to the mean amplitude of the 100-ms baseline period preceding the stimulus. The amplitude measures were entered in a repeated-measures analysis of variance with group (SYN vs. CON) as between-subjects factor and stimulus (target vs. nontarget) and electrode site as within-subjects factors. Two sets of analyses were conducted, one on the three midline electrodes (F_z, C_z, and P_z), which were entered as levels for the electrode site factor, and a second set on the parasagittal electrodes (F_p1/F_p2, F₃/F₄, C₃/C₄, P₃/P₄, O₁/O₂), with data arranged to yield a hemisphere factor (levels left, right) and an electrode site factor (five levels from anterior to posterior). The Greenhouse-Geisser correction³¹ for inhomogeneity of covariance was used whenever applicable, and reported P-values are corrected.

RESULTS

Subjective Reports of the Synesthetic Subjects

Fourteen subjects reported definite synesthesia throughout the experiment, 2 were not sure, and 1 subject denied synesthesia during the experiment.

Twelve SYN subjects reported constant appearance of colors with every character they saw, whereas 2 had stronger perceptions with the vowels. Of the latter 2 subjects, 1 did not experience synesthesia when seeing the “M.”. The same 2 persons reported seeing the “18” as a patch-pattern of the colors for the single digits, standing side to side. All subjects felt that synesthetic perception was unrelated to the target status of a letter.

Behavioral Data

The mean reaction time to target stimuli in the letter task (Experiment 1) was 480 ms (SD=97) for SYN and 500 ms (SD=116) for CON subjects, which was not a significant difference. Similarly, hit rates did not significantly differ between the two subject groups (SYN 95.2%, SD=4.2; CON 94.5%, SD=4.5).

Analogous results were obtained in the number task (Experiment 2), in which reaction times were 460 ms (SD=102) for SYN and 484 ms (SD=134) for CON (not significant). The percentage of correct responses was 95.7% (SD=4.1) in the SYN and 92% (SD=5.1) in the CON subjects (not significant).

Event-related Potentials

Experiment 1:

The grand average potentials (n=17) for the SYN and CON groups are shown in Figure 1A and Figure 2A. Concerning the difference between target and nontarget stimuli, a large-amplitude P300 component²⁵ with a parietal maximum was observed for the targets with an approximate peak latency of 430 ms and a parietal maximum in both groups. Figure 1A shows the grand average potentials for the electrode site P_z, where the P300 component had its maximum amplitude. Regarding the critical comparison of this study, the waveforms for SYN and CON subjects were virtually superimposable at the parietal site for both stimulus classes. A clear-cut difference emerged, however, over frontal and central sites, where the waveforms of the CON group took a more negative course beginning approximately 150 ms after stimulus onset (Figure 2A). Figure 3 illustrates the interindividual variability in the two groups. Although the variability in the two groups appears to be similar, the synesthetic subjects as a group have more positive ERPs at the frontal sites. This pronounced difference in distribution of ERPs between the two groups is best illustrated by looking at mean voltage amplitudes, in the time window 300–600 ms poststimulus, of electrodes in two parasagittal lines and two temporal lines (Figure 4A). The voltages at parieto-occipital sites are similar for the two groups, whereas marked differences are apparent over frontal areas.

The statistical results are summarized in Table 1. For both midline and parasagittal sites, highly significant effects were obtained for the stimulus factor for most time windows. A main effect of the factor group was found in only a single time window. The highly different scalp distributions of the ERPs between the groups was reflected in group by electrode-site interaction effects.

Experiment 2:

The results from the number task are presented in an analogous fashion. Figure 1B shows the very similar potentials at the parietal site in the two groups for every stimulus class displaying the expected difference between target and nontarget stimuli. Figure 2B shows the ERPs from midline locations, with a pronounced difference between groups over frontal and central sites and no clear-cut differences at the parietal site. The corresponding mean voltage amplitudes are shown in Figure 4B. Again, marked differences in distribution of waveforms were found between groups for both stimulus classes. The differences were found predominantly over anterior scalp regions.

The statistical results are summarized in Table 1. Again, the greater positivity to the targets was reflected in a main effect of the stimulus factor in several time windows. A main effect of group was found for only the 200–300-ms time window, whereas group by electrode-site interactions reflecting the different distributions between the two groups were found for most of the time windows.

DISCUSSION

The aim of the current study was to screen for electrophysiological differences between subjects experiencing synesthesia and matched control subjects. In two experiments, highly significant differences emerged in the distribution of brain activity, the waveforms of synesthetic subjects being more positive over anterior scalp areas. On the other hand, task-related ERP changes, such as the late positive component indexing the evaluation of the target stimulus, appeared by and large the same between the two groups, and behavioral indices of task performance did not differ between the two groups.

In order to explore the possible links of these results with previous data, we will structure the discussion in the following way: first, we will address their relation to other findings from the ERP literature; then, we will turn to the question of how they relate to data from the synesthesia literature, especially functional imaging data; and finally, we will present a tentative explanation of the data to be tested in further studies.

Relationship to ERP Data

Slow ERPs have been used increasingly to monitor the brain's responses in a number of tasks. Slow brain potentials occur on a time scale ranging from several hundred milliseconds up to several seconds. Slow negative potentials have been observed in a wide variety of applications, such as selective attention,³² mental arithmetic,³³ mental rotation,³⁴ and concept formation,³⁵ to name but a few. A prevailing view of long-duration slow potentials has been that the surface negative potentials reflect an activation of the underlying cortical areas,^36,37 whereas positivities are believed to reflect cortical inhibition. This view is based on the finding that in a variety of animal models, the columnar cortical architecture and the resulting orientation of the neurons cause excitatory postsynaptic potentials in the upper cortical layers to result in surface negativities.^38,39

Applying this reasoning to the current set of data, one is led to conclude that the increased positivity overlying anterior brain areas in the synesthetic group reflects an inhibition of frontal and prefrontal structures. We will return to this possibility later in this discussion.

In interpreting ERP results one should bear in mind that event-related potentials in general reflect cortical activity, with little contribution of subcortical structures to the field observed at the scalp. One reason for this is that subcortical structures such as the basal ganglia have an architecture that would lead to a closed field; hence there is no orderly orientation of the neurons, leading to a cancellation of the fields of the single neurons. Thus, any influence of subcortical structures in the generation or suppression of synesthetic experience can be assessed only indirectly, via its influences on cortical potentials.

Relationship to Other Results on Synesthesia

The present data bear some resemblance to the PET imaging study by Paulesu et al.¹¹ Besides activation of a number of higher order visual association areas (posterior inferior temporal cortex, parieto-occipital junction), these authors demonstrated activation in extravisual areas as well, such as the right prefrontal cortex, the insula, and the superior temporal gyrus. The frontal activation was interpreted by these authors as “due to the attentional demands generated by the perceptual demands of hearing words, which in this case convey double and sometimes conflicting information.” They also refer to work by Zeki et al.,⁴⁰ who were able to show with the stimulation PET method that illusory visual motion perception led to a prefrontal activation explained by the attentional demands carried by that task.

Role of Prefrontal Cortex

In interpreting our data, we must consider the functions of prefrontal cortex. Distractibility is one of the key features observed in animals or humans with lesions to the prefrontal cortex,^41–43 and it is explained by a failure to suppress attention to irrelevant stimuli. The normal prefrontal output results in an inhibitory modulation of subcortical and cortical structures.^44,45

The question remains of how to interpret the frontal positivity in synesthetic subjects in the present study. Following the reasoning of Elbert and Rockstroh,³⁶ this frontal positivity would have to be interpreted as an inhibition of the frontal lobe, which in turn should lead to an increased distractibility and possibly to a leakage between the modalities. An alternative interpretation draws on the notion that frontal areas are richly endowed with multisensory neurons and that this area could serve as the anatomical site in which synesthesia occurs.^26–30 The inhibition occurring over anterior brain regions could be thought of as an effort of the nervous system to keep the synesthetic perceptions to a minimum, since these are interfering with normal perception and are leading to sensory conflicts. The predictions generated by these alternative theories are currently being tested in our laboratory.

ACKNOWLEDGMENTS

The authors are grateful to the subjects for their participation. Jobst Kilian provided excellent technical support. K.S. was supported by a fellowship of the Biomedical Exchange Program between Germany and the United States. T.F.M. was supported by a grant of the Hermann and Lilly Schilling Foundation (TS 013/177/96). Thanks to Marta Kutas for providing a stimulating work environment during the writing of this article.

TABLE 1.

Enlarge table