Visualizing Sound: Effects of Pitch Height and Tonality on Luminance Matching

Synesthesia is a perceptual condition which challenges the tradition notion of five completely separate senses. Synesthetes experience a blending of the senses; they have additional sensory perceptions in one modality in response to input from another. Although rare, synesthesia can occur between any two or more of the five senses, and research has verified the unusual reports of synesthetes by testing their perceptions and finding them to be unlearned, involuntary, and very consistent over time (Ward, Huckstep, & Tsakanikos 2006).

One form of synesthesia involves the involuntary perception of specific colors in response to sound, especially music. This type of synesthesia, called “colored hearing” or sound-to-color synesthesia, has sparked the interest of researchers who examine emotional correlates of music and color, as well as those interested in cross-modal, particularly audio-visual, associations made by people who lack synesthesia. Comparisons between some attributes of synesthetes’ involuntary color perceptions with audio-visual associations of non-synesthetes has fueled one theory of synesthesia, which claims that this highly unusual condition has roots in normal cross-modal mechanisms common to all human beings (Ward, Huckstep, & Tsakanikos 2006). This study examines potential cross-modal associations made between one visual attribute, luminance, with two different auditory attributes, pitch height and tonality.

Luminance and Cross-Modality

Luminance (the relative lightness or darkness of an object) is relevant to comparisons between synesthesia and normal cross-modal perception because of its associations with overarching emotional and metaphorical concepts. In a 2007 study, Meier, Robinson, Crawford, and Ahlvers found that subjects perceived squares as brighter when paired with positive words, and darker when paired with negative words. This finding suggests that the idea of congruency, or matching the attributes of distinct concepts with each other, can influence individual aspects of perception. Along similar lines, Marks (1982) conducted a study in which participants adjusted the brightness of a light and the loudness of a tone to match given lines of poetry. The subjects adjusted the respective levels of each mode (low, medium, or high brightness or loudness) in almost perfect correlation, proportionate on a line graph. Since subjects in both of these studies seemed to match luminance, a strictly visual attribute by definition, with external, non-visual attributes, we can reasonably conclude that luminance may represent a broader concept, such as intensity, which lends itself to cross-modal mapping.

Hubbard (2010) compared previous empirical findings related to auditory imagery, and suggested that since auditory imagery shares many semantic characteristics with acoustic stimuli, humans’ interpretation of both may be based on shared qualities, such as intensity. Based on this theory, and the findings of Meier et al. and Marks, luminance seems to aptly represent visual intensity, as demonstrated by subjects’ ability to effectively associate it with non-visual concepts.

Pitch Height and Luminance

Research has examined correspondences between pitch height and luminance using several methodologies. In a series of three experiments conducted by Ward, Huckstep, and Tsakanikos in 2006, a group of sound-to-color synesthetes and a non-synesthete control group matched colors with pitches, using a custom color picker on a computer. Although the synesthetes demonstrated much more consistency in their color choices, both groups chose lighter colors for higher pitches, and darker colors for lower pitches. A study by Miller, Boteler, and Larcombe (2003) observed a similar response pattern in a case study of one synesthete, finding that lower-pitched tones elicted larger and darker visualizations than high-pitched tones.

In a 2012 study, Walker found that pairing a light or dark grey background with either low-pitched or high-pitched sounds affected how quickly non-synesthetes correctly completed a word classification task. Congruent combinations (light background, high pitch; dark background, low pitch) enabled faster word classification among the participants. Similar results have been demonstrated in several studies investigating associations of luminance with pitch height among normal populations (Collier 2001; Hubbard 1996; Marks 1989; Melara 1989).

These findings support the idea of a relationship between pitch height and luminance as a function of general cross-modality, which has led researchers to ask whether or not these correspondences are learned. To examine this question, Walker et. al. conducted a study with infants, pairing light or dark animations with various pitches (2009). The infants looked longer at congruent combinations than incongruent combinations, which supports the hypothesis that this type of cross-modality is present very early on, and thus, potentially unlearned. Haryu and Kajikawa conducted a similar study with infants in 2012, pairing greyscale animations of apples (light grey or dark grey) with a standard tone that moved either to a lower pitch or a higher pitch. Consistent with the findings of Walker et al., the infants looked longer at congruent combinations (tone moving up with light grey apple, tone moving down with dark grey apple).

A study by Eitan, Ornoy, and Granot found that congenitally and early blind subjects associated low segments of a Beethoven sonata with “dark,” and high segments with “light,” using a five-point bipolar scale (2012). These findings suggest that blind people make cross-modal correspondences similar to those of sighted people. In contrast to the two studies involving infants, however, this may suggest that such an association is learned.

In a series of experiments varying pitch height, tempo, and pitch contour, Collier found that subjects rated high-pitched tones as not only brighter, but happier and more quickly ascending than low-pitched tones (2001). Collier also found that ascending tones were perceived in similar ways in comparison to descending tones, and discovered interactions between the three musical attributes included in his study. These associations suggest that pitch height is related not only to luminance, but to emotion, and this relationship may be shared with other visual and musical attributes.

Tonality and Luminance

Tonality (for this study’s purposes, major versus minor) is typically characterized in emotional terms borrowed from linguistics. Descriptors for minor key tonality typically include words such as “sad” and “dark,” versus major key tonality descriptors, which often include adjectives like “happy” and “light.” Suzuki et. al. (2008) used positron emission tomography in a study examining the brain areas activated by major and minor chords, and found that distinct regions were associated with the two types. Their findings suggest that the brain processes tonality in accordance with affective valence, e.g., minor chords are processed in an area of the brain which also handles emotions, but is different from the region which processes major chords.

Main effects of tonality on pleasure and surprise were observed in a 1994 study by Kellaris and Kent, in which subjects assigned higher pleasure ratings to major tonality than minor tonality. Participants also rated major music excerpts as happier than minor music excerpts in an experiment by Hunter, Schellenberg, and Schimmack (2010).

Although tonality is likely correlated with emotion, little research has been conducted to examine whether or not descriptors involving luminance, which are commonly assigned to tonality, can extend beyond linguistics and into cross-modal associations where luminance is a real visual concept, not just a metaphor. Previous studies on color and emotion have produced findings very similar to the positive/negative valence observed with regards to tonality (Boyatzis & Varghese 1994; Hemphill 1996), which supports the likelihood of a relationship between color and tonality. Furthermore, a 2004 study by Xin et. al. indicates that luminance is more relevant than some other visual attributes, such as hue, with regards to associations made between color and emotion. If a relationship between luminance and tonality exists, this may support a cognitive explanation for the linguistic metaphors so often used to characterize tonality. These metaphors include the concept of dark and light – a semantic property directly borrowed from luminance.Continued on Next Page »