Visualizing Sound: Effects of Pitch Height and Tonality on Luminance Matching

The Present Study

The present study aims to examine a potential relationship between luminance and pitch height by replicating the results of previous studies. It further aims to investigate a potential relationship between luminance and tonality, based on prior research relating emotion with luminance and tonality, respectively. If found statistically significant, these relationships may demonstrate the existence of cross-modal perception among non-synesthete populations, and provide direction for further research with both synesthetes and normal populations.

Hypotheses

Hypothesis 1 – Pitch Height: High-pitched triads will be matched with lighter colors, while low-pitched triads will be matched with darker colors. This hypothesis is based on previous findings related to luminance and pitch height (Collier 2001; Eitan et al. 2012; Haryu & Kajikawa 2012; Hubbard 1996; Marks 1989; Melara 1989; Miller et al. 2003; Walker 2012; Walker et al. 2009; Ward et al. 2006).

Hypothesis 2 – Tonality: Major triads will be matched with lighter colors, while minor triads will be matched with darker colors. This hypothesis is based on previous associations found between emotion and tonality (Suzuki et al. 2008; Kellaris & Kent 1994; Hunter et al. 2010), and emotion and color (Boyatzis & Varghese 1994; Hemphill 1996; Xin 2004).

Hypothesis 3 – Interaction: The effects of pitch height will override tonality for incongruent triads (high-pitched/minor; low-pitched/major). This hypothesis is based on the theory that pitch height is more salient than tonality, and will thus serve as a more effective vehicle for cross-modal mappings with luminance, since prior research with multiple populations has been conducted with regards to pitch height, but not tonality.

Methodology

Participants

Twenty-eight undergraduate students (17 women, 9 men, 2 unreported) at a small liberal arts university in the Pacific Northwest were recruited through the university’s online psycholgy research familiarization system. Students received either research familiarization credit or extra credit points for participating, depending on their course requirements.

Seven subjects responded “Yes” to an item on the study questionnaire which stated, “I see colors when I hear music, even if no colors are physically present.” Although subjects were not tested to verify this response, they were eliminated from the final analysis to ensure that the sample was composed entirely of subjects without sound-to-color synesthesia. An additional two subjects were eliminated from the final analysis because they reported suspicion of the study’s hypotheses on the questionnaire.

Five subjects reported that they had received no musical training in the past, seven subjects reported one to five years of musical training, five subjects reported five or more years of musical training (but not pursuing a music major or minor), and the remaining two subjects did not report their musical training on the study questionnaire.

Materials

Forty triads were recorded by the researcher on an electronic piano, varying in pitch (high or low) and tonality (major or minor), with equal numbers of triads in each category (n = 10: high major; high minor; low major; low minor). Each sound clip lasted approximately five seconds. Each major triad had a minor triad counterpart that shared the same base pitch; thus, 20 unique base pitches were used. Base pitches for low triads ranged from Bb1 (58.270 Hz) to E3 (164.81 Hz), and base pitches for high triads ranged from A4 (440.00 Hz) to B5 (987.77 Hz).

The researcher created an eight-by-eight block color grid as the dependent measure. The grid varied in hue (left to right) and luminance (top to bottom). Although irrelevent to the study, hue was included as a variable of the grid in order to give subjects a wider range of potential responses. Letters (A-H) and numerals (1-8) were assigned to the grid so that subjects could specify an individual block in each of their responses. To control for a possible response bias, the grid was designed with darker hues at the top, and lighter hues at the bottom (see Figure 1). Thus, if subjects associated their responses in spatial terms (e.g., higher-pitched triads may increase the likelihood of choosing a block higher on color grid), they would do so in a manner opposite of Hypothesis 1.

Procedures

Data collection took place over a period of six separate sessions, with four to five participants in each session. After reading and signing a research participation consent form, subjects listened to the 40 triads via headphones while seated in front of a monitor which displayed the color grid. Computer monitor brightness and headphone volumes were homogenized before the start of each session to ensure uniformity. The triads were played in random order, using the same order for each subject. On-screen instructions directed participants to pick the color they considered the “best fit” for a given triad, and to record the corresponding letter and numeral for each on a sheet of paper provided by the researcher. Subjects could replay any triad multiple times, but were not allowed to return to a previous triad once they had recorded a response and moved on to the next triad.

After listening and responding to each triad, on-screen instructions directed the subjects to respond to the questionnaire items on the reverse side of their sheet. These items related to gender demographics, sound-to-color synesthesia reporting, musical training background, understanding of minor versus major chord tonality, and a suspicion check to ensure that participants had not guessed the study’s hypotheses during the experiment. After completing the questionnaire, subjects were orally debriefed on the study’s purpose and hypotheses, and granted credit for participating.

Results

A two by two repeated-measures ANOVA (Pitch Height[High][Low] by Tonality[Major][Minor]) was conducted to examine the main effect of pitch height, the main effect of tonality, and the interaction between pitch height and tonality on luminance matching. The triads were coded into four categories: high major, high minor, low major, and low minor. Mean luminance scores for the four categories were calculated for each subject, and overall luminance means for each category were used in the ANOVA. Luminance scores could range from 1 (darkest) to 8 (lightest).

Pitch Height: the analysis revealed statistically significant main effects for pitch height, F(1, 18) = 51.67, partial eta-squared = .74, p < .001. Mean luminance scores were higher for high-pitched triads (M = 5.26, SD = .24) than low-pitched triads (M = 2.92, SD = .16).

Tonality: the analysis also revealed statistically significant main effects for tonality, F(1,18) = 6.99, partial eta-squared = .28, p < .05. Mean luminance scores were higher for major triads (M = 4.23, SD = .14) than minor triads (M = 3.95, SD = .13).

Interaction: the interaction between pitch height and tonality was statistically non-significant, F(1, 18) = 1.52.

Discussion

Hypotheses and Implications

The findings supported Hypothesis 1, which predicted that high-pitched triads would be matched with lighter colors, and low-pitched triads would be matched with darker colors. These findings are consistent with previous research in which subjects matched luminance with pitch height (Collier 2001; Eitan et al. 2012; Haryu & Kajikawa 2012; Hubbard 1996; Marks 1989; Melara 1989; Miller et al. 2003; Walker 2012; Walker et al. 2009; Ward et al. 2006).

The findings also supported Hypothesis 2, which predicted that major triads would be matched with lighter colors, and minor triads would be matched with darker colors. These findings support the theory that the affective valence of tonality and luminance are comparable, based on respective prior findings on these two topics (tonality: Suzuki et al. 2008; Kellaris & Kent 1994; Hunter et al. 2010, color: Boyatzis & Varghese 1994; Hemphill 1996; Xin 2004).

Hypothesis 3 was unsupported, since no statistically significant interaction was found between pitch height and tonality. Tonality seems to be autonomously related to luminance matching, and does not depend on pitch height.

Study Limitations and Future Research

These results are not necessarily generalizable, since the study sample was limited to the research familiarization pool at a small liberal arts university. The majority of participants were female, and other demographics may be distinct to this sample. Further research should encompass a larger and more diverse sample, to increase statistical power and determine whether or not the current findings are generalizable. Also, the sample included no sound-to-color synesthetes. Although prior research on luminance matching with pitch height has been conducted with both synesthetes and non-synesthetes (Ward et al. 2006), no studies exist which compare luminance matching with tonality between the two groups. If tonality functions as an auditory attribute which can be cross-modally mapped by non-synesthetes, future research should investigate if a similar pattern is found among sound-to-color synesthetes.

Potential confounding variavles within the current study design also limit generalizability. Although the triads were played in random order, the same order was used for each participant. Future research should include several different orders, and examine potential differences caused by contrast effects or other such variables.

Other aspects of the study design may have biased participants’ responses, such as the numbers assigned to the color grid for identification purposes. Subjects may have potentially chosen higher numbers in response to higher pitches, which would have contributed to the significant results observed. A future study could control for this potential confound by using a non-hierarchal identification system instead of numbers, or randomly assigning participants to grids with different numbering systems.

Conclusion

In the larger scheme of gaining an understanding of cross-modal perception, future studies should continue to investigate relationships between other visual and auditory attributes, such as chroma, saturation, shape, size, texture (both visual and auditory), loudness, timbre, pitch contour, and rhythm, as well as potential interactions between these variables. Correspondences between luminance and tonality in particular should be explored, using larger populations of both synesthetes and non-synesthetes. Research on auditory-visual correspondences among both groups has the potential to shed light on the underlying neural basis of synesthesia, and the possibility that its idosyncratic manifestations are based on universal and consistent perceptual processes.

---

References

Boyatzis, C.J., & Varghese, R. (1994). Children’s emotional associations with colors. The Journal of Genetic Psychology, 155, 77-85. doi: 10.1080/00221325.1994.9914760

Collier, W.G., & Hubbard, T.L. (2001). Judgments of happiness, brightness, speed and tempo change of auditory stimuli varying in pitch and tempo. Psychomusicology, 17, 36-55.

Eitan, Z., Ornoy, E., & Granot, R.Y. (2012). Listening in the dark: Congential and early blindness and cross-domain mappings in music. Psychomusicology: Music, Mind, & Brain, 22, 33-45.

Haryu, E., & Kajikawa, S. (2012). Are higher-frequency sounds brighter in color and smaller in size? Auditory–visual correspondences in 10-month-old infants. Infant Behavior & Development, 35, 727-732.

Hemphill, M. (1996). A note on adults’ color-emotion associations. The Journal of Genetic Psychology, 157, 275-280.

Hubbard, T. (2010). Auditory imagery: Empirical findings. Psychological Bulletin, 136, 302- 329.

Hubbard, T. (1996). Synesthesia-like mappings of lightness, pitch, and melodic interval. The American Journal of Psychology, 109, 219-238.

Hunter, P.G., Schellenberg, E.G., & Schimmack, U. (2010). Feelings and perceptions of happiness and sadness induced by music: Similarities, differences, and mixed emotions. Psychology of Aesthetics, Creativity, and the Arts, 4, 47-56.

Kellaris, J.J., & Kent, R.J. (1994). An exploratory investigation of responses elicited by music varying in tempo, tonality, and texture. Journal of Consumer Psychology, 2, 381-401.

Marks, L. (1982). Synesthetic perception and poetic metaphor. Journal of Experimental Psychology, 8, 15-23.

Marks, L. (1989). On cross-modal similarity: the perceptual structure of pitch, loudness, and brightness. Journal of Experimental Psychology, 15, 586-602.

Meier, B.P., Robinson, M.D., Crawford, L.E., & Ahlvers, W.J. (2007). When “light” and “dark” thoughts become light and dark responses: Affect biases brightness judgments. Emotion, 7, 366-376. doi: 10.1037/1528-3542.7.2.366

Melara, R. (1989). Dimensional interaction between color and pitch. Journal of Experimental Psychology: Human Perception and Performance, 15, 69-79.

Miller, C.B., Boteler, E.H., & Larcombe, G.K. (2003). “Seeing things in my head”: A synesthete’s images for music and notes. Perception, 32, 1359-1376.

Suzuki, M., Okamura, N., Kawachi, Y., Tashiro, M., Arao, H., Hoshishiba, T., Gyoba, J., & Yanai, K. (2008). Discrete cortical regions associated with the musical beauty of major and minor chords. Cognitive, Affective, & Behavioral Neuroscience, 8, 126-131. doi: 10.3758/CABN.8.2.126

Walker, P. (2012). Cross-sensory correspondences and cross talk between dimensions of connotative meaning: Visual angularity is hard, high-pitched, and bright. Attention, Perception, & Psychophysics, 74, 1792-1809. doi: 10.3758/s13414-012-0341-9

Walker, P., Bremner, J.G., Mason, U., Spring, J., Mattock, K., Slater, A., & Johnson, S.P. (2009). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21, 21-25.

Ward, J., Huckstep, B., & Tsakanikos, E. (2006). Sound-colour synaesthesia: To what extent does it use cross-modal mechanisms common to us all? Cortex, 42, 264-280.

Xin, J.H., Cheng, K.M., Taylor, G., Sato, T., & Hansuebsai, A. (2004). Cross-regional comparison of colour emotions part I: Quantitative analysis. Color Research and Application, 29, 451-457.