The Influence of Gaze Direction on Approach- vs. Avoidance-Oriented Emotions

Introduction

The integration of facial expression of emotion and direction of gaze is an important aspect of human communication (Adams & Kleck, 2005). Facial expressions provide the outer response to changes within inner emotional states, where as gaze direction is indicative of how attention is directed and suggests behavioral intention. Previous findings implicated the amygdala as the particular region involved in the integration of emotional expression and gaze (Sato, Kochiyama, Uono, & Yoshikawa, 2010).

Sato, Kochiyama, Uono, and Yoshikawa (2010) noted inconsistent results among neural exploration of the amygdala and its role in the processing of facial expressions of emotion as a function of gaze direction. Their study attributed these differences to the type of stimulus presentation used to display the emotion. In their research, Sato et al. (2010) categorized stimulus presentation as either dynamic or static. Dynamic presentation consisted of video clips that showed a neutral expression evolving into an emotional expression of either anger or happiness, while static presentation used successive still image frames to display the change from neutral to emotional. The authors hypothesized that the integration of angry and happy emotional expressions with either direct or averted gaze would elicit higher amygdala activity in the dynamic condition than in the static condition. The results of this study supported their hypothesis and indicated the strength of dynamic presentation. Furthermore, their findings showed that both angry and happy dynamic facial expressions elicited greater amygdala activity in response to direct gaze orientation compared to averted gaze orientation.

Behavioral research by Adams and Kleck (2005) suggested that the integration of emotional expression and gaze direction implies the expressor’s behavioral intent to either approach or avoid. They suggested that the emotions of anger and happiness are categorized as approach-oriented emotions, where as the emotions of fear and sadness are defined as avoidance-oriented emotions (Adams & Kleck, 2005). Furthermore, Adams and Kleck (2005) implicated direct gaze would enhance approach-oriented emotions (anger, happiness) and averted gaze would enhance avoidance-oriented emotions (fear, sadness). Their behavioral findings indicated this effect: direct gaze increased the perceived intensity of approach-oriented expression of anger and joy, where as averted gaze increased the perceived intensity of avoidant-oriented expressions of fear and sadness.

The findings of Sato and colleagues (2010) are not clear in light of Adams and Kleck’s (2005) behavioral study. Sato et al. (2010) found greater amygdala activity for emotions of anger and happiness when combined with direct gaze compared to averted gaze. Anger and happiness, according to Adams and Kleck (2005), are approach-oriented emotions. Therefore, Sato and colleagues’ (2010) study does not investigate the integration of gaze with avoidance-oriented emotions and how this combination influences amygdala activity. Furthermore, Harmon-Jones and Sigelman (2001) explored the effects of both approach-oriented and avoidance-oriented affect and suggested a lateralization of activity. Their EEG study implicated greater left-hemisphere responses to be associated with processing anger, an approach-oriented emotion, and right hemispheric activity to be more connected to fear, an avoidance-oriented emotion.

Based upon the findings of Sato et al. (2010) and Adams and Kleck (2005), we investigated the neural bases of both approach- and avoidance-oriented emotions with regards to gaze direction. We used the dynamic presentation of stimuli adapted from Sato and colleagues (2010) due to the elicitation of higher amygdala activity compared to the static presentation. Our study contained the approach-oriented emotion of anger and the avoidance-oriented emotion of fear because each affect is negative and produced the most salient responses in their respective studies. We hypothesized that the amygdala response to anger stimuli with direct gaze would be higher than that of anger stimuli with averted gaze. Furthermore, we expected the amygdala response to fear stimuli with averted gaze would be higher than that of fear stimuli with direct gaze. Based upon the findings of Harmon-Jones and Sigelman (2001), we also predicted a lateralization of activity within the amygdala, with the left amygdala eliciting higher responses to the approach-oriented emotion of anger and the right amygdala eliciting greater activity for the avoidance-oriented emotion of fear.

Method

Participant
A 20 year old right-hand male participated in the experiment. The participant was a native English speaker with normal vision. The participant did not have any neurological deficiency and gave informed consent for the study.

Stimuli
The stimuli consisted of angry and fearful facial expressions of eight individuals (four men and four women) presented dynamically as video clips. These faces were unfamiliar to the subject. Direct and averted gaze directions were simultaneously recorded by a video camera directly in front of model and another one from 30 degrees to the model’s right. The stimuli model looked directly ahead for every recorded stimulus. The dynamic presentation of stimuli began with neutral facial expression and changed to emotional (anger or fear). Each clip was greyscale, without sound, and approximately 1500 ms in length. Still examples of the facial expressions and gaze direction are shown in Figure 1.

Procedure
The participant completed the experimental scan once. The scan contained four experimental runs. Each run consisted of nine 24-s epochs with nine 24-s rest periods (blank screen) in between. Each of the four stimulus conditions was presented in different epochs within each run. The order of the epochs within the run was pseudorandomized. The order of the identity and gender of the stimuli within each epoch was randomized. Each epoch was made up of eight trials. Each trial consisted of a fixation point, a small gray cross on a white background with identical size to stimulus, presented centrally on the screen for approximately 1500 ms. The face stimulus was then shown for about 1500 ms. The participant was told to fixate on the center of the screen until the face disappeared and then asked to determine the gender of the face by pressing one of two buttons. This task assured that the subject paid attention to the stimuli without focusing particularly on the emotional expression and gaze direction. After MRI image acquisition, the researchers debriefed the participant and thanked him for his contribution.

Experimental design
This experiment had a within-subject two-factorial design, with dynamic presentation of emotional expression (anger, fear) and gaze direction (direct, averted).

Behavioral data analysis
We analyzed the accuracy and reaction time data for gender classification by calculating the percentage of accurate response, and the mean and the standard deviation of the reaction time.

MRI acquisition
Scanning was performed on a 3 T Siemens MRI scanner using an 8-channel head coil. The researchers used a forehead pad to maintain steady head position. We obtained a T1-weighted high-resolution anatomical image using magnetization prepared rapid-acquisition gradient-echo (MP-RAGE) sequence (field of view = 192 x 256; voxel size = 0.98 x 0.98 x 1.0 mm; number of slices = 160). The functional images consisted of 42 slices attained in an interleaved fashion, interiorly to superiorly. The slices were parallel to the anterior and posterior commissure plane covering the whole brain. The researchers used a T2*-weighted gradient-echo echo-planar imaging sequence with the following parameters: repetition rime (TR) = 3000 ms; matrix size = 64 x 64; voxel size = 3 x 3 x 3 mm. There were 151 TRs in each run.

Image analysis
The statistical parametric mapping package SPM5 implemented in MATLAB version 7 performed the image and statistical analyses. Functional images were slice-time corrected and subsequently realigned using the first scan as a reference to correct for head movement. Next, we coregistered the TI anatomical image to the first slice-corrected functional image. Following this, the coregistered T1 anatomical image was normalized to a standard T1 template image as defined by the Montreal Neurological Institute (MNI). The researchers then applied the parameters for this normalization process to each functional image. Finally, we smoothed the spatially normalized functional images with an isotopic Gaussian kernel of 6 mm.

The researchers conducted random effects analyses to determine any significantly activated voxels that displayed interesting effects. Each task-related condition was modeled with a boxcar function, convoluted with a canonical hemodynamic response function. First, we performed planned contrasts for the interaction of anger-versus-fear conditions and direct-versus-averted directions, according to our hypothesis. Further planned contrasts were performed for main effects (main effect of emotion and main effect of gaze direction) as well for simple effects of each condition (anger-direct, anger-averted, fear-direct, and fear-averted). We used an initial threshold of p < 0.001 (uncorrected) to identify significantly activated voxels. For specific contrasts which did not show significant activity with uncorrected threshold, we used a p < 0.05 (corrected) threshold. The researchers carried out these analyses across the whole brain.

To scrutinize the activation patterns of the amygdala, we further conducted region of interest (ROI) analyses. SPM5 (Pickatlas) determined the location of the amygdala using anatomical landmarks. We analyzed the contrast estimate of each condition in both the left and the right amygdala for the interaction. Furthermore, the left and right fusiform gyrus was also located using anatomical landmarks in SPM5 (Pickatlas) and analyzed using contrast estimates for the interaction between emotion and gaze direction.

Results

Gender classification
Performance on the gender classification task was good (correct overall identification rate=97%). The mean reaction time across all conditions was 1743 ms. For the direct anger condition, mean reaction time was 1763 ms (SD = 53), with an accuracy rate of 100%. Averted anger condition resulted in mean reaction time of 1737 ms (SD = 134) and an accuracy rate of 95%. For direct fear, there was a mean reaction time 1743 ms (SD = 100) and accuracy rate of 96%. Averted fear resulted in a mean reaction time of 1756 ms (SD = 121) and 96% accuracy rate.

Brain activity

Main effect of emotion
The main effect of emotion for (anger > fear) found significant activation (p < .001) in the right fusiform gyrus, right cuneus, left precuneus, paracentral lobule, and the cerebellum (Table 1). There was no significant main effect emotion for (anger < fear).

Main effect of gaze direction
The main effect of gaze direction for (direct < averted) resulted in significant activation (p < .001) in the right cuneus (Table 1). There was no significant main effect for (direct > averted).

Interaction between emotion and gaze direction
A planned contrast of the interaction between emotion and gaze direction did not reveal significant activity across the whole brain.

ROI Analyses

Since there was no statistically significant activation for the interaction between emotion and gaze direction, ROI analyses of the amygdala were performed. The trend of the contrast estimates suggested an interaction between emotion and gaze direction in both left and right amygdala. Within the left amygdala, the anger condition resulted in higher contrast estimates for direct gaze (3.2) compared to averted gaze (1.2). There was little difference between direct gaze (-0.6) and averted gaze (-0.3) within the fear condition (Figure 2). In the right amygdala, the anger condition yielded little difference between direct gaze (2.0) and averted gaze (1.4). However, the fear condition showed less deactivation for averted gaze (-0.6) compared to direct gaze (-3.2) (Figure 2).

An additional planned contrast of the interaction between emotion and gaze direction did not show significant activity within the left and right fusiform gyrus region. However, contrast estimates again suggested an interaction of emotion and gaze direction in the left and right fusifrom gyrus. Within the left fusiform area, the anger condition displayed little difference between direct gaze (5.6) and averted gaze (4.4) direction. On the other hand, the fear condition elicited higher contrast estimates for averted gaze (5.6) compared to direct gaze (1.6) (Figure 3). The right fusiform gyrus showed the opposite trend, in that the anger condition elicited high contrast estimates for direct gaze (3.4) over averted gaze (1.8), while the fear condition showed little difference between the contrast estimates for direct gaze (1.9) and averted gaze (2.4) (Figure 3).

Discussion

As outlined previously, we hypothesized that the approach-oriented emotion of anger would elicit higher amygdala activity when combined with direct gaze, where as the avoidance-oriented emotion of fear would elicit higher amygdala activity when combined with averted gaze. The trend of contrast estimates supported this hypothesis as well as provided evidence for the lateralization of this activity.

In line with our prediction for approach-oriented emotions, ROI analyses of contrast estimates revealed that anger elicited more amygdala activity when presented with direct gaze. This finding is consistent with the results of Sato et al. (2010), who found that dynamic presentation of angry expressions directed toward the subject also produced greater amygdala activity. As Adams and Kleck (2005) discuss, direct gaze often communicates dominance or aggression. Furthermore, when exhibiting an approach-oriented emotion such as anger, an individual most often directly approaches or confronts their target, which implies the use of direct gaze.

In accordance with our prediction for avoidance-oriented emotions, averted gaze enhanced the processing of fear by eliciting a greater amygdala response. This result is consistent with research by Hadjikhani, Hoge, Snyder, and de Gelder (2008), who found that the combination of fearful faces and averted gaze elicited significant activity within the amygdala. Their study concluded that averted gaze’s enhancement of fearful expressions were indicative of a threat to one’s environment (Hadjikhani et al., 2008).

Taken together, these results suggest the significance of the amygdala in processing facial expressions of emotion and gaze direction. Furthermore, these findings combined with previous research (Sato et al., 2010; Adams & Kleck, 2005; Hadjikhani et al., 2008) implicate the amygdala’s role in processing social signals that express either approach or avoidance behavioral intent. Our study further predicted a lateralization of this approach-avoidant effect within the amygdala, which was also confirmed through contrast estimates. The left amygdala produced higher responses for anger, where as the right amygdala elicited higher responses for fear. This lateralization of activity is consistent with Harmon and Jones’ (2001) EEG study, which suggested that approach-oriented emotions are more associated with the left hemisphere, while avoidance-oriented emotions are more connected to the right hemisphere.

Our results also revealed a main effect of emotion, with significant activity for anger over fear within the right fusiform gyrus, right cuneus, and left precuneus. These results implicate the significant role the fusiform gyrus may have in the processing of facial expressions of emotion, a finding we explored further. After localizing the fusiform gyrus area, we performed contrast estimates for the left and right fusiform. The left fusiform gyrus was more responsive for the integration of fear with averted gaze, where as the right fusiform gyrus was more responsive for the integration of anger with direct gaze. Research has suggested that the fusiform gyrus is modulated by the presentation of emotional faces and scenes with high social complexity (Hadjikhani & de Gelder, 2003). However, Sprengelmeyer, Rausch, Eysel, and Przutek (1998) indicated that fearful expressions elicited higher activity within the right fusiform gyrus, which is inconsistent with the present study. These results in the fusiform area are also opposite from the trend of contrast estimates within the amygdala. Hadjikhani and de Gelder’s (2003) imaging study indicated there is a connection between the amygdala and the fusiform gyrus in the processing of emotional recognition, though they did not specify if there was a similar opposite effect. Therefore although both regions appear to be linked in processing of emotion, the exact connection remains unclear.

The current study contains some limitations. First, although our results demonstrate the gaze-intensifying effect of both approach- and avoidance-oriented emotions, we tested only negative types within each domain. Sato et al. (2010) used both a positive (happiness) and negative (anger) type of approach-oriented emotions. The present study used a negative emotion of each type in order to maintain consistency, however it would be beneficial to explore the integration of gaze direction and emotional expression for positive types of approach- and avoidance-oriented emotions. Happiness has been tested as an approach-oriented emotion in other research (Sato et al., 2010; Adams & Kleck, 2005), however there is limited use of a positive avoidance-oriented emotion. Calmness has been suggested as an example of this type, although there is no known study of its effect when integrated with gaze direction.

Second, the focus of this study was on the manipulation of gaze direction in the perception of facial emotions of expression. Due to this factor, other emotional cues such as a tightened brow or pursed lips were not considered. This raises the question of whether gaze direction is a separate cue that affects processing of emotional expressions or rather part of a system of indicators that includes other features of the face. Further research should explore methods that would clarify the role of gaze direction within the emotional cueing system.

Finally, our dynamic presentation of facial expression displayed the sequence of a neutral face changing to an emotional face. Future studies are needed to explore whether different sequences of changes within emotional expression, such as beginning with an emotional expression and ending with a neutral face, would subsequently affect amygdala activity. Sato and colleagues (2010) also noted this limitation and discussed how sequential presentation could have greatly contributed to the emotion-intensifying effect of gaze.

Although further work is required to gain a more complete understanding, our findings indicate the amygdala’s role in the integration of gaze-direction with facial expressions of both approach- and avoidance-oriented emotions. Furthermore, our study suggests a lateralization of amygdala activity, suggesting approach-oriented emotions are processed within the left amygdala, where as avoidance-oriented are processed as a function of the right amygdala.

---

References

Adams, R.B., & Kleck, R.E. (2005). Effects of direct and averted gaze on the perception of facially communicated emotion. Emotion 5, 3-11.

Hadjikhani, N., & de Gelder, B. (2003). Seeing fearful body expressions activates the fusiform cortex and amygdala. Current Biology, 13, 2201-2205.

Hadjikhani, N., Hoge, R., Snyder, J., & de Gelder, B. (2008). Pointing with the eyes: the role of gaze in communicating danger. Brain and Cognition, 68, 1-8.

Harmon-Jones, E., & Sigelman, J. (2001). State anger and prefrontal brain activity: Evidence that insult-related relative left-prefrontal activation is associated with experienced anger and aggression. Journal of Personality and Social Psychology, 80, 797-803.

Sato, W., Kochiyama, T., Uono, S., & Yoshikawa, S. (2010). Amygdala integrates emotional expression in response to dynamic facial expressions. NeuroImage 50, 1658-1665.

Sprengelmeyer, R., Rausch, M., Eysel, U.T., & Przuntek, H. (1998). Neural structures associated with recognition of facial expressions of basic emotions. The Royal Society, 265, 1927-1931.

---

Table 1
Brain Regions Showing Significant Activation* for Facial Expression of Emotion and Gaze Direction

Brain Region	BA	Coordinates	T-value	Cluster size
		x	y	z
Main effect of emotion (Greater activation for Anger over Fear)
R. Fusiform gyrus	19	39	-62	-15	4.22	12
R. Cuneus	18	6	-92	16	4.52	151
L. Precuneus	7	-6	-73	50	3.84	23
Paracentral lobule	7	3	-52	66	3.67	10
Cerebellum	-	39	-62	-15	4.22	12
Cerebellum	-	42	-45	-36	4.15	15
Cerebellum	-	53	-60	-27	3.86	13
Main effect of gaze direction (Greater activation for Averted over Direct)
R. Cuneus	18	6	-93	10	6.44	21

*All the brain regions reported here are significant at p < 0.001 with threshold voxel cluster size 9.

Figure 1.

Representative examples of anger and fearful expressions with direct and averted gaze direction. The direct and averted gaze direction were recorded simultaneously by video camera located in front of the model and at 45º angle to the model’s right.

Figure 2.

Contrast estimates emotions as influenced by gaze direction in the left and right amygdala.The error bars represent 90% confidence interval. The left amygdala displayed an effect of anger, showing higher contrasts for direct orientation compared to averted orientation. The right amygdala displayed an effect of fear, with greater contrasts for averted orientation compared to direct orientation.

Figure 3.

Contrast estimates emotions as influenced by gaze direction in left and right amygdala. The error bars represent 90% confidence interval. The left fusiform area displayed little difference in gaze orientation for the anger condition, where as the fear condition resulted in higher contrast estimates for averted gaze compared to direct gaze. The right fusiform area showed little difference due to gaze orientation for the fear condition, while the anger condition elicited higer contrast estimates for direct gaze over averted gaze.