Post on 24-Feb-2023
The Power of Music Valence: Cross modal
transference of valence from the auditory to the
visual domain
By
Natalia Drosogianni
Abstract
The current study aims to examine cross modal transference of
valence from the auditory to the visual domain, conducting a
simultaneous presentation task with 40 participants.
Furthermore, what is of particular interest in this experiment
is the attempt to bring in the relationship the aspect of
time, as well as the aspect of alexithymia, as a potential
factor mediating the relationship. It is assumed that on the
early stage of the task emotions have been induced but on the
late stage, these emotions have been accumulated to create
mood, that is aggregated emotions, the effects of crossmodal
transfer will be greater. Results however, were non-
significant for all factors, so the null hypotheses were
accepted.
Introduction
Since the ancient times, music has been used as an effective
communicator of emotions. For the ears of the listener, music
can induce a range of emotions and moods, such as anger, fear,
joy, relaxation and so on. It was only in the recent years
however, that researchers started to systematically
investigate this special property of music to influence
emotions, and an even more recent tendency was to investigate
music in a multisensory context, which is closer to real-life
emotional processing where stimuli are not presented in
isolation. In such settings, emotions regarding a current
stimulus may not be solely affected but the contents of it,
but rather, they can be modulated by another affective
stimulus, described as prime stimulus. This process is known
as affective priming (Marin, Gingras, and Bhattacharya, 2011).
Research in the field makes a distinction between musical
emotion, and musical mood, a distinction that is not so clear.
Opinions upon researchers of the field vary greatly, but
typically, mood refers to “an affective states that feature a
lower felt intensity than emotions, that do not have a clear
object, and that last much longer than emotions”, emotions on
the other hand are defined as “relatively intense affective
responses that usually involve a number of sub-components-
subjective feeling, physiological arousal, expression, action
tendency, and regulation- which last minutes to a few hours”
(Marin and Bhattacharya, 2009; Juslin and Vaststfjall 2008, p.
561). Qualitatively, in studies involving longer presentation
of the priming stimuli, it could be said that what is under
examination is mood induction rather than emotion induction.
The study of multisensory integration examines how two
different stimuli from separate sensory modalities are bound
into one single percept. Typically in the research paradigm
conducted in the field, the an auditory stimuli is used as a
prime, and the visual image presented shortly after the prime
is called a target. What is measured in such studies is
typically an evaluation of the target, after participants have
been subjected to different primes. All the more studies show
that cross modal emotional transfer needs to be treated as a
multidimensional phenomenon. This tendency follows the trend
of a more naturalistic approach to information processing.
Crossmodal transfer as defined by Marin, Gingras, and
Bhattacharya refers to “the measurable effects that one
emotional stimulus has on the processing of an emotional
stimulus coming from a different sensory domain”. Over the
years many methods have been used to study transfer of
perceived or felt emotions. From self report measures, to the
paradigm whereby two stimuli that are presented consecutively
have to be congruent or incongruent with respect to at least
one variable under investigation.
Studies on film music are a characteristically example of mood
induction since they normally involve longer stimuli. Studies
following a categorical approach to emotion examining the
simultaneous presentation of musical and visual stimuli have
shown clear cross modal emotional influences in term of
valence. However, as it seems these effects were not valid
for all emotion categories.
In 2006, Baumagartenr, Esslen, and Jancke combined a set of
complex affective pictures from the IAPS that were classified
as fearful, sad, or happy with a set of classical music
excerpts of 70second length from the same emotional
categories, and also presented the stimuli in isolation. The
researchers concluded that participants reported increased
levels of involvement in the combined conditions.
Physiological data showed that in the emotional congruent
combined condition, emotions were more effectively induced,
than in the separate music and separate pictures conditions.
Furthermore participants reported increased emotional
involvement in the combined, and music conditions compared to
the picture condition. The findings however do not hold for
the emotion of fear, as the induced quality of fear and
valence, in the combined condition did not vary significantly
from the picture condition.
Similarly, Kutas, Urbach, Altenmuller, and Munte (2006),
presented participants with two stimuli from the visual or
auditory modality. Participants were asked to ignore one of
the two modalities when rating the perceived emotion expressed
by the other modality. This time, auditory stimulus was an
emotional voice (sung notes) that belong to either the sad,
the happy, or the neutral emotional category, and IAPS
pictures. Findings indicated that only in the condition where
the participants had to ignore the pictures and had to attend
to sounds, for the congruent pairings involving happy and
neutral stimuli valence was indeed more strongly perceived.
This result did not hold for the condition where listeners had
to attend to the pictures or for the condition of the sad
pairings.
Perceptual effects in cross modal presentation have been
demonstrated numerous times n psychophysical research. For
instance, Spelke (1979) presented children with a film in two
conditions. Tin the first one, the film was accompanied by an
appropriate soundtrack, while in the second condition the film
was accompanied by an inappropriate one. Results showed that
children attended significantly more to the film with the
appropriate soundtrack. This effect has been explained using
various theories. What is of essence in all these paradigms is
the sensitivity to invariance which is core to theories such
as the Gestalt theory, information theory, and information
pickup. Congruence between internal structure of music and
visual stimuli alters the attentional strategy employed by the
receiver, and consequently the subsequent encoding of
information of the film. It follows that associations and
connotations from the music bias the listener, setting the
context in which later interpretation of visual stimuli is
interpreted (Marshall S.K, Cohen A.J., 1998). Form, affect,
and connotative nature of the structure of information
contained in music affects the interpretation of music, and
therefore any other stimuli primed by music. In their study
(Marshall S.K, Cohen A.J., 1998) argue that the manner whereby
the interaction arises can be categorized in three distinct
processes by the generation of the meaning of music, by the
information carried, the selective attention which prompts
congruencies in the music and film, and finally, the
association between the meaning of the auditory stimuli and
the attended film items.
“Music is a sequential stimulus for which meaning is achieved
over time” (Marshall S.K, Cohen A.J., 1998) “the assignment of
ascent to events will affect retention, processing, and
interpretation”. This feature of music has lead to the logical
assumption that as time goes by the effects of the structure
of a piece are accumulated, creating stronger congruency
effects.
Research for multisensory integration of musical stimuli
simultaneously presented with static visual stimuli in cross
modal emotional transfer has mainly focused on the role of
valence and arousal, has been largely neglected (Fazio et al.,
1986). This could be partly attributed to the common use of
verbal stimuli in earlier affective priming studies as
discussed above, which as it is expected, vary more in valence
than in intensity. However, a recent trend in the field
involves studying how film, as an art form comprising of both
musical and visual stimuli, can have arousal associated with
music modulate levels of emotion expressed by visual
materials, as well as the meaning of these visual material
(Cohen, 1998). It is widely though that film music can affect
the emotional experience of the film
Following evidence from cognitive studies that the
relationship between film and music is additive Ellis and
Simmons (2005) examined how music affects the perception of
film in terms of emotion self report and physiological
reactions. The experimenters used eight tracks of instrumental
art music of 270-s varying in valance and arousal overlaid
with eight films. Researchers concluded that even though self
reports show a direct and clear cut additive relationship
between music and film, the physiological data indicate a more
complex one.
Expanding on the connotative and structural attributes of
music, and the relations among tones, Cohen conducted an
experiment involving music and film, bringing together the
aspects. It is true that music, as well as visual stimuli can
generate impressions such as aggression or warmth by specific
aspects of its structure that bring to the mind of the
receiver of the stimuli connotative attributes (Marshall S.K,
Cohen A.J., 1998; Levi, 1982; Meyer, 1956). The film selected
for this study was a 2 minute abstract animation of Heider and
Simmel (1944) depicting movements of geometric shapes, which
were interpreted as aggressive by almost all viewers prior to
the main experiment. Te researchers tested the hypothesis that
two musical excerpts that differ on a particular dimension
will influence the judgments of meaning og the film on that
specific dimension. The dimensions under examination were
evaluative (good/bad), potency (weak strong) and activity
(active, passive). The pattern of results indicated that the
meaning of the music for the dimensions of potency and
activity was directly associated with the meaning if the film.
An inverse relation however was observed for the evaluative
dimension, which the experimenters attributed to the
complexity of the interaction of film and music for the
specific dimension, and the type of cognitive congruency that
affects this relation (Marshall S.K, Cohen A.J., 1998).
Typically, individual differences are not involved in research
of crossmodal transfer, however, in the recent years, there
has been expanding interest in the trait of alexithymia, which
relates directly to emotional perception. Studying such a
trait could prove very interesting for the field of crossmodal
transfer of arousal, since, this emotional deficit can serve
to explain the ways in which the transfer of arousal occurs.
Alexithymia was originally related to clinical environments in
disorders characterized with poor social functioning. The
personality construct represents a deficit in cognitive
processing, one that has been especially linked with
somatisation. It is characterized by difficulties in
identifying, differentiating, and describing feelings, as well
as by an important component of externally oriented thinking.
A 2008 study by Karlson, Naatanen, and Stenman, manipulated
for mood induction using films of positive, negative and
neutral valance, while emotional activation was measured in 10
healthy women with alexithymia, and 11 healthy women with no
alexithymia. Results demonstrated clear effects of
differential mode of emotion processing by the two groups.
Women with alexithymia showed a tendency to over-activate
their “bodily” brain regions, as well as reporting self
observed differences in experiencing these emotions such as
difficulty in identifying their emotions.
Overview of the present experiment
The central aim of the current experiment was to examine if
and how could pleasantness induced from music can be
transferred to the visual domain, affecting therefore ratings
of the pleasantness of emotional faces. The present experiment
sought to explore the emotional response to the visual stimuli
by collecting ratings of felt pleasantness, also exploring
possible effects of alexithymia. Of essence to the research
was to select musical pieces that significantly varied in
terms of valence, or else pleasantness. Visual stimuli were a
set of facial expression that also varied in pleasantness. In
light of the current literature on cross modal emotional
priming and the study of the limited research on multisensory
integration, it was hypothesized that musical primes would
affect greatly the participants’ subjective ratings of the
facial expressions in a manner congruent with the valance of
the musical piece. Another important area of research for the
present study however was to also brin in the aspect of time
in this relationship. It is assumed that on the early stage of
the task, emotions have been induced but on the late stage,
these emotions have been accumulated to create mood, that is
aggregated emotions. No clear cut definition of emotion and
mood has been offered by past research on the field of cross
modal study, but in this particular study, mood and emotions
are not treated as different variables, but rather emotion is
what is studied, with the difference that in the late stage
emotion is seen as being accumulated after 2.5 minutes of
listening to the musical excerpt, and becomes what is most
widely known as “mood”.
While examining how valance from the auditory domain
differentially affects perceived valence in the visual domain,
and what is the interaction of time in this relationship, a
third variable was thought to be of particular interest in
this cross examination. This other variable is alexythimia.
Since the that I so highly correlated with emotions, it is
only logical to assume that it must at some part at least
mediate the relationship between the interaction of the
auditory and the visual domain. The hypotheses the
experimenter sets for this study can be grouped into
categories. Firstly, the level of valance of the musical piece
will significantly affect subsequent ratings of the
participants’ evaluations of the facial expressions. It is
expected that this effect of congruency will be very strong,
but the effect of this musical priming will be observed the
most in the neutral emotional valence condition, where the
participants are expected to be more likely to evaluate the
ambiguous face depending on the valance level of the musical
excerpt he/she is given. The second hypothesis made is that
this congruency effect will become stronger in the later
stages of presentation. Simply put, it is assumed that with
time, the effects of positive valance will be aggregated for
the high valance facial expression condition, in the same way
negative valance effects will be aggregated for the low
valance condition. Finally, the trait of alexithymia, as
observed in non-clinical populations, is hypothesized to
significantly affect the interaction between valance from the
auditory domain and ratings of the facial expressions, so that
participans scoring high in alexithymia facets will be less
affected from the valance of the musical piece, and
participants that show low alexithymia traits will be affected
by music in a greater extent.
Method
Design
The experiment had a 2 x 3 x 2 within-participants design with
type of musical valence (positive vs. negative), type of
emotion of facial expression (happy vs. sad vs. neutral), and
the time stage (early stage: from 0 to 2.5 minute vs. late
stage: from 2.5 to 5 minutes) as factors. The dependent
variables (DV) was the participants’ ratings on the valance of
the emotion of the facial expression on a scale from
1=extremely happy to 7=extremely sad. Furthermore,
participants’ scores on a self-reported alexithymia scale
(TAS-20) was used as a between subjects factor.
Subjects
A total of forty university students (27.5 males, 72.5
females) aged between 19 to 34 (Mean age: 21.83, SD= 3.57)
participated in this study. The majority of the participants
were first-year Goldsmiths undergraduate students. They were
obtained through the Psychology Department’s Research
Participation Scheme and took part in exchange for course
credits. Participants were subjected to all levels of the
three independent variables, while for the variable of musical
valence the order was reversed for helf of the participants
for counterbalancing purposes. All participants were living in
the UK for at least the past 3 years and were encultured to
western tonal music. The sample reported normal hearing and
had no neurological or psychiatric disorder. All participants
gave their informed consent to participate. Ethical approval
from the ethics committee of Goldsmiths University of London
was received prior to commencing this research.
Measures and Materials
The task was programmed in MATLAB version 7.10.0 software
(Natick, Massachusetts: The MathWorks Inc., 2010) and was
displayed on a computer.
The task consisted of two musical pieces of 5 minutes length
each, and a set of Nimstim facial expressions (The Research
Network on Early Experience and Brain Development).
a) Musical Pieces:
The two musical pieces were chosen on the basis of dimensions
of arousal and valance. Both stimuli were carefully selected
so that they would be of high arousal and they should only
vary on valance.
For the positive valance mood induction Bach’s Brandenberg
Concerto No. 3 played by Hurbert Laws was used, while for
negative valance mood induction Prokoviev’s Alexander Nevsky:
Russia under the Mongolian Yoke was administered. Both these
musical pieces have been validated from previous mood research
to significantly induce increased and decreased positive
affect respectively on an initial slightly positive mood
(Rowe, Hirsh,and Anderson, 2007)
b) Nimstim set of facial stimuli
A set of face stimuli called the Nimstim Set of Facial
Expressions was used as the second independent variable.
Advantages of the set are that it contains a large multiracial
sample of photos of actors, who are asked to perform naturally
occurring facial expressions. The facial expressions used from
this set for the purposes of the current experiment were those
depicting happy, neutral and sad expressions (see sample
images below). The set has been tested upon a sample of
untrained individuals in a 2009 study by Tottenham et al. and
showed high validity and reliability as well as high intra-
participant agreement across two testing sessions (Tottenham
et al., 2009).
a) b)
Figure 1. Examples of facial expressions of a) sad, and b) happy facial expressions from the NimStim Face Stimulus Set
Courtesy of The Research Network on Early Experience and BrainDevelopment
c) Alexithymia Measure
The Toronto Alexithymia (TAS 20) scale was administered to
participants after the completion of the task to estimate the
levels of alexithymia for each participant (Appendix II). The
scale is the most widely used instrument for assessing
alexithymia in both research and clinical practice. The TAS-20
comprises of three factors, namely; externally oriented
thinking, difficulty identifying feelings, and difficulty
describing feelings. Although it could be argued that a self
report scale involves impairments in self awareness, the
scale has been assessed upon a sample of 1933 individuals
showing replicability of the factor structure as well as
internal reliability (Parker, Taylor, and Bagby, 2003).
Procedure
All participants were tested individually in a quiet cubicle
with no external disturbances. They were firstly asked to
carefully read and sign the consent form (Appendix I). After
that they were seated approximately 40 cm in front of the
computer screen which was set at eye level. During the taskt
participants were asked to listen to the two excerpts of music
lasting 5 minutes each. The experiment was divided into two
tasks, which the participants are asked to complete. In the
first task participants listened to the high arousal auditory
stimulus, during this time, they were presented with 100
randomized facial stimuli from the Nimstim, the presentation
of which lasted for 1 second. After this time the participants
were given 1 second in which they were supposed to provide a
rating of the valance of the emotion presented on a Likert-
type scale from 1 to 7 (1 being extremely sad, 4 being
neutral, and 7 being extremely happy). Their response was
given simply by pressing the number they think the emotion
represents on the keyboard. After this process was repeated
for both songs differing in valance, participants were given
the same musical excepts, however, this time at the point
where in the previous task the visual stimuli appeared, a
fixation cross appeared in the centre of the screen, where the
participants were asked to rate perceived level of valance for
this specific moment of time. The order in which the
participants listened to each song for both tasks was
counterbalanced as to avoid order effects. In total, all four
tasks lasted for approximately 25 minutes. After the two main
asks were finished, participants were given a questionnaire
containing demographics, questions upon the perceived
pleasantness for the two songs, and a short standardized
questionnaire for alexithymia (Appendix II). When participants
finished completing the questionnaire they were fully
debriefed about the aim and the nature of the experiment.
Results
To evaluate emotional spaces, the average and range of all
ratings for the happy, sad, and neutral emotional expressions,
as well as the average and range for both the two musical
pieces were calculated.
Table 1.Descriptive statistics for the perceived valance
of each emotional category of faces.
Minimum Maximum MeanStd.
DeviationHappy Face 4,47 6,82 5,8312 ,43794
Neutral Face 2,03 4,22 3,6294 ,40961
Sad Face 1,03 3,82 2,2343 ,61114
Valid N (listwise)
Table 2. Descriptive statistics for the perceived valance of
each auditory stimuli
N Minimum Maximum MeanStd.
DeviationPositive Music 40 3,00 7,00 5,7250 ,87669Negative Music 40 1,00 5,00 2,5500 1,25983Valid N (listwise)
40
Perceived emotional ratings were assessed using factorial
repeated-measures analysis of variance (ANOVA) with the
factors of valance of musical emotion (two levels: positive
and negative), the valance of facial emotion (three levels:
happy, sad, neutral), the time stage on which the rating takes
place (early stage, late stage) as within subjects factors,
and alexithymia as a between subjects factor (alexithymia or
non-alexithymia)
A) The main interaction between music valence and the
subjective ratings of the valence of presented facial
expressions
Table 3. The mean ratings and standard deviations of the
valance of emotional faces (on a scale from 1=extremely sad to
7=extremely happy), with respect to the valance of the musical
piece.
Music Valence/ Expression of Emotional Face
Mean
Std.Deviatio
n NPositive music, happy face 5,867 ,427 40
Positive music, sad face 2,262 ,622 40
Positive music, neutral face 3,629 ,412 40
Negative music, happy face 5,796 ,460 40
Negative music, sad face 2,207 ,606 40
Negative music, neutral face 3,630 ,413 40
Figure 1. The mean ratings of the valance of facial
expressions (on a scale from 1=extremely sad to 7=extremely
happy), with respect to the valance of the musical piece.
Happy Sad Neutral0
1
2
3
4
5
6
7
Positive musicNegative music
Expression of emotional face
Rati
ngs
of p
osit
ive
vala
nce
The means and standard deviations are presented in Table 1.
Mauchly’s test indicated that the assumption of sphericity had
been violated for the main effect of valence of emotional face
factor, χ²(2)=10,125, p= .006, and for the main interaction
between the valence of the emotional face factor and the music
valence factor χ²(2)= 6,262 p=.044. Therefore degrees of
freedom were corrected using Greenhouse-Geisser corrected
estimated of sphericity (ε=.81 for the main effect of
emotional face and .87 for the main interaction between
emotional faces and musical valence).
The two-factor analysis of variance showed a significant main
effect of the type of expression of the emotional face on the
valance ratings of the facial expression, F(2,78) = 477,73, p<
.001, but no significant main effect for the music valence
factor, F(1,39) = .700, p > .05. The interaction between the
valence of the emotional face factor and the music valence
factor not significant as well, F (2,78) = .750, p > . 05.
B) The effect of time in the interaction of the music
valence on the subjective ratings of the valence of
presented emotional faces
Table 4. The means and standard deviations for the interaction
of music valence on the rating of facial expressions (on a
scale from 1=extremely sad to 7=extremely happy) with respect
to the time stage (early time stage=0-2.5 minutes, late time
stage=2.5 minutes to 5 minutes).
Music Valance
Expression of emotional face Time Stage Mean
Std.Error
Positive Happy Early 5,84 ,079
Late 5,89 ,066
Sad Early 2,23 ,116
Late 2,32 ,096
Neutral Early 3,62 ,070
Late 3,65 ,067
Negative Happy Early 5,81 ,075
Late 5,80 ,075
Sad Early 2,18 ,107
Late 2,25 ,092
Neutral Early 3,65 ,069
Late 3,61 ,069
Figure 2. The mean ratings of valence of facial expressions
(on a scale from 1=extremely sad to 7=extremely happy) with
respect to the time stage (early time stage=0-2.5 minutes,
late time stage=2.5 minutes to 5 minutes) for the musical
pieces of positive and negative valance.
Happy Sad Neutral Happy Sad NeutralPositive Negative
01234567
Early Late
Expression of emotional face
Rati
ngs
of p
osit
ive va
lanc
e
The means and standard deviations are presented in Table 2.
The three-factor analysis of variance (3 by 2 by 3 ANOVA)
showed no significant main effect for the interaction of time
with the valance of music and the rating of the emotional
faces, F(2,78) = .084, p > .05.
C) The effect of alexithymia on music valence and the
subjective ratings of the valence of presented facial
expressions
Table 5. The mean and standard deviation of the effect of
musical valance on ratings of the facial expressions with
respect to the alexithymic state of the participant.
Non alexithymia
Std. Deviation
Alexithymia
Std. Deviation
PositiveMusic
Happy Face 5,93
,365,76
,52
Sad Face 2,21
,572,34
,71
NeutralFace 3,68
,393,55
,44
NegativeMusic
Happy Face 5,88
,435,67
,47
Sad Face 2,1
,582,37
,64
NeutralFace 3,63
,433,63
,40
N=25 N=15
Figure 3. The mean and standard deviation of the effect of
musical valance on ratings of the facial expressions with
respect to the alexithymic state of the participant.
Happy Face
Sad Face Neutral Face
Happy Face
Sad Face Neutral Face
Positive Music Negative Music
01234567
Alexithymia Non-alexithymia
Expression of emotional face
Rati
ngs
of p
osit
ive
vala
nce
Although it clear these are some small effects, the mixed
ANOVA used reveals alexithymia is not a strong factor that
will have an effect on the relationship between the valance of
the musical piece and the rating of the emotional faces.
The valance of the musical piece X valance of the facial
expression X alexithymia interaction was non-significant
F(2,76)=0.43.
Discussion
The current study provides evidence of a dissociation of the
emotional transfer of valance from the auditory to the visual
domain. The results obtained are in contrast with past
research on the filed, that has used similar paradigms, the
majority of which have found strong associations of transfer
of valance from the auditory to the visual domain (Logeswaran
and Bhattacharya, 2009 ; Ellis and Simmons, 2005). In their
2009 study, Logeswaran and Bhattacharya used expressive faces
as visual targets, and found a significant effect of musical
priming on the perceived happiness of the face presented..
Also, electrophysiological data showed event related brain
potential component at a very early stage of neuronal
information processing. In a study by Britton et al. 2006,
participants viewing expressive faces showed increased
activation of the superior temporal gyrous, insula, and the
anterior cingulated, all of which are areas of the brain
associated with emotional responses to music (Marin et al.
2011; Britton et al. 2006). The researcher also argues that as
a social phenomenon, music can strongly affect perception of
emotional expressions, which are inherently more social than
complex affective pictures. Therefore, in the current
experiment, the choice of facial emotions as visual stimuli,
instead of complex affective pictures for example, is
justified and supposedly could not account for the
insignificance of the results.
Results are in line with Chen et al. 2008, and Marin et al.
2011, who both found dissociation between transfer of valance
from the visual to the auditory domain. In the research
conducted by the latter, the authors argue this dissociation
arises from the difference in the variance of the prime and
target stimuli. A similar procedure of pre-evaluation of
emotional spaces was carried out for the present experiment
as well, indicating a very high variance reflected in the
auditory stimuli, compared to a relatively small variance in
the visual stimuli (Tables 1 and 2), for which mean evaluation
were very close, especially regarding sad and neutral facial
expressions.
This could be possibly attributed to the face set. In the
experimenting session, participants would often comment upon
the end of the task that it seemed like the model of the
picture was exaggerating, especially for the sad face,
creating a somewhat “grotesque” depiction of the facial
expression, that the participant received as not being
actually sad. Although a slight pulling down of the lip corner
is present other characteristics relating to sadness such as
lost focus of the eyes and dropping upper eyelids are not
present. It is argued that actors exaggerate aspects of
naturally occurring expressions which can lead to confusion by
the viewer (Tottenham et al. 2009). The Perhaps, since the
emotional aspects of the musical piece reflect differences in
the valence, while high arousal levels remain equal, the
effect of crossmodal transfer could be studied for facial
emotions of equal dimensions, that is happy, perhaps with open
mouth detail, or exuberant, and angry, which reflects an
emotion of high arousal and low valence, in contrast to
sadness, which reflects an emotion of low valance but low
arousal as well. It could be possible that this comparison
could provide a significant effect for the interaction between
musical valance and emotion of facial expression.
One of the main aims of this experiment was to address the
effect of time. It was hypothesized that in the later time
stage, emotions would have been accumulated, creating an even
larger effect of congruency, so that happy emotional
expressions would be perceived as being more, happy, sad as
being more sad, in the emotionally congruent situations, and
neutral faces were thought to follow the emotional valance of
the song. As it can be seen from figure 2, even though there
is a small effect of elevated congruent emotions, for both the
neutral and happy emotional categories, this effect is non-
significant. What is of particular interest however is that
from time stage 1 to time stage 2 for the negative valance
music the ratings of the sad faces show a minor increase
instead of a minor decrease that would be expected. Even
though again, this responds to a non-significant observation,
the effect could be explained in terms of negative emotion
bias that is that negative unpleasant emotions induce higher
arousal levels (Marin, Gingras, and Bhattacharya, 2011)
Alexythimia, was a factor also suggested by the experimenter’s
hypothesis to mediate between the crossmodal transfer of
valance from the auditory to the visual domain. As the mixed
ANOVA revealed however, the effect of this personality
construct is also non-significant. A close look of figure 3
will reveal small differences between the conditions of
alexithymia and non-alexithymia, however, the lack of any
significant effect can be attributed to the
underrepresentation of individuals scoring high in
alexithymia.
The overall non-significant results can be attributed to the
nature of the visual stimuli, which was static. Perhaps, using
a film as the visual stimuli, can be argued to create a more
prolonged emotional response, since it would stimulate real-
life emotional situations better that the stimuli of static
nature.
Another limitation includes the overrepresentation of women
and underrepresentation of men. Women have been shown to show
increased negativity bias as well as other characteristics
that may have hindered the results of the experiment. A more
equal representation of the two sexes could have been more
reliable, and would probably had yielded interesting results
for comparing the two.
The current study explored the emotional spaces of primes and
targets on the same rating scales, providing a clearer
interpretation of the results. Since the emotional spaces have
been established, future research on crossmodal emotional
transfer between the auditory and the visual domain should be
focused on replicating the procedure by combining various
other types of visual and auditory stimuli varying in
emotional valance and semantic contents.
It is also suggested that larger samples could be of vital
importance as they could extenuate the effect of congruency,
and be able to show a significant interaction between the typr
of musical valance, the perception of facial emotion, time,
and alexithymia.
Following research paradigms of Ellis and Simmons (2005),
Logeswaran and Bhattachaya(2009), who focused not only on
behavioral data but also physiological measures as objective
indicators of emotion, as well as the recent Karlson, Naatanen
and Stenman (2008) experiment, that accounted for cortical
activation as response to emotional stimuli in alexithymia,
it is clear that research should be directed into
physiological data, which provide a more rich and truthful
account as to what are the effects of cross modal transfer of
valance from the auditory to the visual domain.
One of the dominant research questions that would also be of
great interest to answer is how a lifetime of intensive
musical training changes the brains of musicians compared to
non-musicians. Imaging studies have found that musicians have
increased gray matter in auditory, and increased connectivity
between the two hemispheres via the corpus callosum (Robert J.
Ellis, 2011). Electrophysiological studies have revealed that
musicians show enhanced cortical representation of musical
stimuli, speech stimuli, and emotional vocalizations (Robert
J. Ellis, 2011).These results suggest that a lifetime of
musical training does not just selectively enhance sensitivity
to music itself, but has facilitatory transfer effects into
broader cognitive processes such as attention, language
processing, and memory. For these reasons, it is argued that
an invaluable insight would be provided by comparing the
ratings of musicians and non-musicians in the experimental
paradigm used in the present study. This is something future
research should focus upon.
At last, future research could focus on other dimensions of
emotions such as arousal, or include other personality
measures as well, that could be shown to relate to the
interaction of valance from the auditory to the visual domain.
Conclusions
There is reasonable evidence to suggest that there are
significant overlaps between musical emotions and visual
emotions. The present study aimed to examine whether beyond
this overlapping, musically induced emotions could affect the
interpretation of visually induced ones. Despite past research
that reports a significant transfer of arousal from the
auditory to the visual domain, in the present experiment did
not yield any significant results regarding this interaction.
Also, both further null hypotheses were accepted regarding the
effect of time in this interaction, which was found to be non
significant, as well the effect of alexithymia which was also
found to be non-significant.