Sound symbolism in infancy: Evidence for sound–shape cross-modal correspondences in 4-month-olds

Journal of Experimental Child Psychology 114 (2013) 173–186

Contents lists available at SciVerse ScienceDirect

Journal of Experimental ChildPsychology

journal homepage: www.elsevier .com/locate/ jecp

Sound symbolism in infancy: Evidence for sound–shapecross-modal correspondences in 4-month-olds

Ozge Ozturk ⇑,1, Madelaine Krehm ⇑,1, Athena VouloumanosDepartment of Psychology, New York University, New York, NY 10003, USA

a r t i c l e i n f o a b s t r a c t

Article history:Received 23 March 2012Revised 8 May 2012Available online 7 September 2012

Keywords:Sound symbolismSynesthesiaLanguage acquisitionInfant perceptionCross-modal perceptionPreferential looking

0022-0965/$ - see front matter � 2012 Elsevier Inhttp://dx.doi.org/10.1016/j.jecp.2012.05.004

⇑ Corresponding authors.E-mail addresses: [email protected] (O. Ozt

1 Joint first authors. The first two authors contrib

Perceptual experiences in one modality are often dependent onactivity from other sensory modalities. These cross-modal corre-spondences are also evident in language. Adults and toddlers spon-taneously and consistently map particular words (e.g., ‘kiki’) toparticular shapes (e.g., angular shapes). However, the origins ofthese systematic mappings are unknown. Because adults and tod-dlers have had significant experience with the language mappingsthat exist in their environment, it is unclear whether the pairingsare the result of language exposure or the product of an initial pro-clivity. We examined whether 4-month-old infants make the samesound–shape mappings as adults and toddlers. Four month-oldsconsistently distinguished between congruent and incongruentsound–shape mappings in a looking time task (Experiment 1). Fur-thermore, mapping was based on the combination of consonantsand vowels in the words given that neither consonants (Experi-ment 2) nor vowels (Experiment 3) alone sufficed for mapping.Finally, we confirmed that adults also made systematic sound–shape mappings (Experiment 4); however, for adults, vowels orconsonants alone sufficed. These results suggest that somesound–shape mappings precede language learning, and may in factaid in language learning by establishing a basis for matching labelsto referents and narrowing the hypothesis space for young infants.

� 2012 Elsevier Inc. All rights reserved.

c. All rights reserved.

urk), [email protected] (M. Krehm).uted equally to this work.

http://dx.doi.org/10.1016/j.jecp.2012.05.004

mailto:[email protected]

mailto:[email protected]

http://dx.doi.org/10.1016/j.jecp.2012.05.004

http://www.sciencedirect.com/science/journal/00220965

http://www.elsevier.com/locate/jecp

174 O. Ozturk et al. / Journal of Experimental Child Psychology 114 (2013) 173–186

Introduction

The mapping between labels and referents has been considered arbitrary in mainstream linguistics(De Saussure, 1916/1983). However, research findings suggest some systematicity between specificlabels and referents; adults and toddlers spontaneously and consistently map particular shapes to par-ticular words even when there appears to be no obvious physical basis for the mapping (e.g., Köhler,1947; Maurer, Pathman, & Mondloch, 2006). However, because adults and toddlers already havesignificant experience with language mappings in their environment, it is unclear whether thesebiases are the result of an acquired sensitivity or an initial proclivity. Here we investigated the originsof these sound–shape mapping biases by examining whether 4-month-old infants share the samesound–shape mapping biases as adults and toddlers.

A fundamental tenet of modern linguistics is that there is no systematic relationship between lin-guistic labels and the meaning they convey (De Saussure, 1916/1983). Some have even proposed thatthis arbitrary connection between linguistic labels and their referents is a fundamental feature of hu-man language (e.g., Monaghan, Fitneva, & Christiansen, 2011; Ramachandran & Hubbard, 2001) andthe basis for its referential power (e.g., Gasser, 2004). Moreover, if the sound structure of wordswas related in a systematic way to their referents, we would expect similarities across languages, withsimilar inventories of sounds corresponding to similar types of meaning. This is clearly not the casebecause even a simple concept such as tree is realized with phonologically distinct words in differentlanguages—agaç in Turkish, zuhaitza in Basque, árbol in Spanish, and dendro in Greek. Indeed, most lin-guistic labels have different sound structures in different languages; these labels are conventionsagreed on by the speakers of the languages with no apparent systematic relationship to their referents(Hockett, 1977).

However, not all label–referent mappings appear to be arbitrary. Cross-linguistic observations sug-gest that there are words in natural language whose sounds are systematically related to their mean-ings. Ideophones—words that are used by speakers to evoke vivid associations with particular sensoryperceptions (e.g., smell, color, shape, sound, action, or movement across languages)—are widely at-tested in the languages of the world (Voeltz & Kilian-Hatz, 2001). West African, East Asian, and South-east Asian languages, and to a lesser extent Amerindian languages, are known for their largeinventories of ideophonic vocabulary, but many other languages also make use of ideophones (e.g.,Turkish citir citir ‘crispy’; Basque mara mara ‘falling softly, said of rain’; Ewe gbadzaa ‘flat, spreadingout over a wide area’; English bling bling ‘glitter, sparkle’). However, the sounds of these words evokeor imitate the meaning, creating systematic nonarbitrary connections.

People also systematically map particular speech sounds to properties of objects cross-linguisti-cally. For instance, across languages, there are phonetic classes of speech sounds that are systemati-cally found in vocabulary related to size (e.g., /i/ vs. /a/ for size as in Ewe kitsikitsi ‘small’ vs.gbaggbagba ‘big’, Greek micros ‘small’ vs. macros ‘large’) or distance (e.g., words for here are more likelyto include an /i/ sound and words for there are more likely to include an /a/ sound; see Tanz, 1971).Experimental findings also support this idea of sound–size relations. Syllables containing low backvowels (e.g., mal) are consistently matched to large objects, whereas syllables containing high frontvowels (e.g., mil) are consistently matched to small objects (Sapir, 1929). Recently, cross-modal map-ping of sound and size has also been demonstrated in infants, with 4-month-olds matching [o] or [a] tolarge objects and [i] or [e] to small objects (Peña, Mehler, & Nespor, 2011). Thus, systematic mappingof sound to size is widespread and observed even during infancy.

Cross-modal correspondences between sound and size have been noted for centuries (Descartes,1641/1986; Gibson, 1966; Ohala, 1997; Walker et al., 2010) and appear to be grounded in physicalreality. High front vowels such as [i] and [e] tend to have higher fundamental frequencies (F0) thanlow back vowels such as [o] and [a] (Whalen & Levitt, 1995). At the same time, objects that are phys-ically thinner or smaller tend to produce a higher pitch than wider larger objects. For instance, thepitch of a cello is lower compared with the pitch made by its smaller cousin, the violin. Similarly,the vocal folds of a larger animal are longer, and longer vocal folds tend to generate sounds that arelower in pitch (Shayan, Ozturk, & Sicoli, 2011; Zbikowski, 1998). Peña and colleagues (2011) proposedthat infants may map vowels to object size based on their experience in seeing mouths open to

O. Ozturk et al. / Journal of Experimental Child Psychology 114 (2013) 173–186 175

different extents when humans produce different vowels. The systematic mapping between vowelsand object size appears to be grounded on observable physical relationships between the pitch ofvowels and the size of objects and, therefore, may simply reflect associations based on participants’prior multisensory experiences.

Another type of cross-modal mapping that has been observed in adults, and recently in infants asyoung as 3 or 4 months, is a correspondence between auditory pitch and visuospatial height or visualsharpness (Marks, 1987; Walker et al., 2010). Infants (and adults) map higher pitched tones withsharper objects and objects that appear higher in space. Walker and colleagues (2010) proposed thatthe mapping between sharpness and pitch might be mediated by the hardness of natural materials.Sharp and pointed objects tend to be formed from hard materials, and they tend to producehigh-pitched sounds when struck. These natural correspondences are also grounded on observablephysical relationships and may reflect prior multisensory experience.

However, there are other cross-modal mappings between linguistic labels and their referents thatmay rely less on prior multisensory experience in a phenomenon known as sound symbolism. AdultSpanish speakers consistently matched curvy shapes with the novel word ‘baluba’ and angular shapeswith another novel word ‘takete’ (Köhler, 1947). These mapping biases were later documented inAmerican college undergraduates and Tamil speakers in India, with the vast majority of both groups(as many as 98%) selecting a curvy shape as ‘bouba’ and an angular shape as ‘kiki’, and this effect be-came known as the ‘‘bouba/kiki effect’’ (Köhler, 1947). These mapping biases also affect processing;adults are more accurate and faster at classifying a novel object if the category name matches theshape of the object (Kovic, Plunkett, & Westermann, 2010). Electrophysiological responses also differ-entiate between sound symbolic and non-sound symbolic categories, producing an early neural acti-vation similar to that elicited by highly learned categories, possible evidence of auditory–visualintegration (Kovic et al., 2010). Curvy and angular shapes may be related to the shape of a speaker’slips when producing the vowels (open and rounded or wide and narrowed; see Ramachandran & Hub-bard, 2001). However, this would require attending to and mapping two sets of visual properties oftwo different entities (the object shape and the speaker’s mouth) that are not spatially colocalizedor even always temporally copresent—a much less straightforward mapping than noticing naturalphysical relationships between sounds and objects. Thus, sound–shape correspondences are less eas-ily explained by observable physical relationships but are systematic nevertheless.

Nonarbitrary mappings between sound and shape extends into childhood. When asked to labelKöhler-type curvy and angular drawings with the word ‘takete’ or ‘uloomo’, both English-speakingchildren (11–14 years old) from England and Swahili-speaking children (8–14 years old) from centralAfrica matched the curvy shape with the word ‘uloomo’ and the angular shape with the word ‘takete’(Davis, 1961). This systematic sound–shape mapping has even been demonstrated in 2½-year-olds.Children systematically matched ‘bouba’ to curvier shapes and ‘kiki’ to more angular shapes (Maureret al., 2006).

Both adults and children make systematic cross-modal mappings between linguistic labels andtheir referents. However, the origins of these sound–shape mappings are unknown. Because adultsand toddlers have had significant experience with the statistics of label mappings that exist in theirenvironment, it is unclear whether these systematic pairings are the result of language exposure orthe product of an initial proclivity. Young infants who have yet to develop a lexicon have not been ex-posed to the statistical regularities in their language long enough to be biased by vocabulary-widesystematicity in labels for specific sound–shape patterns; for example, corpus analyses showed thatvelars are found more in words for angular – rather than rounded – objects. (Monaghan, Mattock, &Walker, 2012). Therefore, if infants share the same sound–shape mappings as adults and toddlers,these biases may precede and perhaps aid in word or category learning, narrowing the massive num-ber of possible sound–shape correspondences that learners face (Imai, Kita, Nagumo, & Okada, 2008;Monaghan et al., 2012; Nygaard, Cook, & Namy, 2009; Parault, 2006; Parault & Schwanenflugel, 2006).

To investigate the origins of these biases, we examined whether prelinguistic infants exhibit thesame sound–shape mapping biases as adults and toddlers. Furthermore, we systematically exploredwhether infant mappings are based on a combination of vowels and consonants, vowels alone, or con-sonants alone, and we compared these results with those for adults. In Experiment 1, we presentedinfants with a large red shape that was either curvy or angular, accompanied by a nonsense word,


either ‘bubu’ or ‘kiki’. We predicted that if 4-month-olds who have less language exposure and few, ifany, word–object mappings in their receptive vocabulary (Fenson et al., 1994; see also Bergelson &Swingley, 2012) have the same biases as adults and toddlers, they would look differently at trials con-sidered as congruent by adults (pairing the angular shape with ‘kiki’ or the curvy shape with ‘bubu’)and at incongruent trials (pairing the curvy shape with ‘kiki’ or the angular shape with ‘bubu’). If, onthe other hand, the biases demonstrated by adults and toddlers are developed through significantlanguage experience, 4-month-olds should look equally at incongruent and congruent trials. InExperiments 2 and 3, we examined whether mappings are based on vowels or consonants. Finally,in Experiment 4, we tested adults on these materials and examined their use of vowels or consonantsin making systematic mappings.

Experiment 1

Experiment 1 examined whether prelinguistic infants make the same mappings between nonsensewords and shapes as adults (Köhler, 1947; Ramachandran & Hubbard, 2001) and toddlers (Maureret al., 2006). We presented 4-month-old infants with two sounds: ‘kiki’ and ‘bubu’. These two soundswere paired with a curvy shape or an angular shape. We examined whether infants looked differentlyat pairings that adults and toddlers in prior studies had rated as incongruent (the curvy shape with‘kiki’ and the angular shape with ‘bubu’) than at pairings that had been rated as congruent (the angularshape with ‘kiki’ and the curvy shape with ‘bubu’).

Method

ParticipantsThe participants were 12 full-term 4-month-olds (mean age = 4 months 4 days, range = 3 months

22 days to 4 months 20 days, 5 girls and 7 boys). Infants were recruited at birth from local birthinghospitals. A research assistant subsequently contacted the families to participate in the study. Parentswere compensated for transportation, and infants were given a small gift for participating. Infants’parents filled out a demographic information form and reported a range of educational backgrounds;of the 24 parents, 1 had not completed any high school, 1 had completed some high school, 2 had com-pleted some college, 8 had completed college, 6 had completed graduate degrees, and 6 declined toanswer). Of the 12 infants, 6 were reported as being White, 4 were of mixed ethnicity, 1 was Hispanic,and 1 parent declined to answer. An additional 13 infants were tested but excluded from the analysisbecause of fussiness (6), inattentiveness (2), equipment failure (3),2 or experimenter error (2). Allprocedures were approved by the New York University institutional review board.

MaterialsAuditory stimuli. A female English–Turkish bilingual speaker recorded several productions of ‘kiki’ and‘bubu’ in a sound attenuated room with a Shure SM58 microphone. We decided to use ‘bubu’ insteadof ‘buba’, which had been used in previous studies (Maurer et al., 2006), to match the reduplicativestructure of ‘kiki’. Three ‘kiki’ and three ‘bubu’ tokens were selected and matched on duration (‘kiki’:M = 415 ms, range = 400–425; ‘bubu’: M = 408 ms, range = 400–417) and modified using Praat (Boers-ma & Weenink, 2010) to match on amplitude (all tokens were 70 dB) and pitch (‘kiki’: M = 230 Hz,range = 227–233; ‘bubu’: M = 230 Hz, range = 228–232). For each word, the three selected tokens werecombined in a semirandomized order, with an interstimulus interval jittered around 1250ms (rangingfrom 1000 to 1500 ms), creating a 40-s ‘kiki’ string and a 40-s ‘bubu’ string, each of which contained 24tokens.

2 The relatively higher attrition rate for Experiment 1 was due to data loss from equipment failure. Specifically, there was nosound played during the experiment. This was corrected in subsequent experiments.


Visual stimuli. We created two visual stimuli: one of a curvy shape and one of an angular shape (seeFig. 1). Both shapes were red and presented on a black background. The curvy and angular shapes filled22.6% and 22.3% of the screen, respectively.

ProcedureInfants were tested in a sound attenuated room. They sat on a parent’s lap 1 m away from a 30-inch

Apple monitor. Sounds were played from an M-Audio BX5a speaker located behind the monitor. ASony DCR-HC96 camera mounted below the monitor recorded the infants and allowed the experi-menter to watch the silent video feed in a nearby room. Parents wore Extreme Isolation Ex-25noise-canceling headphones that played music mixed with spoken words to mask experimentalsounds.

Each infant was tested in an infant-controlled cross-modal matching paradigm with sequentialpresentation of congruent and incongruent trials. The infant controlled both the onset and the offsetof the trials. When the infant oriented to the screen, the experimenter started the trial. The trial endedwhen the infant looked away from the screen for a minimum of 2 s or after a maximum trial length of40 s. The experiment began with a red flashing light on a black background. The first trial was a pretesttrial to familiarize the infant with the procedure. During the pretest trial, the infant saw a shape thathad a combination of curvy and angular edges and also heard music. After the pretest trial, the exper-iment began. The experiment consisted of four identical blocks. Each block consisted of four trials witheach of the four possible word/shape pairings: ‘kiki’ with a curvy shape, ‘kiki’ with an angular shape,‘bubu’ with a curvy shape, and ‘bubu’ with an angular shape. The trials were in semirandom order such

Fig. 1. Curvy and angular shapes.


that each pair of stimuli within a block consisted of one congruent trial and one incongruent trial.Whether the first trial was congruent or incongruent was counterbalanced across infants. The redflashing light was presented between trials to draw the infant’s attention back to the screen. An ani-mated video clip was played between the blocks to retain the infant’s attention. The experiment waspresented using Habit X software (Cohen, Atkinson, & Chaput, 2004) on an iMac Apple computer, withthe experimenter recording online looking time with a key press. Infant looking times were coded off-line by a coder naive to the experimental conditions, and the offline data were used in the analyses.Online and offline coding had a correlation of r = .78 across all three infant experiments, and 25% oftrials for each infant were reliability coded by a second offline coder. Data from the two offline coderswere highly correlated at r = .87.

Results and discussion

The 4-month-olds differentiated between incongruent word–object pairings (i.e., ‘bubu’ with anangular shape or ‘kiki’ with a curvy shape) and congruent word–object pairings (i.e., ‘bubu’ with a cur-vy shape or ‘kiki’ with an angular shape). Infants looked longer at incongruent pairings (M = 16.8 s,SE = 2.3) than at congruent pairings (M = 13.0 s, SE = 2.0), t(11) = 2.77, p < .02, r = .64, paired-samplest test (see Fig. 2, and see Table 1 for mean infant looking times by stimulus characteristics in Exper-iment 1). Of the 12 infants, 10 looked longer at incongruent trials, p < .05, using a binomial test. Therewere no effects of individual sounds or shapes as measured by a two-factor analysis of variance (AN-OVA) with shape (curvy or angular) and sound (‘bubu’ or ‘kiki’) (all ps = ns). There was no interactionbetween congruency and block in a repeated-measures ANOVA, F(3,21) = 0.47, p = .71, indicating thatthe looking time difference between congruent and incongruent trials was not due to learning duringthe experiment. See Table 2 for a summary of the looking times by block and congruency. There was asignificant effect of block, F(3,21) = 5.74, p < .01, partial g2 = .45, because infants’ looking times de-creased as the experiment progressed.

These results indicate that infants may have some of the same sound–shape associations as adultsand toddlers even before they have learned reliable word–object mappings. This pattern of results dif-fers from that of a prior study showing that 4-month-olds look longer at congruent trials (matching [o]or [a] to large objects and [i] or [e] to small objects) than at incongruent trials (Peña et al., 2011). How-ever, this difference may stem from significant methodological differences between the two studies inthe number of stimuli presented and in the mode of presentation. The previous study presented sixdifferent sounds with four different shapes presented in four different colors, whereas the currentstudy presented two sounds with two shapes. Infants presented with more variables and more com-plex stimuli tend to look longer at relatively familiar or congruent pairings, whereas infants presentedwith simpler stimuli tend to look longer at relatively novel or incongruent pairings (Hunter & Ames,1988). In addition, in the previous study, the two objects were presented simultaneously with onesound, which encourages matching behavior (e.g., Patterson & Werker, 2003). In contrast, in the cur-rent study, sound–object pairs were presented sequentially, which can lead infants to look longer atinconsistent audiovisual pairings (e.g., Stager & Werker, 1997). However, both studies provide con-verging evidence that 4-month-olds differentiate between congruent and incongruent sound–objectmappings.

Table 1Mean infant looking times (in seconds) by stimulus characteristics in Experiments 1, 2, and 3.

Congruent Incongruent

Combination of vowels and consonants Curvy 13.3 (1.9) 17.6 (2.9)Angular 12.8 (2.4) 13.6 (1.9)

Vowels Curvy 12.8 (1.8) 12.1 (2.1)Angular 11.1 (2.1) 11.9 (1.9)

Consonants Curvy 9.7(1.5) 6.9 (1.1)Angular 7.2 (0.9) 8.5 (1.1)

Note. Standard errors are in parentheses.

Table 2Mean infant looking times (in seconds) in Experiment 1 for each of the four blocks.

Congruent Incongruent

Block 1 18.6 (3.5) 21.4 (3.5)Block 2 13.6 (2.5) 18.0 (3.2)Block 3 10.9 (2.6) 13.9 (2.9)Block 4 5.7 (1.5) 8.9 (2.5)

Note. Standard errors are in parentheses.


Experiment 2

Infants in Experiment 1 showed the same sound–shape mappings as adults and toddlers. However,the words tested in Experiment 1 varied on both vowels (u/i) and consonants (b/k); therefore, it is un-clear which segments of the words infants use to make the mappings—consonants, vowels, or a com-bination of vowels and consonants.

In Experiment 2, we examined whether infant mappings are based on vowels. Vowels are generallylonger in duration and have more intensity than consonants (Lagefoged & Johnson, 2010). Moreover,infants can map different vowels onto small and large objects reliably (Peña et al., 2011). Thus, infantsmay be more likely to use information from the vowels than from the consonants to map sounds toshapes. In Experiment 2, we presented infants with the same two shapes but now did so with twowords that differed only in their vowels: ‘kiki’ and ‘kuku’.

Method


22 days to 4 months 23 days, 7 girls and 5 boys). Infants were recruited in the same way as in Exper-iment 1. Of the infants’ parents, 1 had completed some high school, 2 had completed some college, 11had completed college, 8 had completed graduate degrees, and 2 declined to answer. Of the 12 infants,8 were White, 2 were of mixed ethnicity, 1 was Hispanic, and 1 infant’s parent declined to answer. Anadditional 10 infants were tested but excluded from the analysis because of fussiness (6), inattentive-ness (1), or experimenter error (3).

StimuliAuditory stimuli. The stimuli for Experiment 2 were recorded during the same session as the stimulifor Experiment 1. The tokens for each word were matched on duration (‘kuku’: M = 435 ms,range = 433–439; ‘kiki’: M = 434 ms, range = 431–437), amplitude (all tokens were 70 dB), and pitch(‘kuku’: M = 232 Hz, range = 231–233; ‘kiki’: M = 232 Hz, range = 231–232). We selected /k/ as the car-rier vowel rather than /b/ to limit the variation in rounding information in the vowel and to reduce anycoarticulation of rounding from the bilabial consonant /b/ on the vowel. We created 40-s strings in thesame way as in Experiment 1.

Visual stimuli. We used the same stimuli as in Experiment 1.

ProcedureThe procedure was identical to that in Experiment 1.


Infants looked equally at incongruent pairings (M = 12.3 s, SE = 1.8) and congruent pairings(M = 11.6 s, SE = 1.8), t(11) = 0.64, p = .54, r = .19 (see Fig. 2, and see Table 1 for mean infant lookingtimes by stimulus characteristics in Experiment 2). Of the 12 infants, 6 looked longer at the congruenttrials, p = 1.00, using a binomial test. There was no effect of individual sounds or shape as measured by


a two-factor ANOVA with shape (curvy or angular) and sound (‘kuku’ or ‘kiki’) (all ps = ns). Eventhough infants in prior studies systematically matched vowels to the size of objects (Peña et al.,2011), we found no evidence that infants matched vowels alone to the shape of objects.

Experiment 3

Vowel information was not sufficient for infants to match words and shapes. In Experiment 3, weexamined whether infants use the consonant information to map words and shapes. Adults preferusing consonants to vowels when making sound–shape pairings (Nielsen & Rendall, 2011). Infants’use of consonants to make label–shape mappings would be consistent with the role that consonantshave been proposed to play in lexical processing, as compared with the role of vowels in indexical,prosodic, and grammatical processing (Bonatti, Peña, Nespor, & Mehler, 2005; Nespor, Peña, & Mehler,2003). We presented infants with the same two shapes but now did so with two words that differedonly in their consonants: ‘bubu’ and ‘kuku’.

Method


27 days to 4 months 19 days, 7 girls and 5 boys). Of the infants’ parents, 1 had completed some highschool, 4 had completed some college, 9 had completed college, 7 had completed graduate degrees,and 3 declined to answer. Of the 12 infants, 5 were Hispanic, 4 were White, 2 were of mixed ethnicity,and 1 infant’s parents declined to answer. An additional 4 infants were tested but excluded from theanalysis because of fussiness (1), parental interference (1), or experimenter error (2).

StimuliAuditory stimuli. The stimuli for Experiment 3 were recorded at the same session as the stimuli forExperiments 1 and 2. The tokens for each word were matched on duration (‘kuku’: M = 398 ms,range = 387–409; ‘bubu’: M = 398 ms, range = 390–407), amplitude (all tokens were 70 dB), and pitch(‘kuku’: M = 231 Hz, range = 228–238; ‘bubu’: M = 230 Hz, range = 228–232). We selected /u/ ratherthan /i/ as the carrier vowel because velars such as /k/ have especially low resistance to coarticulationfrom /i/ (Fowler & Brancazio, 2000). We created 40 s strings in the same way as in Experiment 1.

0

5

10

15

20

25

Combination Vowel Consonant

Look

ing

time

(s)

CongruentIncongruent

*

Fig. 2. Mean infant looking times and standard errors for congruent and incongruent trials across the three experiments testinga combination of consonants and vowels (bubu or kiki), vowels only (kiki or kuku), and consonants only (bubu or kuku). Infantsdifferentiated between incongruent and congruent trials only with a combination of consonants and vowels.


Visual stimuli. We used the same stimuli as in Experiment 1.

ProcedureThe procedure was identical to that in Experiment 1.


Infants looked equally at incongruent pairings (M = 7.5 s, SE = 1.0) and congruent pairings(M = 8.5 s, SE = 0.9), t(11) = 0.87, p = .40, r = .25 (see Fig. 2, and see Table 1 for mean infant lookingtimes by stimulus characteristics in Experiment 3). Of the 12 infants, 7 looked longer at congruent tri-als (p = .77) using a binomial test. There were no effects of individual sounds or shape as measured bya two-factor ANOVA with shape (curvy or angular) and sound (‘bubu’ or ‘kuku’) (all ps = ns). Despitethe importance of consonants in lexical processing (Bonatti et al., 2005; Nespor et al., 2003), we foundno evidence that infants matched consonants alone to the shape of objects.

Comparison of infants’ performance across the three infant experimentsTo compare the three infant experiments, we derived each infant’s relative looking time to incon-

gruent trials by calculating a difference score between incongruent and congruent looking times(Incongruent Looking Time – Congruent Looking Time). A one-way ANOVA comparing differencescores across the three experiments revealed a reliable effect of experiment, F(2,33) = 4.66, p = .02,g2 = .22). Post hoc least-squares differences tests confirmed that infants looked relatively longer toincongruent trials in Experiment 1, which combined both vowels and consonants, than in Experiment2, which compared vowels (p = .02, r = .45), and in Experiment 3, which compared consonants (p = .01,r = .48). Infants differentiated between congruent and incongruent trials only when the labels differedon both consonants and vowels (see Fig. 2).

Experiment 4

Previous work examining the features of speech sounds used by adults to make sound–object map-pings found that participants were more likely to respond based on consonant quality than on vowelquality when the two made opposite predictions (Nielsen & Rendall, 2011). The current experimentswith infants used different words (‘bubu’, ‘kiki’, and ‘kuku’) from those used in previous studies. InExperiment 4, we tested adults on these new materials to confirm that adults generalize sound–shapemappings to these words. Furthermore, we examined adults’ use of vowels or consonants in makingsystematic mappings.

Method

ParticipantsThe participants were 10 undergraduate students (9 women and 1 man) either enrolled at New

York University or participating in a summer internship program at the university. All of the partici-pants were native English speakers and between 18 and 22 years old. An additional 3 participants’data were not included in the analysis because they were not native English speakers (2) or did notfollow task instructions (1).

StimuliAuditory stimuli. These were the same as those used in the previous experiments with the addition of‘bibi’. The tokens were adjusted using Praat (Boersma & Weenink, 2010) to be similar in duration(M = 425 ms, range = 423–438), pitch (M = 227 Hz, range = 225–230), and intensity (all 70 dB).

Visual stimuli. These were the same shapes as those used in the previous experiments except that weincluded more colors to encourage independent judgments for each trial: red, light pink, bright pink,maroon, brown, rust, orange, mustard, marigold, yellow, bright green, seafoam green, emerald green,


olive green, teal, turquoise, French blue, royal blue, light gray, dark gray, gray–green, blue–purple, andpurple.

ProcedureOne-label task. Participants sat at a desktop computer wearing Extreme Isolation Ex-25 noise-cancel-ing headphones. They were presented with two shapes: a curvy shape and an angular shape of thesame color, one on each side of the computer screen. The shapes remained consistent across all ofthe trials but varied in color between trials. The two objects presented in any given trial were ofthe same color. Below these shapes was a teardrop (half curvy and half angular). Participants were in-structed to click on the teardrop to hear a word and then to record whether the word was a bettermatch for the shape on the left or the right. Two trials for each of the nonwords used in the infantexperiments (‘bubu’, ‘kiki’, or ‘kuku’), with the addition of ‘bibi’, were presented for a total of eightexperimental trials.3

Two-label task. Participants sat at a desktop computer wearing Extreme Isolation Ex-25 noise-cancel-ing headphones. They were presented with the same two shapes as in the one-label task but with anadditional teardrop above the objects. The second teardrop played a different word when clicked. Par-ticipants were instructed to click on the two teardrops to hear two words and then to pair each of thewords with one of the two objects. This procedure allowed the isolation of specific features for word–object pairings. For example, a trial with ‘kiki’ and ‘kuku’ measured whether participants could use vo-wel information alone to make consistent pairings without conflicting or supporting consonant infor-mation. There was a total of six test trials—one trial for each permutation of /b/ and /k/ combined with/i/ and /u/ (kiki/kuku, bibi/bubu, kuku/bubu/, kiki/bibi, bibi/kuku, and bubu/kiki).


One-label taskTrials were scored according to whether participants used a vowel (e.g., pairing /i/ with the angular

shape), a consonant (e.g., pairing /k/ with the angular shape), or a combination of the two (e.g., pairing/kiki/ with the angular shape). Participants had the highest score using the combinations (M = 95%),followed by consonants (M = 88%). Scores on trials comparing vowels (M = 63%) were significantlylower than either consonant trials, t(9) = 4.58, p < .01, r = .84, or combination trials, t(9) = 9.80,p < .01, r = .96.

Two-label taskTrials were scored according to whether participants assigned the words to the congruent shapes

(/i/ or /k/ with the spiky shape and /u/ or /b/ with the curvy shape). Accuracy was equivalent betweentrials when only vowel information distinguished the words (kiki/kuku or bibi/bubu, M = 90%), whenonly consonant information distinguished the words (kiki/bibi or kuku/bubu, M = 90%), and when theconsonant–vowel combination distinguished the words (bubu/kiki, M = 100%). However, when wordscontained conflicting vowel and consonant information (kuku/bibi), 8 of the 10 participants matchedthe words according to consonant information (p < .05) using a binomial test.

Adults’ use of consonants and vowels differed from results of some previous studies in which adultsused only consonant information to match shapes and sounds (Nielsen & Rendall, 2011). This may bebecause previous studies used different materials with multiple consonants and vowels—‘takete’ and‘maluma’ rather than ‘bubu’/‘buba’ and ‘kiki’. However, the difference in findings is more likely due tothe different tasks. The task used by Nielsen and Rendall (2011) is more similar to the one-label task,in which participants need to choose whether to use vowels or consonants to match the shapes. Theone-label task results are consistent with Nielsen and Rendall’s findings in showing that participantswill use consonants instead of vowels. However, Nielsen and Rendall did not contrast labels differing

3 Participants were also presented with trials where they heard the non-English vowels (/i/ and /y/) combined with both /b/ and/k/, but these trials were not analyzed.


only on vowels; thus, their results cannot speak to whether participants can use vowel information atall. Our results suggest that participants prefer to match using consonant information but can usevowels when no consonant information is available (see also Monaghan et al., 2012).

General discussion

We predicted that if 4-month-old infants have the same sound–shape mapping biases as adults andtoddlers, they would differentiate between congruent trials (pairing the angular shape with ‘kiki’ orthe curvy shape with ‘bubu’) and incongruent trials (pairing the curvy shape with ‘kiki’ or the angularshape with ‘bubu’). Infants looked longer at incongruent pairings than at congruent pairings. Thismapping appears to be based on a combination of vowels and consonants given that neither vowelsalone nor consonants alone elicited systematic mapping. Our results indicate that adult sound–shapemapping biases have their origins during infancy and might not result uniquely from language expo-sure creating statistical regularities in the lexicon.

Adults in our study, like infants, used a combination of consonant and vowel information to matchthe labels they heard with the shapes they saw. Their better performance with a combination of con-sonants and vowels rather than either vowels or consonants alone may be because the combinationof vowels and consonants contained more sound symbolic phonemes, thereby inducing a strongersound symbolism effect (Thompson & Estes, 2011). However, this was not the only strategy thatwas available to them. Adults, unlike infants, were also able to use consonant information aloneand vowel information alone to match the labels to the shapes, albeit less frequently than the conso-nant–vowel combination. When vowels and consonants were put in conflict, adults used consonantsmore often than vowels. Adults’ primary use of consonants is consistent with the role that consonantshave been proposed to play in lexical processing, as compared with the role of vowels in indexical,prosodic, and grammatical processing (Bonatti et al., 2005; Nespor et al., 2003). Specifically, conso-nants and vowels are processed differently; adults extract statistical regularities from consonantsbut not from vowels (Bonatti et al., 2005; Toro, Shukla, Nespor, & Endress, 2008; but see Newport& Aslin, 2004), whereas rule extraction is privileged over vowels but not over consonants for adultsand infants (Bonatti et al., 2005; Hochmann, Benavides-Varela, Nespor, & Mehler, 2011; Pons & Toro,2010; Toro et al., 2008).

Where do these biases for systematically mapping sounds to shapes come from? One possibility isthat sound symbolism is related to cross-modal synesthetic tendencies (e.g., Ramachandran & Hub-bard, 2001; Spector & Maurer, 2009). An adult with synesthesia experiences additional involuntarysensations in one sensory modality when another sensory modality is stimulated (Cytowic & Eagl-eman, 2009; Cytowic & Wood, 1982). Synesthetic associations are stronger in younger infants thanin older infants (Wagner & Dobkins, 2011), indicating that these associations may play a larger rolein the perceptions of prelinguistic infants than of toddlers or adults. However, because synesthesia oc-curs in only 4 to 5% of the adult population (Simner & Ward, 2006), whereas 90% to 98% of adults showsystematic sound–shape mapping biases (Köhler, 1947), our results are unlikely to be explained byadult synesthesia. At the same time, the human proclivity for systematically mapping sounds toshapes is also consistent with data from nonhuman primates showing biased cross-modal correspon-dences in chimpanzees (Ludwig, Adachi, & Matsuzawa, 2011). Thus, sound symbolism may be anintrinsic feature of primate sensory systems.

Another potential explanation for the origins of cross-modal biases comes from theories claiminglinks between perception and production mechanisms (Calvert & Campbell, 2003; Keysers et al., 2003;Kohler et al., 2002; Liberman & Mattingly, 1985; Wilson, Saygin, Sereno, & Iacoboni, 2004) and is sup-ported by studies with individuals with autism spectrum disorder and patients with brain lesions.These individuals do not show as systematic a bias for sound–shape mappings as typical individualsdo. Where typically developing individuals agree with the standard result 90% to 98% of the time,individuals with autism or lesions to the angular gyrus agree only 56% of the time (Ramachandran,Azoulai, Stone, Srinivasan, & Bijoy, 2005). Children with autism spectrum disorder also do not showthe sound–shape mapping biases that neurologically typical adults and children show (Oberman &Ramachandran, 2008). Individuals with autism spectrum disorder or brain lesions are proposed to


have a dysfunction in the mirror neuron system linking perception and production that prevents thesystematic bias for sound–shape mappings that typical adults show (Altschuler et al., 2000; Nishitani,Avikainen, & Hari, 2004; Oberman et al., 2005; Theoret et al., 2005).

The current results provide evidence that prelinguistic infants show adult-like nonarbitrary map-ping biases between speech labels and objects. Infants in our experiments showed these biases onlywhen the two labels differed in both consonants and vowels. Our results differ from those of priorstudies in which vowels alone were sufficient to convey object size to 4-month-olds (Peña et al.,2011). The use of vowels alone in prior studies may be grounded in observable physicalrelationships between the pitch of vowels and the size of objects built through participants’ prior mul-tisensory experiences. The relationship between vowel quality and object shape, in contrast, might notbe as easily observable, consistent with findings that consonants, and not vowels, are preferred inadult sound–shape mappings (Experiment 4 in the current study; see also Nielsen & Rendall, 2011).It must be noted that our experiments used only two consonants (/b/ and /k/) and two vowels (/i/and /u/). Although these data are consistent with previous infant studies linking vowel and object size(Peña et al., 2011) and pitch and object shape (Walker et al., 2010), further research with different con-sonants and vowels is needed to understand what may drive infants’ sound–shape mapping biases.

Our perceptual system must combine or segregate concurrent sensory inputs from different modal-ities to quickly and accurately detect objects or events and to choose appropriate responses (Evans &Treisman, 2010). In a world full of multisensory experiences, developing infants must be able to inte-grate multisensory inputs (e.g., Bahrick, Lickliter, & Flom, 2004; Lewkowicz, 2000). The early biases forcross-modal correspondences during infancy provide a basis for processing the multisensory nature ofour daily experiences.

Acknowledgments

This research was supported by New York University (NYU) research funds to Athena Vouloum-anos. We thank Ilana Shanks and the members of the NYU Infant Cognition and Communication Labfor their help in running and coding the studies and thank the parents and infants who participatedin this research.

References

Altschuler, E. L., Vankov, A., Hubbard, E. M., Roberts, E., Ramachandran, V. S., & Pineda, J. A. (2000). Mu wave blocking by observerof movement and its possible use as a tool to study theory of other minds. In Poster session presented at the 30th annualmeeting of the Society for Neuroscience, New Orleans, LA.

Bahrick, L. E., Lickliter, R., & Flom, R. (2004). Intersensory redundancy guides the development of selective attention, perception,and cognition in infancy. Current Directions in Psychological Science, 13, 99–102.

Bergelson, E., & Swingley, D. (2012). At 6–9 months, human infants know the meanings of many common nouns. Proceedings ofthe National Academy of Sciences of the United States of America, 109, 3253–3258.

Boersma, P., & Weenink, D. (2010). Praat: Doing phonetics by computer (Version 5.3.08) [computer program]. Retrieved 26 July2010 from http://www.praat.org.

Bonatti, L., Peña, M., Nespor, M., & Mehler, J. (2005). Linguistic constraints on statistical computations: The role of consonantsand vowels in continuous speech processing. Psychological Science, 16, 451–459.

Calvert, G. A., & Campbell, R. (2003). Reading speech from still and moving faces: The neural substrates of visible speech. Journalof Cognitive Neuroscience, 15, 57–70.

Cohen, L. B., Atkinson, D. J., & Chaput, H. H. (2004). Habit X: A new program for obtaining and organizing data in infant perceptionand cognition studies (Version 1.0). Austin: University of Texas.

Cytowic, R. E., & Eagleman, D. M. (2009). Wednesday is indigo blue: Discovering the brain of synesthesia. Cambridge, MA: MIT Press.Cytowic, R. E., & Wood, F. B. (1982). Synesthesia: I. A review of major theories and their brain basis. Brain and Cognition, 1, 23–35.Davis, R. (1961). The fitness of names to drawings: A cross-cultural study in Tanganyika. British Journal of Psychology, 52,

259–268.De Saussure, F. (1983). Course in general linguistics. In C. Bally & A. Sechehaye (Eds), R. Harris, Trans. La Salle, IL: Open Court.

(Original work published 1916).Descartes, R. (1986). Meditations on first philosophy. New York: Cambridge University Press (Original work published 1641).Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10,

1–12.Fenson, L., Dale, P. S., Reznick, J. S., Bates, E., Thal, D. J., & Pethick, S. J. (1994). Variability in early communicative development.

Monographs of the Society for Research in Child Development, 59(5), 1–173 (discussion, pp. 174–185).Fowler, C. A., & Brancazio, L. (2000). Coarticulation resistance of American English consonants and its effects on trans-

consonantal vowel-to-vowel coarticulation. Language and Speech, 43, 1–41.

http://www.praat.org


Gasser, M. (2004). The origins of arbitrariness in language. In Proceedings of the annual conference of the Cognitive Science Society(Vol. 26). Mahwah, NJ: Lawrence Erlbaum.

Gibson, J. J. (1966). The senses considered as perceptual systems. Boston: Houghton Mifflin.Hochmann, J. R., Benavides-Varela, S., Nespor, M., & Mehler, J. (2011). Consonants and vowels: Different roles in early language

acquisition. Developmental Science, 14, 1445–1458.Hockett, C. F. (1977). The view from language: Selected essays 1948–1974. Athens: University of Georgia Press.Hunter, M. A., & Ames, E. W. (1988). A multifactor model of infant preferences for novel and familiar stimuli. Advances in Infancy

Research, 5, 69–95.Imai, M., Kita, S., Nagumo, M., & Okada, H. (2008). Sound symbolism facilitates early verb learning. Cognition, 109, 54–65.Keysers, C., Kohler, E., Umilta, M. A., Nanetti, L., Fogassi, L., & Gallese, V. (2003). Audiovisual mirror neurons and action

recognition. Experimental Brain Research, 153, 628–636.Kohler, E., Keysers, C., Umilta, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). Hearing sounds, understanding actions: Action

representation in mirror neurons. Science, 297, 846–848.Köhler, W. (1947). Gestalt psychology (2nd ed.). New York: Liveright.Kovic, V., Plunkett, K., & Westermann, G. (2010). The shape of words in the brain. Cognition, 114, 19–28.Lagefoged, P., & Johnson, K. (2010). A course in phonetics (6th ed.). Boston: Wadsworth.Lewkowicz, D. J. (2000). The development of intersensory temporal perception: An epigenetic systems/limitations view.

Psychological Bulletin, 126, 281–308.Liberman, A. M., & Mattingly, I. G. (1985). The motor theory of speech perception revised. Cognition, 21, 1–36.Ludwig, V. U., Adachi, I., & Matsuzawa, T. (2011). Visuoauditory mappings between high luminance and high pitch are shared by

chimpanzees (Pan troglodytes) and humans. Proceedings of the National Academy of Sciences of the United States of America,108, 20661–20665.

Marks, L. E. (1987). On cross-modal similarity: Auditory–visual interactions in speeded discrimination. Journal of ExperimentalPsychology: Human Perception and Performance, 13(3), 384–394.

Maurer, D., Pathman, T., & Mondloch, C. J. (2006). The shape of boubas: Sound–shape correspondences in toddlers and adults.Developmental Science, 9, 316–322.

Monaghan, P., Christiansen, M. H., & Fitneva, S. A. (2011). The arbitrariness of the sign: Learning advantages from the structureof the vocabulary. Journal of Experimental Psychology: General, 140, 325–347.

Monaghan, P., Mattock, K., & Walker, P. (2012). The role of sound symbolism in language learning. Journal of ExperimentalPsychology: Learning, Memory, & Cognition. http://dx.doi.org/10.1037/a0027747. [Epub ahead of print].

Nespor, M., Peña, M., & Mehler, J. (2003). On the different roles of vowels and consonants in speech processing and languageacquisition. Lingue & Linguaggio, 2, 203–231.

Newport, E. L., & Aslin, R. N. (2004). Learning at a distance. I. Statistical learning of non-adjacent dependencies. CognitivePsychology, 48, 127–162.

Nielsen, A., & Rendall, D. (2011). The sound of round: Evaluating the sound–symbolic role of consonants in the classic takete–maluma phenomenon. Canadian Journal of Experimental Psychology, 65, 115–124.

Nishitani, N., Avikainen, S., & Hari, R. (2004). Abnormal imitation-related cortical activation sequences in Asperger’s syndrome.Annals of Neurology, 55, 558–562.

Nygaard, L. C., Cook, A. E., & Namy, L. L. (2009). Sound to meaning correspondences facilitate word learning. Cognition, 112,181–186.

Oberman, L. M., Hubbard, E. M., McCleery, J. P., Altschuler, E. L., Ramachandran, V. S., & Pineda, J. A. (2005). EEG evidence formirror neuron dysfunction in autism spectrum disorders. Cognitive Brain Research, 24, 190–198.

Oberman, L. M., & Ramachandran, V. S. (2008). Reflections on the mirror neuron system: Their evolutionary functions beyondmotor representation. In J. A. Pineda (Ed.), Role of mirroring in social cognition. Totowa, NJ: Humana.

Ohala, J. J. (1997). Sound symbolism. In Proceedings of the 4th Seoul International Conference on Linguistics (pp. 98–103). SICOL:Seoul, Korea.

Parault, S. J. (2006). Sound symbolic word learning in context. Contemporary Educational Psychology, 31, 228–252.Parault, S. J., & Schwanenflugel, P. J. (2006). Sound symbolism: A possible piece in the puzzle of word learning. Journal of

Psycholinguistic Research, 35, 329–351.Patterson, M. L., & Werker, J. F. (2003). Two-month-old infants match phonetic information in lips and voice. Developmental

Science, 6, 191–196.Peña, M., Mehler, J., & Nespor, M. (2011). The role of audiovisual processing in early conceptual development. Psychological

Science, 22, 1419–1421.Pons, F., & Toro, J. M. (2010). Structural generalizations over consonants and vowels in 11-month-old infants. Cognition, 116,

361–367.Ramachandran, V. S., Azoulai, S., Stone, L., Srinivasan, A. V., & Bijoy, N. (2005, April). Grasping metaphors and thinking with

pictures: How brain damage might affect thought and language. In Poster presented at the 12th annual meeting of theCognitive Neuroscience Society, New York.

Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia: A window into perception, thought, and language. Journal ofConsciousness Studies, 8, 3–34.

Sapir, E. (1929). The status of linguistics as a science. Language, 5, 209.Shayan, S., Ozturk, O., & Sicoli, M. A. (2011). The thickness of pitch: Crossmodal metaphors in Farsi, Turkish, and Zapotec. Senses

& Society, 6, 96–105.Simner, J., & Ward, J. (2006). The taste of words on the tip of the tongue. Nature, 444, 438.Spector, F., & Maurer, D. (2009). Synesthesia: A new approach to understanding the development of perception. Developmental

Psychology, 45, 175–189.Stager, C. L., & Werker, J. F. (1997). Infants listen for more phonetic detail in speech perception than in word-learning tasks.

Nature, 388, 381–382.Tanz, C. (1971). Sound symbolism in words relating to proximity and distance. Language & Speech, 14, 266–276.

http://dx.doi.org/10.1037/a0027747


Theoret, H., Halligan, E., Kobayashi, M., Fregni, F., Tager-Flusberg, H., & Pascual-Leone, A. (2005). Impaired motor facilitationduring action observation in individuals with autism spectrum disorder. Current Biology, 15, 84–85.

Thompson, P. D., & Estes, Z. (2011). Sound symbolic naming of novel objects is a graded function. Quarterly Journal ofExperimental Psychology, 64, 2392–2404.

Toro, J. M., Shukla, M., Nespor, M., & Endress, A. (2008). The quest for generalizations over consonants: Asymmetries betweenconsonants and vowels are not the by-product of acoustic differences. Perception and Psychophysics, 70, 1515–1525.

Voeltz, F. K. E., & Kilian-Hatz, C. (Eds.). (2001). Ideophones. Amsterdam: John Benjamins (Typological Studies in Language, Vol.44).

Wagner, K., & Dobkins, K. R. (2011). Synaesthetic associations decrease during infancy. Psychological Science, 22, 1067–1072.Walker, P., Bremner, J. G., Mason, U., Spring, J., Mattock, K., Slater, A., et al (2010). Preverbal infants’ sensitivity to synaesthetic

cross-modality correspondences. Psychological Science, 21, 21–25.Whalen, D. H., & Levitt, A. G. (1995). The universality of intrinsic F0 in vowels. Journal of Phonetics, 23, 349–366.Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004). Listening to speech activates motor areas involved in speech

production. Nature Neuroscience, 7, 701–702.Zbikowski, L. M. (1998). Metaphor and music theory: Reflections from cognitive science. Music Theory Online, 4(1).

Sound symbolism in infancy: Evidence for sound–shape cross-modal correspondences in 4-month-olds

Documents

Transcript of Sound symbolism in infancy: Evidence for sound–shape cross-modal correspondences in 4-month-olds