The Ontogeny of Human Multisensory Object Perception: A Constructivist Account

Chapter 16The Ontogeny of Human Multisensory ObjectPerception: A Constructivist Account

David J. Lewkowicz

16.1 Introduction

Our perceptual world is multisensory in nature (J. J. Gibson, 1966; Marks, 1978;Stein and Meredith, 1993; Werner, 1973). This means that the objects in our naturalworld are usually specified by some combination of visual, auditory, tactile, olfac-tory, and gustatory attributes. From a theoretical perspective, it is possible that themélange of multisensory object attributes might be confusing. Fortunately, however,the human perceptual system and its underlying neural mechanisms have evolved toenable us to integrate the various multisensory attributes that are usually available.As a result, we are able to perceive multisensory attributes as part-and-parcel ofcoherent, spatiotemporally continuous, and bounded physical entities.

The fact that adults easily perceive coherent multisensory objects raises theobvious and central question regarding the developmental origins of this ability.Specifically, the question is when and how does the ability to perceive the coherentnature of multisensory objects emerge? The answer is that this ability takes time toemerge. Furthermore, the answer is decidedly contrary to recent nativist claims thatinfants come into this world with “core knowledge” that endows them with a pre-determined set of principles about objects and their properties (Spelke and Kinzler,2007). The empirical findings that will be reviewed here will show that the develop-mental emergence of the ability to perceive specific types of intersensory relationsis a heterochronous affair (Lewkowicz, 2002) and it will be argued that this reflectsthe outcome of a complex and dynamic interaction between constantly changingneural, behavioral, and perceptual processes. As a result, the theoretical conclusionthat will be reached here will be opposite to the nativist one. It will be argued thatthe current empirical evidence is consistent with constructivist and developmentalsystems approaches to perceptual and cognitive development (Cohen et al., 2002;Piaget, 1954; Spencer et al., 2009). According to these approaches, perceptual skills

D.J. Lewkowicz (B)Department of Psychology, Florida Atlantic University, 777 Glades Rd., Boca Raton, FL, USAe-mail: [email protected]

303M.J. Naumer, J. Kaiser (eds.), Multisensory Object Perception in the Primate Brain,DOI 10.1007/978-1-4419-5615-6_16, C© Springer Science+Business Media, LLC 2010

304 D.J. Lewkowicz

at particular points in the life of an organism are the product of a complex develop-mental process that involves the co-action of multiple factors at different levels oforganization (cellular, neural, behavioral, and extraorganismic).

16.2 Setting the Theoretical Problem

It might be argued that multisensory objects present a difficult challenge for devel-oping infants. Their nervous system is highly immature at birth and they areperceptually and cognitively inexperienced. As a result, it is reasonable to expectthat infants come into the world with no prior knowledge of their physical worldand, thus, that they would not know a priori whether certain multisensory attributesbelong together. For example, they would not be expected to know whether a par-ticular melodious and high-pitched voice belongs to one or another person’s face.Indeed, the relation between various modality-specific object attributes (e.g., color,pitch, taste, or temperature) is arbitrary and infants must learn to bind them on acase-by-case basis in order to perceive multisensory objects as coherent entities.This is complicated by the fact that the sensory systems develop and mature overthe first few months of life and, as a result, may not always be able to detect rele-vant modality-specific attributes with sufficient precision to bind them. Interestingly,however, as will be shown here, infants have a way of getting around this problem byessentially ignoring many multisensory object features (e.g., the various attributesthat specify a person’s identity) and by simply binding them on the basis of theirspatial and temporal coincidence. In other words, even though young infants possesrather poor auditory and visual sensory skills, and even though they may have dif-ficulty resolving the detailed nature of auditory and visual inputs, they possessbasic mechanisms that enable them to bind modality-specific multisensory attributesand that permit them to take advantage of the redundancy inherent in multisensoryobjects and events. As a result, despite their limitations, infants possess one basic butpowerful mechanism that enables them to begin the construction of a multisensoryobject concept.

A second way for infants to construct a multisensory object concept is to relyon the various amodal invariant attributes that are normally inherent in multisensoryinputs. These types of attributes provide equivalent information about objects acrossdifferent modalities. For example, objects can be specified in audition and vision bytheir common intensity, duration, tempo, and rhythm. Likewise, in vision and touchobjects can be specified by their common shape and texture and in audition andtouch objects can be specified by their common duration, tempo, and rhythm. Ingeneral, amodal information is inherently relational and, as a result, intersensorybinding is not necessary. What is necessary, however, is that the perceiver be able todetect the amodal invariance and here, again, evidence indicates that this mechanismemerges gradually during infancy (Lewkowicz, 2000a, 2002).

The behavioral literature has clearly demonstrated that human adults are verygood at perceiving coherent multisensory objects and events (Calvert et al., 2004;

16 Ontogeny of Human Multisensory Object Perception 305

Marks, 1978; Welch and Warren, 1986) and, thus, that they can bind modality-specific attributes and detect amodal invariant attributes. In addition, a broadercomparative literature has demonstrated that the redundancy inherent in multi-sensory objects and events is highly advantageous because it facilitates detection,discrimination, and learning in human adults as well as those of many other species(Partan and Marler, 1999; Rowe, 1999; Stein and Stanford, 2008; Summerfield,1979). Underlying this ability is a nervous system that has evolved mechanismsspecifically devoted to the integration of multisensory inputs and to the percep-tion of amodal invariance (Calvert et al., 2004; Stein and Meredith, 1993; Steinand Stanford, 2008). Indeed, intersensory integration mechanisms are so perva-sive throughout the primate brain that some have gone so far as to suggest that theprimate brain is essentially a multisensory organ (Ghazanfar and Schroeder, 2006).

The behavioral developmental literature also has provided impressive evidenceof intersensory perception in infancy. It has shown that infants can bind multi-sensory inputs and perceive various types of intersensory relations (Lewkowicz,2000a, 2002; Lewkowicz and Lickliter, 1994; Lickliter and Bahrick, 2000; Walker-Andrews, 1997), and that they can take advantage of multisensory redundancy(Bahrick et al., 2004; Lewkowicz, 2004; Lewkowicz and Kraebel, 2004). What isparticularly interesting about this evidence is that it is broadly consistent with thedominant theory of intersensory perceptual development put forth by E. Gibson(1969). Gibson’s theory is based on the concept of perceptual differentiation andholds that infants are equipped with the ability to detect amodal invariance at birthand that, as development progresses, they learn to differentiate increasingly finer andmore complex forms of amodal invariance. Although at first blush the extant empir-ical evidence appears to be consistent with Gibson’s theory, a more detailed look atthe evidence indicates that the developmental picture is considerably more complex.In essence, a good bit of the empirical evidence on infant intersensory perceptionamassed since Gibson proposed her theory has indicated that certain intersensoryperceptual skills are absent early in life, that specific ones emerge at specific timesduring infancy, and that following their emergence they continue to improve andchange as infants grow and acquire perceptual experience (Bremner et al., 2008;Lewkowicz, 2000a, 2002; Lewkowicz and Lickliter, 1994; Lickliter and Bahrick,2000; Walker-Andrews, 1997). This evidence is actually consistent with the the-ory that posits the opposite process to that of differentiation, namely developmentalintegration (Birch and Lefford, 1963; Piaget, 1952). According to this theory, inter-sensory perceptual abilities are initially absent and only emerge gradually as infantsacquire experience with multisensory inputs and discover the relations among them.

Given that evidence can be found to support the developmental differentiationand the developmental integration views, one reasonable theoretical stance mightbe that both developmental differentiation and developmental integration processescontribute to the emergence of the multisensory object concept. It turns out, how-ever, that some recent empirical evidence suggests that a third process, that involvesperceptual narrowing, also contributes in an important way to the development ofintersensory perceptual functions. The importance of this process in the develop-ment of intersensory perception has recently been uncovered by Lewkowicz and

306 D.J. Lewkowicz

Ghazanfar (2006) and is described later in this chapter. As a result, the most rea-sonable theoretical framework that best describes the processes involved in thedevelopment of intersensory perception is one that incorporates the processes ofdevelopmental differentiation and integration as well as developmental narrowing.

The new theoretical framework raises several distinct but highly inter-relatedquestions that must be answered if we are to better understand how infants cometo acquire a stable and coherent conception of their multisensory world. First, doinfants possess the necessary perceptual mechanisms for perceiving various mul-tisensory object attributes as part-and-parcel of coherent object representations?Second, if they do, when do these mechanisms first begin to emerge? Third, whatis the nature of these early mechanisms and how do they change over development?Finally, what underlying processes govern the emergence of these early mecha-nisms? These four questions will be addressed by reviewing research from mylaboratory as well as related research from other laboratories, with a specific focuson infant perception of audio-visual (A-V) relations.

16.3 Response to A-V Intensity Relations

There is little doubt that infants’ ability to perceive multisensory coherence is ham-pered by their neurobehavioral immaturity, relative perceptual inexperience, and themultisensory and diverse character of the external world. As noted earlier, how-ever, the relational and invariant nature of the typical multisensory perceptual arrayprovides many ready-made sources of coherence (J. J. Gibson, 1966). Indeed, it isprecisely for this reason that some theorists have asserted that infants come into theworld prepared to pick up multisensory coherence (E. J. Gibson, 1969; Thelen andSmith, 1994). Although this view is a reasonable one, given their neurobehavioraland perceptual limitations, young infants are not likely to perceive multisensorycoherence in the same way that adults do. That is, young infants probably comeinto the world armed with some basic perceptual skills that enable them to discoverrelatively simple types of multisensory coherence and only later, as they graduallyacquire the ability to perceive higher-level types of intersensory relations, discovermore complex forms of multisensory coherence. Thus, initially, infants are likely tobe sensitive to relatively low-level kinds of intersensory perceptual relations such asintensity and temporal synchrony.

Consistent with the above developmental scenario, Lewkowicz and Turkewitz(1980) found that newborn infants are, in fact, sensitive to A-V intensity relations.This study was prompted by Schneirla’s (1965) observation that young organismsof many different species are sensitive to the quantitative nature of stimulation,defined as a combination of the physical intensity of stimulation and organismicfactors (i.e., level of arousal). Lewkowicz and Turkewitz (1980) reasoned that ifyoung infants are primarily responsive to the effective intensity of stimulation, thenyoung infants should be able to perceive intensity-based A-V relations. To test thisprediction, Lewkowicz and Turkewitz (1980) habituated 3-week-old infants to a


constant-intensity visual stimulus (a white disc) and then tested their response todifferent-intensity white-noise stimuli. Importantly, the white-noise stimuli spannedan intensity range that included one stimulus that adults judged to be most equiva-lent in terms of its intensity to the visual stimulus presented to the infants during thehabituation phase. As a result, Lewkowicz and Turkewitz expected that if infantsspontaneously attempted to equate the auditory and visual stimuli, they wouldexhibit differential response to the auditory stimuli. Specifically, it was expectedthat infants would exhibit the smallest response recovery to the auditory stimuluswhose intensity was judged by adults to be equivalent to the visual stimulus andincreasingly greater response recovery to those auditory stimuli whose intensitieswere increasingly more discrepant from the visual stimulus. The results were con-sistent with this prediction and, thus, indicated that neonates do, indeed, have thecapacity to perceive A-V equivalence on the basis of intensity.

In a follow-up study, Lewkowicz and Turkewitz (1981) tested the generality ofintensity-based intersensory responsiveness by testing newborn infants’ response todifferent intensities of visual stimulation following exposure to auditory stimulationof a constant intensity. If intensity-based intersensory responsiveness is a gen-eral characteristic of neonatal perceptual responsiveness then newborns’ responseto visual stimulation varying in intensity should be affected in a systematic wayby prior auditory stimulation (presumably mediated by the induction of arousalchanges). To test this possibility, Lewkowicz and Turkewitz (1981) presented pairsof visual stimuli varying in intensity to two groups of newborns. One group firstheard a moderate-intensity white-noise stimulus whereas the other group did not.Consistent with expectation, results indicated that the group that heard the white-noise stimulus preferred to look at a low-intensity visual stimulus whereas the groupthat did not hear the white-noise stimulus preferred to look at a higher-intensityvisual stimulus. This shift in the looking preference was interpreted to reflectthe effects of increased arousal caused by exposure to the white-noise stimulus.Presumably, when the increased internal arousal was combined with the physicalintensity of the moderate-intensity stimulus, the effective intensity of stimulationexceeded the infants’ optimal level of preferred stimulation and caused them to shifttheir attention to the lower-intensity stimulus in order to re-establish the optimallevel of stimulation.

16.4 Response to A-V Temporal Synchrony Relations

The natural multisensory world is characterized by patterns of correlated andamodally invariant information (J. J. Gibson, 1966). Part of this invariance is dueto the fact that, under normal conditions, the auditory and visual sensory attributesthat specify multisensory objects and events are usually temporally coincident. Thatis, the temporal patterns of auditory and visual information that specify every-day objects and events always have concurrent onsets and offsets and, thus, arealways temporally synchronous (e.g., a talker’s vocalizations always start and stop

308 D.J. Lewkowicz

whenever the talker’s lips start and stop moving). Audio-visual synchrony canprovide infants with an initial opportunity to perceive audible and visible objectand event attributes as part-and-parcel of coherent entities (E. J. Gibson, 1969;Lewkowicz, 2000a; Thelen and Smith, 1994). For infants, this is especially impor-tant for two reasons. First, it provides them with an initial opportunity to detect themultisensory coherence of their world. Second, once infants discover that auditoryand visual inputs are synchronous, this sets the stage for the discovery of other typesof A-V relations including those based on equivalent durations, tempos, and rhyth-mical structure, gender, affect, and identity. In other words, an initial perceptualsensitivity to A-V temporal synchrony relations (as well as A-V intensity relations)provides infants with the initial scaffold that enables them to subsequently discovermore complex forms of intersensory correspondence.

16.4.1 Infant Perception of A-V Synchrony Relations

Detection of temporal A-V synchrony is relatively easy because it only requiresthe perception of synchronous energy onsets and offsets and because it is mediatedby relatively low-level, subcortical tecto-thalamo-insular pathways that make thedetection of A-V temporal correspondence at an early level of cortical processingpossible (Bushara et al., 2001). This suggests that despite the fact that the infant ner-vous system is immature, infants should be able to detect A-V synchrony relationsquite early in postnatal life. Indeed, studies at the behavioral level have indicatedthat, starting early in life, infants are sensitive and responsive to A-V temporal syn-chrony relations. For example, Lewkowicz (1996) conducted systematic studies toinvestigate the detection of A-V temporal synchrony relations across the first year oflife by studying 2-, 4-, 6-, and 8-month-old infants and compared their performanceto that of adults tested in a similar manner. The infants were habituated to a two-dimensional object that bounced up and down on a computer monitor and made animpact sound each time it changed direction at the bottom of the monitor. Followinghabituation, infants were given a set of test trials to determine the magnitude of theirA-V asynchrony detection threshold. This was done either by presenting the impactsound prior to the object’s visible bounce (sound-first condition) or by presenting itfollowing the visible bounce (sound-second condition).

Results yielded no age differences and indicated that the lowest A-V asynchronythat infants detected in the sound-first condition was 350 ms and that the lowestasynchrony that they detected in the sound-second condition was 450 ms. In con-trast, adults who were tested in a similar task and with the same stimuli detected theasynchrony at 80 ms in the sound-first condition and at 112 ms in the sound-secondcondition. Based on these results, Lewkowicz concluded that the intersensory tem-poral contiguity window (ITCW) is considerably wider in infants than in adults andthat, as a result, infants tend to perceive temporally more disparate multisensoryinputs as coherent.


16.4.2 Infant Perception of A-V Speech Synchrony and Effectsof Experience

In subsequent studies, Lewkowicz has found that the ITCW is substantially largerfor multisensory speech than for simple abstract events. In the first of these stud-ies, Lewkowicz (2000b) habituated 4-, 6- and 8-month-old infants to audio-visuallysynchronous syllables (/ba/ or /sha/) and then tested their response to audio-visuallyasynchronous versions of these syllables. Results indicated that infants successfullydetected a 666 ms asynchrony. In a subsequent study, Lewkowicz (2003) found that4- to 8-month-old infants detected an asynchrony of 633 ms.

In the most recent study, Lewkowicz (2010) investigated whether detectionthresholds for audio-visual speech asynchrony are affected by initial short-termexperience. The purpose of this study was, in part, to determine whether the effectsof short-term exposure to synchronous vs. asynchronous audio-visual events haveeffects similar to those observed in adults. Studies with adults have shown thatwhen they are first tested with audio-visually asynchronous events, they perceivethem as such, but that after they are given short-term exposure to such events theyrespond to them as if they are synchronous (Fujisaki et al., 2004; Navarra et al.,2005; Vroomen et al., 2004). In other words, short-term adaptation to audio-visuallyasynchronous events appears to broaden the ITCW in adults. If this adaptationeffect is partly due to an experience-dependent synchrony bias that is due to alifetime of nearly exclusive exposure to synchronous events – resulting in whatWelch and Warren (1980) call the “unity assumption” then infants may not exhibitsuch a bias because of their relative lack of experience with synchronous multi-sensory events. Lewkowicz (2010) conducted a series of experiments to test thisprediction. In the first experiment, 4- to 10-month-old infants were habituated toan audio-visually synchronous audio-visual syllable and then were tested to see ifthey could detect three increasing levels of asynchrony (i.e., 366, 500, and 666 ms).Figure 16.1a shows the data from the habituation trials – indicating that infantshabituated to the synchronous audio-visual syllable – and Fig. 16.1b shows theresults from the test trials. Planned contrast analyses of the test trial data, com-paring the duration of looking in each novel test trial, respectively, with the durationof looking in the FAM 0 test trial indicated that response recovery was not sig-nificant in the NOV 366 and NOV 500 test trials but that it was significant inthe NOV 666 test trial. These results are consistent with previous findings ofinfant detection of asynchrony in showing that infants do not detect A-V speechasynchrony below 633 ms. In the second experiment, 4- to 10-month-old infantswere habituated to an asynchronous syllable (666 ms) and were then tested fortheir ability to detect decreasing levels of asynchrony (i.e., 500, 366, and 0 ms).Figure 16.2a shows the data from the habituation trials – indicating that infantshabituated to the asynchronous audio-visual syllable – and Fig. 16.2b shows theresults from the test trials. Planned contrast analyses of the test trial data, compar-ing the duration of looking in each novel test trial, respectively, with the duration oflooking in the FAM 666 test trial indicated that infants did not exhibit significant

310 D.J. Lewkowicz

Fig. 16.1 Infant detection of A-V synchrony relations following habituation to a synchronousaudio-visual syllable. (a) The mean duration of looking during the first three (A, B, and C) and thelast three (X, Y, and Z) habituation trials is shown. (b) The mean duration of looking in responseto the various levels of asynchrony during the test trials is shown. Error bars indicate the standarderror of the mean

Fig. 16.2 Infant detection of A-V synchrony relations following habituation to an asynchronousaudio-visual syllable. (a) The mean duration of looking during the habituation phase is shown and(b) the mean duration of looking in response to the various levels of asynchrony during the testtrials is shown. Error bars indicate the standard error of the mean

response recovery in the NOV 500 test trial but that they did in the NOV 366and the NOV 0 test trials. The pattern of responsiveness in this experiment isopposite to what might be expected on the basis of the adult adaptation findings.Rather than exhibit broadening of the ITCW following short-term adaptation toa discriminable A-V asynchrony, the infants in the second experiment exhibited


narrowing of the ITCW in that they discriminated not only between the 666 msasynchrony and synchrony (0 ms) but also between the 666 ms asynchrony and anasynchrony of 366 ms.

Together, the findings from studies of infant response to audio-visual speech indi-cate that it is more difficult for infants to detect a desynchronization of the auditoryand visual streams of information when that information is a speech syllable. Thatis, following initial learning of a synchronous audio-visual speech token infants onlydetect an asynchrony of 666 ms, whereas following initial learning of a synchronousbouncing/sounding object infants detect an asynchrony of 350 ms, a difference ofmore than 300 ms. The most likely reason for this difference is that the audio-visual speech signal consists of continuous changes in facial gestures and vocalinformation, making it more difficult to identify the precise point where the desyn-chronization occurs. The findings also indicate that an initial short-term experiencewith an asynchronous speech event facilitates subsequent detection of asynchronyin infants, an effect opposite to that found in adults. This seemingly unexpectedadaptation effect actually makes sense when the infant’s relative lack of perceptualexperience and the presumed lack of a unity assumption are taken into account. Inthe absence of a unity assumption, short-term exposure to an asynchronous multi-sensory event does not cause infants to treat it as synchronous but rather focusestheir attention on the event’s temporal attributes and, in the process, appears tosharpen their detection of the A-V temporal relationship.

16.4.3 Binding of Multisensory Attributes

A number of studies have shown that from birth on infants can bind spatiotempo-rally congruent abstract objects and the sounds that accompany them and that theycan bind human faces and the vocalizations they make (Bahrick, 1983, 1988, 1992;Brookes et al., 2001; Lewkowicz, 1992a, b, 2000b, 2003; Reardon and Bushnell,1988; Slater et al., 1997). What makes these findings particularly interesting is thatdespite the fact that infants are able to detect A-V temporal synchrony relationsfrom a very young age on, and the fact that their asynchrony detection thresholdsdo not change during the first year of life (Lewkowicz, 1996, 2010), infants’ abilityto associate different types of multisensory attributes varies with age and dependson the familiarity of the attributes. Thus, when the attributes are human faces andvoices, infants as young as 3 months of age can associate them (Brookes et al.,2001), but when the attributes are various object attributes such as color, shape,taste, or temperature, only older infants are able to associate them. For example,whereas neither 3.5- nor 5-month-old infants can associate the color/shape of anobject and the pitch of an accompanying sound, 7-month-old infants can (Bahrick,1992, 1994). Likewise, only 6-month-old infants can bind the color or pattern ofan object with its particular shape (Hernandez-Reif and Bahrick, 2001). Finally, itis not until 7 months of age that infants can bind the color of an object and itstaste (Reardon and Bushnell, 1988) and even at this age they do not associate colorand temperature (Bushnell, 1986). Together, this set of findings indicates that the

312 D.J. Lewkowicz

ability to bind familiar modality-specific object properties emerges relatively earlyin infancy but that the ability to form more arbitrary associations of less familiarobject properties emerges later. This suggests that sensitivity to A-V temporal syn-chrony relations can facilitate intersensory binding but that the facilitation is greatestfor familiar modality-specific object properties.

16.4.4 Binding of Nonnative Faces and Vocalizations

Although the familiarity of modality-specific attributes seems to contribute tosuccessful intersensory binding, recent evidence from our studies indicates that sen-sitivity to synchronous intersensory relations is much broader in younger than inolder infants. This evidence comes from one of our recent studies (Lewkowicz andGhazanfar, 2006) in which we found that young infants bind nonnative faces andvocalizations but that older infants do not. The developmental pattern of initial broadperceptual tuning followed by narrowing of that tuning a few months later was notconsistent with the conventional view that development is progressive in nature andthat it usually leads to a broadening of perceptual skills. It was consistent, how-ever, with a body of work on infant unisensory perceptual development showingthat some forms of unisensory perceptual processing also narrow during the firstyear of life. For example, it has been found that young infants can detect nonnativespeech contrasts but that older infants do not (Werker and Tees, 1984) and that younginfants can discriminate nonnative faces (i.e., of different monkeys) as well as thefaces of other races, but that older infants do not (Kelly et al., 2007; Pascalis et al.,2002). Based on these unisensory findings, we (Lewkowicz and Ghazanfar, 2006)asked whether the perceptual narrowing that has been observed in the unisensoryperceptual domain might reflect a general, pan-sensory process. If so, we hypothe-sized that young infants should be able to match nonnative faces and vocalizationsbut that older infants should not. We put this hypothesis to test by showing side-by-side faces of a monkey producing two different visible calls (see Fig. 16.3) to groupsof 4-, 6-, 8-, and 10-month-old infants. During the initial two preference trials weshowed the faces in silence while during the second two trials we showed the facestogether with the audible call that matched one of the two visible calls (Fig. 16.3).The different calls (a coo and a grunt) differed in their durations and, as a result, thematching visible and audible calls corresponded in terms of their onsets and offsetsas well as their durations. In contrast, the non-matching ones only corresponded interms of their onsets.

We expected that infants would look longer at the visible call that matched theaudible call if they perceived the correspondence between them. As predicted, andconsistent with the operation of a perceptual narrowing process, we found that thetwo younger groups of infants matched the corresponding faces and vocalizationsbut that the two older groups did not. These findings confirmed our prediction thatintersensory perceptual tuning is broad early in infancy and that it narrows over time.We interpreted the narrowing effects as a reflection of increasing specialization for


Fig. 16.3 Single video frames showing the facial gestures made by one of the monkeys whenproducing the coo and the grunt. The gestures depicted are at the point of maximum mouth opening.Below the facial gestures are the corresponding sonograms and spectrograms of the audible call

human faces and vocalizations that is the direct result of selective experience withnative faces and vocalizations and a concurrent lack of experience with nonnativeones.

Because the matching faces and vocalizations corresponded in terms of both theironset and offset synchrony and their durations, the obvious question was whetherthe successful matching was based on one or both of these perceptual cues. Thus,in a subsequent study (Lewkowicz et al., 2008), we tested this possibility by repeat-ing the Lewkowicz and Ghazanfar (2006) study except that this time we presentedthe monkey audible calls out of synchrony with respect to both visible calls. Thismeant that the corresponding visible and audible calls were only related in terms oftheir durations. Results indicated that A-V temporal synchrony mediated the suc-cessful matching in the younger infants because this time neither 4- to 6-month-oldnor 8- to 10-month-old infants exhibited intersensory matching. The fact that theyounger infants did not match despite the fact that the corresponding faces andvocalizations still corresponded in terms of their duration shows that duration alone

314 D.J. Lewkowicz

was not sufficient to enable infants to make intersensory matches. Indeed, this lat-ter finding is consistent with previous findings (Lewkowicz, 1986) indicating thatinfants do not match auditory and visual inputs that are equated in terms of theirduration unless the corresponding inputs also are synchronous.

If A-V temporal synchrony mediates cross-species intersensory matching inyoung infants, and if responsiveness to this intersensory perceptual cue depends ona basic and relatively low-level process, then it is possible that cross-species inter-sensory matching may emerge very early in development. To test this possibility,we (Lewkowicz et al., 2010) asked whether newborns also might be able to matchmonkey facial gestures and the vocalizations that they produce. In Experiment 1of this study we used the identical stimulus materials and testing procedures usedby Lewkowicz and Ghazanfar (2006) and, as predicted, found that newborns alsomatched visible monkey calls and corresponding vocalizations (see Fig. 16.4a).Given these results, we then hypothesized that if the successful matching reflectedmatching of the synchronous onsets and offsets of the visible and audible calls thennewborns should be able to make the matches even when some of the identity infor-mation is removed. To test this possibility, we conducted a second experiment wherewe substituted a complex tone for the natural audible call. To preserve the critical

Fig. 16.4 Newborns’ visual preference for matching visible monkey calls in the absence and pres-ence of the matching natural audible call or a tone. (a) The mean proportion of looking at thematching visible call when it was presented during silent and in-sound trials is shown, respec-tively, when the natural audible call was presented. (b) The mean proportion of looking at thematching visible call in the silent and in-sound test trials is shown, respectively, when the complextone was presented. Error bars represent the standard errors of the mean


temporal features of the audible call, we ensured that the tone had the same durationas the natural call and that, once again, its onsets and offsets were synchronous withthe matching visible call. Results indicated that despite the absence of identity infor-mation and despite the consequent absence of the correlation between the dynamicvariations in facial gesture information and the amplitude and formant structureinherent in the natural audible call, newborns still performed successful intersen-sory matching (see Fig. 16.4b). These results suggest that newborns’ ability to makecross-species matches in Experiment 1 was based on their sensitivity to the tempo-rally synchronous onsets and offsets of the matching faces and vocalizations andthat it was not based on their identity information nor on the dynamic correlationbetween the visible and audible call features.

Together, the results from the two experiments with newborns demonstrate thatthey are sensitive to a basic feature of their perceptual world, namely stimulusenergy onsets and offsets. As suggested earlier, it is likely that this basic sensi-tivity bootstraps newborns’ entry into the world of multisensory objects and enablesthem to detect the coherence of multisensory objects despite the fact that this pro-cess ignores the specific identity information inherent in the faces and vocalizations.Moreover, it is interesting to note that our other recent studies (Lewkowicz, 2010)have shown that the low-level sensitivity to stimulus onsets and offsets continuesinto the later months of life; this is indicated by the finding that infants between4 and 10 months of age detect an A-V desynchronization even when the stimulusconsists of a human talking face and a tone stimulus. Thus, when the results fromthe newborn study are considered together with the results from this latter studywith older infants it appears that sensitivity to a relatively low-level kind of inter-sensory relation makes it possible for infants to detect the temporal coherence ofmultisensory inputs well into the first year of life.

Needless to say, although an early sensitivity to energy onsets and offsets pro-vides infants with a very powerful and useful cue for discovering the coherence ofmultisensory objects and events, it has its obvious limitations. In particular, if younginfants are primarily responsive to energy onsets and offsets then the finding that it isnot until the latter half of the first year of life that infants become capable of makingarbitrary intersensory associations (e.g., color/shape and pitch or color and taste)makes sense. That is, making an association between a static perceptual attributesuch as shape, color, or temperature is far more difficult than between attributesrepresenting dynamic naturalistic events where energy transitions are available andclear. For example, when we watch and hear a person talking, we can see whenthe person’s lips begin and stop moving and can hear the beginning and end of theaccompanying vocalizations. In contrast, when we watch and hear a person talk-ing and need to detect the correlation between the color and shape of the person’sface and the pitch of the person’s voice, it is more difficult to detect the correlationbecause the color and shape of the person’s face do not vary over time. Likewise,because the shape and color of an object are features that do not vary over time, theircorrelation with an object’s taste (e.g., an apple) is arbitrary and, thus, less salientto an infant who is primarily responsive to energy onsets and offsets as well as thedynamic properties of objects.

316 D.J. Lewkowicz

The pervasive and fundamental role that A-V temporal synchrony plays in infantperceptual response to multisensory attributes suggests that sensitivity to this inter-sensory perceptual cue reflects the operation of a fundamental early perceptualmechanism. That is, even though sensitivity to A-V temporal synchrony is mediatedby relatively basic and low-level processing mechanisms, as indicated earlier, thisintersensory relational cue provides infants with a powerful initial perceptual toolfor gradually discovering, via the process of perceptual learning and differentiation(E. J. Gibson, 1969), that multisensory objects are characterized by other forms ofintersensory invariance. In essence, the initial developmental pattern consists of thediscovery of basic synchrony-based intersensory relations enabling infants to per-ceive multisensory objects and events as coherent entities. This, however, does notpermit them to perceive higher-level, more complex features. For example, at thispoint infants essentially ignore the rich information that is available in-between theenergy onsets and offsets of auditory and visual stimulation. The situation changes,however, once infants discover synchrony-based multisensory coherence becausenow they are in a position to proceed to the discovery of the more complex infor-mation that is located “inside” the stimulus. For example, once infants start to bindthe audible and visible attributes of talking faces, they are in a position to discoverthat faces and the vocalizations that accompany them also can be specified by com-mon duration, tempo, and rhythm, as well as by higher-level amodal and invariantattributes such as affect, gender, and identity.

Newborns’ sensitivity to A-V temporal synchrony relations is particularly inter-esting when considered in the context of the previously discussed sensitivity to theeffective intensity of multisensory stimulation. One possibility is that newborn’sability to detect stimulus energy onsets and offsets in A-V synchrony detectiontasks is actually directly related to their sensitivity to multisensory intensity. Inother words, it may be that A-V intensity and temporal synchrony detection mech-anisms work together to bootstrap newborns’ entry into the multisensory world andthat, together, they enable newborns to discover a coherent, albeit relatively simple,multisensory world. As they then grow and learn to differentiate increasingly fineraspects of their perceptual world, they gradually construct an increasingly morecomplex picture of their multisensory world.

16.4.5 The Importance of Spatiotemporal Coherencein Object Perception

In addition to permitting infants to bind and match auditory and visual objectattributes, evidence from my laboratory and that of my colleagues (Scheier et al.,2003) has shown that A-V spatiotemporal synchrony cues can help infants dis-ambiguate what are otherwise ambiguous visual events. Specifically, when twoidentical visual objects move toward one another, pass through each other, and thencontinue to move away, the majority of adults watching such a display report seeingthe two objects streaming through one another. When, however, a simple sound is


presented at precisely the point when the objects coincide, a majority of adults reportthat the objects now bounce against each other (Sekuler et al., 1997). The specificspatiotemporal relationship between the sound and the motion of the two objectsis responsible for this illusion and helps resolve the visual ambiguity (Watanabeand Shimojo, 2001). We (Scheier et al., 2003) asked whether infants might alsoprofit from such A-V spatiotemporal relations to resolve visual ambiguity. Giventhe already reviewed evidence of the power of A-V temporal synchrony to organizethe infant’s multisensory world, it would be reasonable to expect that infants mightalso exhibit the bounce illusion. To test this possibility, we habituated 4-, 6-, and8-month-old infants either to the streaming display with the sound occurring at thepoint of coincidence or to the same display with the sound presented either prior tocoincidence or after it. Then, we tested infants in the two groups with the oppositecondition. Results indicated that both groups exhibited response recovery at 6 and 8months of age but not at 4 months of age, indicating that the two older age groupsdetected the specific spatiotemporal relationship between the sound and the spatialposition of the moving objects. Thus, the specific temporal relationship between asound and an ambiguous visual event can disambiguate the event for infants startingat 6 months of age. Given that this phenomenon is dependent on attentional factors(Watanabe and Shimojo, 2001), we interpreted these findings to mean that the emer-gence of this multisensory object perception system reflects the emergence of moreadvanced attentional mechanisms located in the parietal cortex that can quickly andflexibly switch attention. In essence, one perceives the bounce illusion when atten-tion to the motion of the objects is briefly interrupted by the sound. This, in turn,requires the operation of parietal attentional mechanisms that emerge by around6 months of age (Ruff and Rothbart, 1996).

16.4.6 Summary of Effects of A-V Temporal Synchronyon Infant Perception

Overall, the findings to date on infant response to A-V temporal synchrony and theirreliance on it for the perception of coherent multisensory objects yield the followinginterim conclusions. First, infants are sensitive to A-V temporal synchrony relationsfrom birth on and this sensitivity does not appear to change during early humandevelopment but does between infancy and adulthood. Second, response to A-Vtemporal synchrony relations appears to be based mainly on sensitivity to stimulusenergy onsets and offsets. Third, despite the absence of age-related changes in theA-V asynchrony detection threshold during infancy (a) the threshold can be mod-ified by short-term experience and (b) the effects of such experience are oppositeto those found in adults. Fourth, early infant sensitivity to A-V temporal synchronyrelations is so broad that it permits younger but not older infants to even bind non-native facial gestures and accompanying vocalizations. Fifth, infant ability to bindmodality-specific multisensory attributes on the basis of A-V temporal synchronychanges over the first months of life in that young infants can bind the audible

318 D.J. Lewkowicz

and visible attributes of familiar and naturalistic types of objects and events butonly older infants can bind the multisensory attributes that specify more abstract,less common, types of relations. Finally, response to A-V temporal synchrony cuescan be overridden by competing temporal pattern cues during the early months oflife and only older infants can respond to synchrony cues when they compete withrhythmic pattern cues.

16.5 Response to A-V Colocation

Findings from studies of infant response to multisensory colocation cues, likethose from studies of infant response to A-V temporal synchrony cues, also haveyielded evidence of developmental changes. In the aggregate, these findings indi-cate that even though infants are sensitive to multisensory colocation cues, theyalso exhibit major changes in the way they respond to such cues (Morrongiello,1994). Thus, even though starting at birth infants exhibit coordinated auditory andvisual responses to lateralized sounds, these kinds of responses are reflexive innature and, as a result, do not constitute adequate evidence of true intersensory per-ception. Nonetheless, despite their being reflexive, such responses provide infantswith an initial opportunity to experience collocated auditory and visual stimulusattributes. As a result, from birth on infants have coordinated multisensory experi-ences and have an initial basis for gradually learning to expect to see objects wherethey hear sounds and, in the process, can construct a coordinated map of audio-visual space.

The developmental improvement in the ability to perceive spatial colocation islikely due to a combination of maturational and experiential factors working in con-cert with one another. In terms of sheer sensory processing capacity, the auditory andvisual systems undergo major changes. In the auditory system, accuracy of soundlocalization changes dramatically during infancy as indicated by the fact that theminimum audible angle is 27◦ at 8 weeks, 23.5◦ at 12 weeks, 21.5◦ at 16 weeks,20◦ at 20 weeks, 18◦ at 24 weeks, and 13.5◦ at 28 weeks of age (Morrongiello et al.,1990) and then declines more slowly reaching a minimum audible angle of around4◦ by 18 months of age (Morrongiello, 1988). In the visual modality, visual fieldextent is relatively small early in infancy – its magnitude depends on the specificperimetry method used but is around 20◦ at 3.5 months – and then roughly dou-bles in the nasal visual field and roughly triples in the lateral visual field, reachingadult-like levels by 6–7 months of age (Delaney et al., 2000). Although data oninfants younger than 3.5 months of age are not available, it is safe to assume that theextent of the visual field is smaller in younger infants. In addition to the dramaticchanges in the minimum audible angle and visual field extent during infancy, majorchanges occur in spatial resolution, saccadic localization, smooth pursuit abilities,and localization of moving objects (Colombo, 2001). The fact that all of these sen-sory/perceptual skills are rather poor at first and then gradually improve over thefirst months of life means that localization and identification of objects is rather


crude early in infancy and only improves gradually. Doubtless, as the various sen-sory/perceptual skills improve, concurrent experiences with collocated auditory andvisual sources of stimulation help refine multisensory localization abilities.

My colleagues and I conducted a comprehensive study of infant localization ofauditory, visual, and audio-visual stimulation across the first year of life (Neil et al.,2006). In this study, we investigated responsiveness to lateralized unisensory andbisensory stimuli in 2-, 4-, 6-, 8-, and 10-month-old infants as well as adults usinga perimetry device that allows the presentation of auditory, visual, or audio-visualtargets at different eccentricities. We presented targets at 25◦ or 45◦ to the right andleft of the subjects. At the start of each trial, the infant’s attention was centered bypresenting a central stimulus consisting of a set of flashing light-emitting diodes(LEDs) and bursts of white noise. As soon as the infant’s attention was centered,the lateralized targets – a vertical line of flashing LEDs, a burst of white noise, orboth – were presented at one of the two eccentricities and either to the subject’sright or left in a random fashion. To determine whether and how quickly infantslocalized the targets, we measured latency of eye movements and/or head turns tothe target.

As expected, response latency decreased as a function of age regardless of typeof stimulus and side on which it was presented. Second, even the 8- to 10-month-oldinfants’ response latencies were longer than those found in adults, indicating that theorienting systems continue to improve beyond this age. Third, response latency wasslowest to the auditory targets, faster to visual targets, and fastest to the audio-visualtargets, although the difference in latency to visual as opposed to audio-visual wasrelatively small. Of particular interest from the standpoint of the development of amultisensory object concept, we found the greatest age-related decline in responselatency to audio-visual targets during the first 6 months of life, suggesting that itis during the first 6 months of life that infants are acquiring the ability to localizethe auditory and visual attributes of objects in a coordinated fashion. Consistentwith this pattern, it was only by 8–10 months of age that infants first exhibitedsome evidence of an adult-like non-linear summation of response latency to audio-visual as opposed to auditory or visual targets. In other words, prior to 8 months ofage responsiveness to audio-visual stimuli reflected the faster of the two unisensoryresponses whereas by 8 months of age it reflected non-linear summation for the firsttime. Importantly, however, this kind of multisensory enhancement was only foundin the 8- to 10-month-old infants’ response to targets presented at 25◦. The absenceof this effect at 45◦ indicates that integration of auditory and visual localization sig-nals is still not fully mature by 8–10 months of age and, therefore, that it undergoesfurther improvement past that age. In sum, these results are consistent with a con-structivist account by showing that adult-like localization responses do not emergeuntil the end of the first year of life and that even then they are not nearly as goodas they are in adults.

When the kinds of developmental changes in infant localization responsesdescribed above are considered together with the previously cited evidence oflong-term changes in A-V temporal synchrony thresholds, they suggest that themultisensory object concept develops slowly during infancy and probably continues

320 D.J. Lewkowicz

to develop well past the first year of life. In brief, the evidence shows that the abilityto integrate various multisensory object attributes is quite poor at birth and that itemerges gradually over the first months of postnatal life. This gradual emergenceis likely to reflect the concurrent development of a myriad of underlying and inter-dependent factors such as neural growth and differentiation, improvement in sensorythresholds and sensory processing, improvement in various motor response systems,and accumulating sensory, perceptual, and motor experience. Given that so manyunderlying processes are changing and interacting with one another during earlydevelopment, and given that the nature of the information to be integrated and thespecific task requirements affect integration, it is not surprising that the various andheterogeneous intersensory integration skills emerge in a heterochronous fashion(Lewkowicz, 2002).

Regardless of the heterochronous emergence of heterogeneous intersensory inte-gration skills and regardless of the fact that the mature multisensory object conceptemerges relatively slowly, there is little doubt that young infants have some rudi-mentary integration skills that enable them to embark on the path to adult-likeintegration. Nonetheless, the relatively slow emergence of the multisensory objectconcept means that nativist accounts that claim that the object concept is presentat birth are incorrect and that they mis-characterize the complex developmentalprocesses underlying its emergence. This conclusion receives additional supportfrom studies of the development of the unisensory (i.e., visual) object concept. Forexample, it is well-known that 4-month-old infants can take advantage of the spa-tiotemporal coherence of the parts of a moving but seemingly incomplete objectto perceive the parts as belonging to a single object (Kellman and Spelke, 1983).Consistent with the development of multisensory object coherence, studies haveshown that the ability to perceive coherent visual objects is not present at birthand that it only emerges gradually during the first months of life (S. P. Johnson,2004). Furthermore, its gradual emergence is partly due to the development of avoluntary, cortical, and attention-driven saccadic eye movement system that slowlycomes to dominate an initially reflexive, subcortically controlled, saccadic system(M. H. Johnson, 2005).

16.6 Perception of Multisensory Sequences in Infancy

Multisensory objects often participate in complex actions that are sequentially orga-nized. For example, when people speak they produce sequences of vocalizationsalong with highly correlated facial gestures. Similarly, when a drummer plays thedrum he produces a patterned series of sounds that are correlated with the patternof his hand and arm motions. In both cases, the patterns produced are perceived asunitary events that carry specific meanings. In the case of an audio-visual speechutterance, it is the syntactically prescribed order of the syllables and words thatimbue the utterance with a specific meaning. In the case of the drummer, it is theorder of different drum beats that imbues the musical passage with specific meaning


(e.g., a particular rhythm and mood). Infants must at some point become capableof perceiving, learning, and producing sequences to function adaptively. This factraises the obvious question of when this ability might emerge in development. Priortheoretical accounts have claimed that sequence perception and learning abilities areinnate (Greenfield, 1991; Nelson, 1986). More recent research on infant pattern andsequence perception has clearly demonstrated, however, that this is simply not thecase.

Although pattern and sequence perception skills are related, they also are distinct.They are related because both enable perceivers to detect the global organization ofa set of distinct stimulus elements and they are distinct because only sequence per-ception skills enable the extraction of specific ordinal relations among the distinctelements making up specific sequences. Bearing the similarities and differencesin mind, studies of pattern perception have shown that infants are sensitive tounisensory as well as bisensory rhythmic patterns from birth on (Lewkowicz, 2003;Lewkowicz and Marcovitch, 2006; Mendelson, 1986; Nazzi et al., 1998; Pickensand Bahrick, 1997) and, as shown below, studies also have shown that infantsexhibit rudimentary sequence perception and learning abilities. Although no studiesto date have investigated sequence perception and learning at birth, the extant evi-dence indicates that infants can learn adjacent and distant statistical relations, simplesequential rules, and ordinal position information. Of particular relevance to the cur-rent argument that the multisensory object concept is constructed during early lifeis the fact that these various sequence perception and learning abilities emerge atdifferent points in infancy.

The earliest sequence learning ability to emerge is the ability to perceive andlearn adjacent statistical relations. Thus, beginning as early as 2 months of ageinfants can learn the adjacent statistical relations that link a series of looming visualshapes (Kirkham et al., 2002; Marcovitch and Lewkowicz, 2009), by 8 months ofage they can learn the statistical relations that link adjacent static object features(Fiser and Aslin, 2002) as well as adjacent nonsense words in a stream of sounds(Saffran et al., 1996), and by 15 months of age infants begin to exhibit the ability tolearn distant statistical relations (Gómez and Maye, 2005). The most likely reasonwhy the ability to perceive and learn adjacent statistical relations emerges earliest isbecause it only requires the perception and learning of the conditional probabilityrelations between adjacent sequence elements and, thus, only requires the forma-tion of paired-associates. The more complex ability to learn abstract sequential rules(e.g., AAB vs. ABB) emerges by 5 months of age when they are specified by abstractobjects and accompanying speech sounds (Frank et al., 2009), by 7.5 months of agewhen the rules are instantiated by nonsense syllables (Marcus et al., 2007; Marcuset al., 1999), and by 11 months of age when the rules are instantiated by loomingobjects (S. P. Johnson et al., 2009). Finally, it is not until 9 months of age that infantscan track the ordinal position of a particular syllable in a string of syllables (Gerken,2006).

It is interesting to note that most of the studies of infant sequence learningto date have investigated this skill by presenting unisensory stimuli. As alreadynoted, however, our world is largely multisensory in nature and multisensory

322 D.J. Lewkowicz

redundancy facilitates learning and discrimination (Bahrick et al., 2004; Lewkowiczand Kraebel, 2004). Consequently, it is important to determine how and wheninfants are able to perceive and learn multisensory sequences. With this goal inmind, we have conducted several studies of sequence perception and learning ininfancy. In some of these studies, we provided infants with an opportunity to learn asingle audio visual sequence consisting of distinct moving objects and their impactsounds whereas in others we allowed infants to learn several different sequenceswith each composed of different objects and their distinct impact sounds. In eithercase, during the habituation phase, the different objects could be seen appearing oneafter another at the top of a computer monitor and then moving down toward a rampat the bottom of the stimulus display monitor. When each object reached the ramp,it produced its distinct impact sound, turned to the right and moved off to the sideand disappeared. This cycle was repeated for the duration of each habituation trial.Following habituation, infants were given test trials during which the order of one ormore of the various sequence elements was changed and the question was whetherinfants detected it.

In an initial study (Lewkowicz, 2004), we asked whether infants can learn asequence composed of three moving/impacting objects and, if so, what aspects ofthat sequence they encoded. Results indicated that 4-month-old infants detectedserial order changes only when the changes were specified concurrently by audi-ble and visible attributes during both the learning and the test phase, and only whenthe impact part of the event – a local event feature that by itself was not informativeabout the overall sequential order – was blocked from view. In contrast, 8-month-oldinfants detected order changes regardless of whether they were specified by unisen-sory or bisensory attributes and regardless of whether they could see the impact thatthe objects made or not. In sum, younger infants required multisensory redundancyto learn the sequence and to detect the serial order changes whereas the older infantsdid not.

In a follow-up study (Lewkowicz, 2008), we replicated the earlier findings,ruled out primacy effects, extended the earlier findings by showing that even3-month-old infants can perceive, learn, and discriminate three-element dynamicaudio visual sequences, and that 3-month-olds also require multisensory redun-dancy to successfully learn the sequences. In addition, we found that object motionplays an important role because infants exhibited less robust responsiveness toaudio visual sequences consisting of looming/sounding objects than of explicitlymoving/sounding objects. Finally, we found that 4-month-old infants can perceiveand discriminate longer (i.e., 4-element) sequences, that they exhibit more robustresponsiveness to the longer sequences, and, somewhat surprisingly, that they do soeven when they can see the impact part of the event. At first blush, this last resultappeared to be paradoxical in relation to the earlier results with 4-month-olds butwhen the fact that a sequence composed of more objects/sounds makes it harderto focus on the individual objects is taken into account, this finding makes sense.That is, when there are more objects/sounds, infants’ attention is shifted to the moreglobal aspects of the sequence because there is less time to attend to each individualobject and its impact sound.


The two previous studies established firmly that young infants are able to per-ceive and learn audio visual sequences and that they can detect changes in the orderof the sequence elements. What is not clear from these findings, however, is whatspecific sequence property underlies infants’ ability to detect order changes. Asindicated earlier, infants are sensitive to statistical relations from an early age. Thechanges in sequential order presented in our initial two studies involved changesnot only in the order of a particular object and its sound but also in its statisticalrelations to other sequence elements. As a result, it was necessary to investigatethe separate contribution of each of these sequential attributes to infant sequencelearning and discrimination. We did so in our most recent study (Lewkowicz andBerent, 2009). Here, we investigated directly whether 4-month-old infants can trackthe statistical relations among specific sequence elements (e.g., AB, BC) and/orwhether they also can encode abstract ordinal position information (e.g., that B isthe second element in a sequence of ABC). Thus, across three experiments we habit-uated infants to sequences of four moving/sounding objects. In these sequences,three of the objects and their sounds varied in their ordinal positions whereasone target object/sound maintained its invariant ordinal position (e.g., ABCD andCBDA). Following habituation to such sequences, we then presented sequenceswhere the target sequence element’s ordinal position was changed and the questionwas whether infants detected this change. Figure 16.5 shows that when the ordi-nal change disrupted the statistical relations between adjacent sequence elements,infants exhibited significant response recovery (i.e., discrimination). If, however,the statistical relations were controlled for when the ordinal change was made (i.e.,

Fig. 16.5 Infant learning and discrimination of audio visual sequences. (a) The duration of lookingduring the first three (A, B, and C) and last three (X, Y, and Z) habituation trials is shown. (b) Themean duration of looking in the test trials when the target element changed its ordinal positionand statistical relations were disrupted and when only ordinal relations were disrupted. Error barsindicate the standard error of the mean

324 D.J. Lewkowicz

when no statistical relations were disrupted), infants did not exhibit evidence of suc-cessful learning and discrimination. These results indicate that 4-month-old infantscan learn the order of sequence elements and that they do so by tracking their statis-tical relations but that they do not track their invariant ordinal position. When thesefindings are combined with the previously reviewed findings on sequence learningin infancy, they clearly show that different sequence learning abilities emerge atdifferent times, with more sophisticated ones emerging later than less sophisticatedones. In addition, it is reasonable to assume that the developmental emergence ofthese different sequence learning abilities is largely a function of experience andthat, as a result, these abilities are constructed early in life by infants through theirinteractions with their everyday spatiotemporally and sequentially structured world.

16.7 Conclusion

Infants’ perception and conception of multisensory objects is a developmental prod-uct of a complex set of dynamic and ever-changing processes. As is obvious fromthe foregoing, major changes in how infants respond to multisensory objects occurduring infancy. The challenge of explicating the full set of interactions that con-tribute to this critical skill is still ahead of us. The hope is that the current review ofsome of the extant empirical evidence on infants’ response to multisensory objectswill spur further inquiry into this fundamental topic and will shed additional light onthe exquisite complexity of the developmental processes underlying the emergenceof the multisensory object concept.

References

Bahrick LE (1983) Infants’ perception of substance and temporal synchrony in multimodal events.Infant Behav Dev 6(4): 429–451

Bahrick LE (1988) Intermodal learning in infancy: learning on the basis of two kinds of invariantrelations in audible and visible events. Child Dev 59:197–209

Bahrick LE (1992) Infants’ perceptual differentiation of amodal and modality-specific audio-visualrelations. J Exp Child Psychol 53:180–199

Bahrick LE (1994) The development of infants’ sensitivity to arbitrary intermodal relations. EcologPsychol 6(2):111–123

Bahrick LE, Lickliter R, Flom R (2004) Intersensory redundancy guides the development ofselective attention, perception, and cognition in infancy. Curr Dir Psycholog Sci 13(3):99–102

Birch HG, Lefford A (1963) Intersensory development in children. Monogr Soc Res Child Dev25(5):1–48

Bremner AJ, Holmes NP, Spence C (2008) Infants lost in (peripersonal) space? Trends Cogn Sci12(8):298–305

Brookes H, Slater A, Quinn PC, Lewkowicz DJ, Hayes R, Brown E (2001) Three-month-old infantslearn arbitrary auditory-visual pairings between voices and faces. Infant Child Dev 10(1–2):75–82

Bushara KO, Grafman J, Hallett M (2001) Neural correlates of auditory-visual stimulus onsetasynchrony detection. J Neurosci 21(1):300–304


Bushnell EW (1986) The basis of infant visual-tactual functioning: Amodal dimensions ormultimodal compounds? Adv Infan Res 4:182–194

Calvert G, Spence C, Stein B (eds) (2004) The handbook of multisensory processes. MIT Press,Cambridge, MA

Cohen LB, Chaput HH, Cashon CH (2002) A constructivist model of infant cognition. Cogn Dev.Special Issue: Constructivism Today 17(3–4):1323–1343

Colombo J (2001) The development of visual attention in infancy. Annu Rev Psychol 52:337–367Delaney SM, Dobson V, Harvey EM, Mohan KM, Weidenbacher HJ, Leber NR (2000) Stimulus

motion increases measured visual field extent in children 3.5 to 30 months of age. Optom VisSci 77(2):82–89

Fiser J, Aslin RN (2002) Statistical learning of new visual feature combinations by infants. ProcNatl Acad Sci 99(24):15822–15826

Frank MC, Slemmer JA, Marcus GF, Johnson SP (2009) Information from multiple modalitieshelps 5-month-olds learn abstract rules. Dev Sci 12:504–509

Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity.Nat Neurosci 7(7):773–778

Gerken L (2006) Decisions, decisions: infant language learning when multiple generalizations arepossible. Cogn 98(3):B67–B74

Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends Cogn Sci10(6):278–285. Epub 2006 May 2018

Gibson EJ (1969) Principles of perceptual learning and development. Appleton, New YorkGibson JJ (1966) The senses considered as perceptual systems. Houghton-Mifflin, BostonGómez RL, Maye J (2005) The developmental trajectory of nonadjacent dependency learning.

Infancy 7(2):183–206Greenfield PM (1991) Language, tools and brain: the ontogeny and phylogeny of hierarchically

organized sequential behavior. Behav Brain Sci 14(4):531–595Hernandez-Reif M, Bahrick LE (2001) The development of visual-tactual perception of objects:

modal relations provide the basis for learning arbitrary relations. Infancy 2(1):51–72Johnson MH (2005) Developmental cognitive neuroscience, 2nd edn. Blackwell, LondonJohnson SP (2004) Development of perceptual completion in infancy. Psychol Sci 15(11):769–775Johnson SP, Fernandes KJ, Frank MC, Kirkham N, Marcus GF, Rabagliati H et al. (2009) Abstract

rule learning for visual sequences in 8- and 11-month-olds. Infancy 14:2–18Kellman PJ, Spelke ES (1983) Perception of partly occluded objects in infancy. Cogn Psychol

15(4):483–524Kelly DJ, Quinn PC, Slater AM, Lee K, Ge L, Pascalis O (2007) The other-race effect develops

during infancy: evidence of perceptual narrowing. Psycholog Sci 18(12):1084–1089Kirkham NZ, Slemmer JA, Johnson SP (2002) Visual statistical learning in infancy: evidence for

a domain general learning mechanism. Cognition 83(2):B35–B42Lewkowicz DJ (1986) Developmental changes in infants’ bisensory response to synchronous

durations. Infant Behav Dev 9(3):335–353Lewkowicz DJ (1992a). Infants’ response to temporally based intersensory equivalence: the effect

of synchronous sounds on visual preferences for moving stimuli. Infant Behav Dev 15(3):297–324

Lewkowicz DJ (1992b). Infants’ responsiveness to the auditory and visual attributes of asounding/moving stimulus. Percept Psychophys 52(5):519–528

Lewkowicz DJ (1996) Perception of auditory-visual temporal synchrony in human infants. J ExpPsychol: Hum Percept Perform 22(5):1094–1106

Lewkowicz DJ (2000a). The development of intersensory temporal perception: an epigeneticsystems/limitations view. Psycholog Bull 126(2):281–308

Lewkowicz DJ (2000b). Infants’ perception of the audible, visible and bimodal attributes ofmultimodal syllables. Child Dev 71(5):1241–1257

Lewkowicz DJ (2002) Heterogeneity and heterochrony in the development of intersensoryperception. Cogn Brain Res 14:41–63

326 D.J. Lewkowicz

Lewkowicz DJ (2003) Learning and discrimination of audiovisual events in human infants: thehierarchical relation between intersensory temporal synchrony and rhythmic pattern cues. DevPsychol 39(5): 795–804

Lewkowicz DJ (2004) Perception of serial order in infants. Dev Sci 7(2):175–184Lewkowicz DJ (2008) Perception of dynamic and static audiovisual sequences in 3- and 4-month-

old infants. Child Dev 79(5):1538–1554Lewkowicz DJ (2010) Infant perception of audio-visual speech synchrony. Dev Psycholog

46(1):66–77Lewkowicz DJ, Berent I (2009) Sequence learning in 4 month-old infants: do infants represent

ordinal information? Child Dev 80(6):1811–1823Lewkowicz DJ, Ghazanfar AA (2006) The decline of cross-species intersensory perception in

human infants. Proc Natl Acad Sci U S A 103(17):6771–6774Lewkowicz DJ, Kraebel K (2004) The value of multimodal redundancy in the development of

intersensory perception. In: Calvert G, Spence C, Stein B (eds) Handbook of multisensoryprocessing. MIT Press, Cambridge, pp 655–678

Lewkowicz DJ, Leo I, Simion F (2010) Intersensory perception at birth: newborns match non-human primate faces & voices. Infancy 15(1):46–60

Lewkowicz DJ, Lickliter R (eds) (1994) The development of intersensory perception: comparativeperspectives. : Lawrence Erlbaum Associates, Inc., Hillsdale, NJ

Lewkowicz DJ, Marcovitch S (2006) Perception of audiovisual rhythm and its invariance in 4- to10-month-old infants. Dev Psychobiol 48:288–300

Lewkowicz DJ, Sowinski R, Place S (2008) The decline of cross-species intersensory percep-tion in human infants: Underlying mechanisms and its developmental persistence. Brain Res1242:291–302. [Epub (ahead of print)]

Lewkowicz DJ, Turkewitz G (1980) Cross-modal equivalence in early infancy: Auditory-visualintensity matching. Dev Psychol 16:597–607

Lewkowicz DJ, Turkewitz G (1981) Intersensory interaction in newborns: modification of visualpreferences following exposure to sound. Child Dev 52(3):827–832

Lickliter R, Bahrick LE (2000) The development of infant intersensory perception: advantages ofa comparative convergent-operations approach. Psycholog Bull 126(2):260–280

Marcovitch S, Lewkowicz DJ (2009) Sequence learning in infancy: the independent contributionsof conditional probability and pair frequency information. Dev Sci 12(6):1020–1025

Marcus GF, Fernandes KJ, Johnson SP (2007) Infant rule learning facilitated by speech. PsychologSci 18(5):387–391

Marcus GF, Vijayan S, Rao S, Vishton P (1999) Rule learning by seven-month-old infants. Sci283(5398):77–80

Marks L (1978) The unity of the senses. Academic Press, New YorkMendelson MJ (1986) Perception of the temporal pattern of motion in infancy. Infant Behav Dev

9(2):231–243Morrongiello BA (1988) Infants’ localization of sounds in the horizontal plane: estimates of

minimum audible angle. Dev Psychol 24:8–13Morrongiello BA (1994) Effects of colocation on auditory-visual interactions and cross-modal

perception in infants. In: Lewkowicz DJ, Lickliter R (eds) The development of intersensoryperception: comparative perspectives. Lawrence Erlbaum, Hillsdale, NJ, pp 235–263

Morrongiello BA, Fenwick KD, Chance G (1990) Sound localization acuity in very young infants:an observer-based testing procedure. Dev Psychol 26:75–84

Navarra J, Vatakis A, Zampini M, Soto-Faraco S, Humphreys W, Spence C (2005) Exposure toasynchronous audiovisual speech extends the temporal window for audiovisual integration.Cogn Brain Res, 25(2):499–507

Nazzi T, Bertoncini J, Mehler J (1998) Language discrimination by newborns: toward anunderstanding of the role of rhythm. J Exp Psychol: Hum Percept Perform 24(3):756–766

Neil PA, Chee-Ruiter C, Scheier C, Lewkowicz DJ, Shimojo S (2006) Development of multisen-sory spatial integration and perception in humans. Dev Sci 9(5):454–464


Nelson K (1986) Event knowledge: structure and function in development. Erlbaum, Hillsdale, NJPartan S, Marler P (1999) Communication goes multimodal. Science 283(5406):1272–1273Pascalis O, Haan M de, Nelson CA (2002) Is face processing species-specific during the first year

of life? Science 296(5571):1321–1323Piaget J (1952) The origins of intelligence in children. International Universities Press, New YorkPiaget J (1954) The construction of reality in the child. Routledge & Kegan, LondonPickens J, Bahrick LE (1997) Do infants perceive invariant tempo and rhythm in auditory-visual

events? Infant Behav Dev 20:349–357Reardon P, Bushnell EW (1988) Infants’ sensitivity to arbitrary pairings of color and taste. Infant

Behav Dev 11(2):245–250Rowe C (1999) Receiver psychology and the evolution of multicomponent signals. Animal Behav

58:921–931Ruff HA, Rothbart MK (1996) Attention in early development: themes and variations. Oxford

University Press, New York, NYSaffran JR, Aslin RN, Newport EL (1996) Statistical learning by 8-month-old infants. Science

274(5294):1926–1928Scheier C, Lewkowicz DJ, Shimojo S (2003) Sound induces perceptual reorganization of an

ambiguous motion display in human infants. Dev Sci 6:233–244Schneirla TC (1965) Aspects of stimulation and organization in approach/withdrawal processes

underlying vertebrate behavioral development. In: Lehrman DS, Hinde RA, Shaw E (eds)Advances in the study of behavior. Academic Press, New York, pp 1–71

Sekuler R, Sekuler AB, Lau R (1997) Sound alters visual motion perception. Nature 385:308Slater A, Brown E, Badenoch M (1997) Intermodal perception at birth: newborn infants’ memory

for arbitrary auditory-visual pairings. Early Dev Parent 6(3–4):99–104Spelke ES, Kinzler KD (2007) Core knowledge. Dev Sci 10(1):89–96Spencer JP, Blumberg MS, McMurray B, Robinson SR, Samuelson LK, Tomblin JB (2009) Short

arms and talking eggs: why we should no longer abide the nativist-empiricist debate. Child DevPerspect 3(2):79–87

Stein BE, Meredith MA (1993) The merging of the senses. The MIT Press, Cambridge, MAStein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the

single neuron. Nat Rev Neurosci 9(4):255–266Summerfield AQ (1979) Use of visual information in phonetic perception. Phonetica 36:314–331Thelen E, Smith LB (1994) A dynamic systems approach to the development of cognition and

action. MIT Press, Cambridge, MAVroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception

by exposure to audio-visual asynchrony. Cogn Brain Res 22(1):32–35Walker-Andrews AS (1997) Infants’ perception of expressive behaviors: differentiation of multi-

modal information. Psycholog Bull 121(3):437–456Watanabe K, Shimojo S (2001) When sound affects vision: effects of auditory grouping on visual

motion perception. Psychol Sci 12(2):109–116Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy.

Psycholog Bull 88:638–667Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds)

Handbook of perception and human performance: Sensory processes and perception, vol. 1.J. Wiley & Sons, New York, pp 1–36

Werker JF, Tees RC (1984) Cross-language speech perception: evidence for perceptual reorganiza-tion during the first year of life. Infant Behav Dev 7(1):49–63

Werner H (1973) Comparative psychology of mental development. International Universities Press,New York

The Ontogeny of Human Multisensory Object Perception: A Constructivist Account

Documents

Transcript of The Ontogeny of Human Multisensory Object Perception: A Constructivist Account