1 IntroductionFaces presented in photographic negative are difficult to recognise (Galper 1970). Theeffect of photographic negation is to preserve edge information whilst disrupting surface-based information (Hayes et al 1986; Johnston et al 1992; Santos and Young 2008).The outcome of the latter is a change in both pigmentation and shape-from-shadinginformation. Disruption to both of these sources of information appears to underliethe difficulties in recognition caused by photographic negation (Bruce and Langton 1994;Kemp et al 1996; Russell et al 2006; Vuong et al 2005). In contrast, perception of facialexpression seems little affected by photographic negation (White 2001). Same/differentjudgments based on either identity or expression showed pronounced increases in reac-tion time and error rates for identity but not for expression judgments. White (2001)explains this pattern of results by noting that facial expressions are readily recognisedfrom line drawings; their recognition (in contrast to identity) is therefore likely to primarilybe dependent upon edge information.

If expression perception is based upon edge information, and is largely unaffectedby surface information, what observable consequences might this have? Surface infor-mation, through shape-from-shading, is an important cue to 3-D structure. If this cueis given little weight, then, in our perception of facial expression, 3-D informationmight be largely ignored. This is supported by studies looking at viewpoint depen-dence in perceived expression (Cavanagh et al 2004; Kappas et al 1994; Lyons et al2000). These studies show that, when tilted forward with respect to the perceiver, a facewith a neutral or ambiguous expression is judged as happier than when that sameface is tilted backwards. The perceived expression difference is likely driven by the factthat, in the orthographic projection of the forward-tilted face, the corners of the mouthare upturned, whereas in the downward-tilted face, they are downturned.

Similarly, when a photograph of a face is folded with three vertical folds (figure 1),then the facial expression may be similarly dependent upon the projected 2-D imageöthe folded 3-D structure seems largely to be ignored (Cavanagh et al 2004). Suchobservations sit well with a view in which our perception of facial expression is basedprimarily on the edge information present in the orthographic projection.

Effect of photographic negation on face expressionaftereffects

Perception advance online publication

Christopher P BentonDepartment of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol BS8 1TU, UK;e-mail: chris.benton@bristol.ac.uk; web: http://psychology.psy.bris.ac.uk/people/chrisbenton.htmReceived 18 May 2009, in revised form 20 July 2009; published online 10 September 2009

Abstract. Our visual representation of facial expression is examined in this study: is this repre-sentation built from edge information, or does it incorporate surface-based information? Toanswer this question, photographic negation of grey-scale images is used. Negation preservesedge information whilst disrupting the surface-based information. In two experiments visualaftereffects produced by prolonged viewing of images of facial expressions were measured. Thisadaptation-based technique allows a behavioural assessment of the characteristics encoded bythe neural systems underlying our representation of facial expression. The experiments show thatphotographic negation of the adapting images results in a profound decrease of expression after-effect. Our visual representation of facial expression therefore appears to not just be built fromedge information, but to also incorporate surface information. The latter allows an appreciationof the 3-D structure of the expressing face that, it is argued, may underpin the subtlety and range ofour non-verbal facial communication.

doi:10.1068/p6468

The paragraphs above develop a line of reasoning supporting the notion that ourperception of facial expression is based on an analysis of edge information that largelyignores 3-D structure. However, it is worth questioning this view and asking whatthese findings or observations actually show. Do they tell us about the nature of ourrepresentations of facial expression; or are they more about the information drawnfrom those representations and the manner of its use?

One way in which we can probe the representation of facial expression is to usethe phenomenon of face adaptation (Rhodes et al 2005; Webster et al 2004). Prolongedviewing of a face changes our percept of subsequently viewed faces; the adaptation-induced change of perception is termed an `aftereffect'. For example, if you view a facewith a happy expression for some 60 s and then briefly view a neutral face, the neutralface will appear sad (Benton and Burgess 2008; Benton et al 2007). It is generallyunderstood that the process of adaptation is functional and reflects the recoding ofneural systems to better represent the attribute that that neural system seeks to repre-sent (Clifford 2005). By this view, expression adaptation is a powerful behaviouraltool that targets the neural systems encoding facial expression.

The following experiments look at the effect of photographic negation on expressionadaptation. If our representation of facial expression is primarily built from edge infor-mation, then photographic negation of the adapting stimulus should have little effecton expression aftereffects. On the other hand, if our perception of facial expression isdependent upon some representation that incorporates surface information derived throughshape-from-shading then we should expect a substantial reduction of aftereffect strengthover contrast inversion of the adaptor.

2 Experiment 12.1 Method2.1.1 Participants. Twelve Bristol University students (seven females, five males; mean age20.5 years) participated in the experiment. All had normal or corrected-to-normal vision.

2.1.2 Stimuli. Two images of facial expression (anger and happinessösee figure 2) of asingle actor were chosen (Ekman and Friesen 1976). A finely graded series of 100 grey-scale images, which varied in 99 even steps between the two expressions, was producedwith Gryphon Morph v2.5. Images measured 525 pixels high by 350 pixels across;subjects viewed the images from approximately 1 m (the images consequently subtended

(a) (b) (c)

Figure 1. Barack Obama and the folded-face illusion. A photograph of a face (b) is folded into anM-shape; viewed from below (a) the face appears to smile, viewed from above (c) it appears to frown.

2 C P Benton

Figure 2. Examples of the images used. The first two rows contain images from the angry-to-happy morph sequence from image 12 (top left) to image 89 (bottom right) in steps of 11 images.The third row shows the adaptation images used in experiment 1 (images 1 and 100 from themorph sequence). Bottom row shows the adaptation images used in experiment 2.

Effect of photographic negation on face expression aftereffects 3

approximately 14.0 by 9.5 deg of visual angle). Images were presented centred on alinearised (mean luminance 58 cd mÿ2) Sony CPD-G500 CRT monitor driven bya Cambridge Research Systems VSG 2/5 graphics system. Photographic negation wasachieved by contrast inversion around mean luminance.

2.1.3 Procedure. An adaptive-method-of-constants psychophysical task was used to mea-sure `balance points' (Watt and Andrews 1981). Balance point is the point on the morphsequence at which a participant is equally likely to classify an image as either angryor happy. The psychophysical task works by presenting a series of test images (a run)from the morph sequence; participants classify each image presentation as either angry orhappy. During a run, an image can be presented a number of times, the image presentedbeing dependent upon the response history of the participant during the task. A total of64 responses were gathered from each run (resulting in a single balance point per run).Balance points were gathered under adaptation by incorporating an adaptation top-upprocedure into the psychophysical task (Benton et al 2006). Prior to each test image, anadaptation image was presented. Test images were presented for 1 s, the initial adaptationimage within a run was presented for 300 s, thereafter the adaptation image was pre-sented for 8 s (adaptation top-up). There was a 500 ms interstimulus interval betweenadaptor and test during which the screen was set to mean luminance. For each participantwe gathered four balance points, one for each adaptation condition (anger in photo-graphic positive, happiness in positive, anger in negative, and happiness in negative).

2.2 ResultsData from each run were coded by the proportion of times that each participant classifiedeach image as showing a happy expression. Balance points were calculated by fitting acumulative Gaussian to those data (Wichmann and Hill 2001), the balance point being themean of the underlying Gaussian. In an experiment such as described here, the effect ofadaptation is to move the balance point towards the adaptor. Aftereffect magnitude wascalculated, for each polarity, as the balance point in the adapt-to-happy condition minusthat found in the adapt-to-sad condition (Benton et al 2007). Aftereffect magnitudesare shown in figure 3. Dependent t-tests looking at differences in balance points showsignificant aftereffects for both the positive (t11 � 24:10, p 5 0:001, d � 6:51) and nega-tive adaptors (t11 � 4:24, p 5 0:01, d � 1:04). More importantly, a dependent t-testcomparing aftereffect magnitudes shows a significant reduction of some 80% in after-effect strength with photographic negation of the adaptor (t11 � 12:07, p 5 0:001,d � 5:82). Effect sizes were calculated by using the standard deviations of the measureswithin conditions rather than their paired values (Dunlap et al 1996).

50

40

30

20

10

0

ÿ10

Aftereffect

size

Experiment 1

Experiment 2

positive negativeAdaptor type

Figure 3. Results from the two experiments.Aftereffect size is given in number of images(see text for details). Error bars indicate 95%confidence limits.

4 C P Benton

The results show a clear and strong reduction in aftereffect strength with photographicnegation; a finding that argues against the notion that our perception of facial expressionis primarily dependent upon edge information. However, a recent study looking atexpression adaptation has shown a marked reduction in aftereffect strength whenadaptor and test are drawn from different identities (Fox and Barton 2007). Theauthors suggest this to show two separate populations of neurons that encode facialexpression, one that is sensitive to identity and one that is not. It is possible that thepolarity-dependent reduction in aftereffect described here is based on that portionof the neural population that encodes both expression and identity. The reduction ofexpression aftereffect might be secondary, being primarily dependent itself upon theeffect of negation on facial identity. If so, a repeat of the current experiment whichuses different identities for adaptor and test might find very different results.

3 Experiment 23.1 Method3.1.1 Participants. Twelve Bristol University students (six males, six females; mean age20.5 years) participated in the experiment. All had normal or corrected-to-normalvision and had not taken part in the first experiment.

3.1.2 Stimuli and procedure. Apart from adaptor identity, the test stimuli, their manner ofpresentation and the design of the experiment were unchanged. For choice of adaptoridentity, a face with a very different appearance was deliberately chosen (Ekman andFriesen 1976).

3.2 ResultsThe method of analysis was identical to that in experiment 1; results are shown infigure 3. There was a significant aftereffect for the positive adaptors (t11 � 5:89,p 5 0:001, d � 1:59) but not for those in photographic negative (t11 � 2:15, p � 0:055,d � 0:55). Crucially, there was a significant reduction of approximately 75% in after-effect strength with photographic negation (t11 � 4:77, p 5 0:001, d � 1:51).

The results confirm the findings of Fox and Barton (2007) that the change ofadaptor identity (for the positive polarity adaptors) leads to a decrease in aftereffectstrength (t22 � 6:08, p 5 0:001, d � 2:59). It is possible that the reduction of after-effect strength is based upon a reduction in retinotopic adaptation such as, for example,changes in perceived curvature due to local tilt aftereffects (Xu et al 2008); withthe change of identity there would be little local feature correspondence and thereforelittle retinotopic adaptation. However, this explanation of the identity-dependent reduc-tion seems unlikely. When studying faces, people generally make regular exploratoryeye movements (Groner et al 1984; Walker-Smith et al 1977; Yarbus 1967); these wouldgreatly lessen the effects of retinotopic adaptation. In support of the latter, a previousstudy of expression adaptation found no evidence of a difference in aftereffect strengthbetween free-viewing and retinotopic-adaptation-controlled conditions (Benton et al2007). Furthermore there are substantial differences in contour placement and localcontour shapes between the images of the adaptor and test identities used in the currentexperiment (see figure 2).

4 DiscussionThe above experiments show, even with different identities for adaptor and test, asubstantial decrease in adaptation strength over photographic negation. It seems thatthe disruption of surface properties caused by photographic negation disrupts ouridentity-independent representation of facial expression. These results therefore supportthe idea that our neural representation of expression, particularly that which is inde-pendent of identity, incorporates surface information.

Effect of photographic negation on face expression aftereffects 5

A substantial component of the sensitivity to contrast inversion is likely to be basedupon the disruption to shape-from-shading information. Facial expression or, morebroadly, non-verbal facial communication involves a meaningful 3-D deformation ofa 3-D object. A more accurate representation of the 3-D structure of the face willresult in a more accurate understanding of the deformation and its meaning (whetherintentional or not). Whilst changes in pigmentation may be associated with changesin facial expression (Montoya et al 2005), it is clear that we can recognise facialexpression over a range of pigmentation changes both within and between individ-uals. Pigmentation changes may well constitute an independent source of informationabout emotional state; however, they do not affect the structure of the deformationdefining the facial expression.

The results of the present study show that our encoding of facial expression isstrongly affected by photographic negation. White (2001), however, showed that compari-sons of facial expression were not affected by photographic negation. The difference infindings might reasonably be explained by the difference in task. When making a same/different judgment between distinct expressions, the judgment is based upon informationdrawn from the representations of those facial expressions. If edge information is suffi-cient for that distinction, then the judgment may be unaffected by surface-dependentchanges in the underlying representation. In contrast, the adaptation-based task of thepresent study was used in order to look at the information going into the representa-tion, rather than that drawn from it.

Although the comparison was not explicitly made, a similar dependence of after-effect magnitude on surface information can be seen in another adaptation study(Butler et al 2008); a reduction of expression aftereffect was found when faceswere stripped of surface information (compare their graphs 3 and 4). This observa-tion reinforces the view that surface information plays an important role in theneural encoding of facial expressions. In relation to this, the previous paragraphmade a distinction between the information going to form a representation andthat which is drawn from it. This in turn raises the obvious question of why moreshould be encoded than is necessary for recognition. The answer, of course, liesin the question; the recognition of gross facial expressions is not the sole purposeof the system.

What we demand from our expression-recognition system on a day-to-day basisgoes far beyond the categorisation of the major expressions. It seems entirely reason-able to propose that finer distinctions between expressions (both static and, perhapsmore importantly, dynamic) need more information than that required for gross recog-nition. The surface information necessary for the recognition of 3-D structure maywell form part of that information. If true, then a dependence upon surface structureshould be found in tasks that tap into the subtlety of our ability to analyse expressions,such as fine judgments of intent based upon dynamic expressions, or in judgments ofsmall differences between complex facial expressions.

What then about those studies that show viewpoint-dependence in our perceptionof facial expression? The changing of perceived expression with different viewpoints ofthe folded face (see figure 1) is clearly unusual, at least in terms of what it means forour perceptual systems. A 3-D object has been rendered onto a 2-D surface whichhas then been subject to a 3-D distortion (the folding). Presumably, this distortion ofthe rendered 3-D image is not something that our visual system is evolved to handle,and one can readily imagine visual analysis proceeding as if the surface were not folded.If this is the case, then the face appears to smile or frown because that is the 3-Dconstruction that would produce that 2-D image, the folding of the image not beingfactored into the percept.

6 C P Benton

The viewpoint-dependence of perceived expression with real faces (Cavanagh et al2004; Kappas et al 1994; Lyons et al 2000) is more difficult to reconcile with a view inwhich substantial 3-D information is incorporated into our representation of facial expres-sions. However, it is likely the case that our structural representation of facial expressionis not unitary and contains both view-dependent and view-independent mechanisms(Benton et al 2007). A realistic account of expression processing might see our perceptionof facial expression as the weighted outputs of such mechanisms. The weighting mightreasonably incorporate certainty such that, as the certainty of information decreases,that information is given proportionally less weightöthose studies that have shownview-dependence of facial expression have done so when the actual base 3-D expressionis ambiguous or uncertain (such as might be found with a neutral expression).

In conclusion, the adaptation-based experiments described in this paper imply thatsurface-based information plays a strong role in our representation of facial expression.This fits with a view where our representation of facial expression incorporates 3-Dinformation derived from shape-from-shading. Indeed, a misunderstanding of 3-D struc-ture, such as that which may be found with the folded face effect (figure 1), could wellresult in a misperception of facial expression.

Acknowledgment. In remembrance of Charlotte Jones who gathered the data presented in thispaper and who sadly died in the Boxing-Day tsunami of 2004.

ReferencesBenton C P, Burgess E C, 2008 `̀ The direction of measured face aftereffects'' Journal of Vision

8(15) 1 ^ 6Benton C P, Etchells P J, Porter G, Clark A P, Penton-Voak I S, Nikolov S G, 2007 `̀ Turning the

other cheek: the viewpoint dependence of facial expression after-effects''Proceedings of the RoyalSociety, Series B 274 2131 ^ 2137

Benton C P, Jennings S J, Chatting D J, 2006 ``Viewpoint dependence in adaptation to facialidentity'' Vision Research 46 3313 ^ 3325

Bruce V, Langton S, 1994 `̀ The use of pigmentation and shading information in recognising thesex and identities of faces'' Perception 23 803 ^ 822

Butler A, Oruc I, Fox C J, Barton J J S, 2008 `̀ Factors contributing to the adaptation aftereffectsof facial expression'' Brain Research 1191 116 ^ 126

Cavanagh P, Gru« nau M von, Zimmerman L, 2004 ``View dependence of 3D recovery fromfolded pictures and warped 3D faces'', in Proceedings of the 2nd International Symposium on3D Data Processing, Visualization, and Transmission pp 35 ^ 41

Clifford C W G, 2005 `̀ Functional ideas about adaptation applied to spatial and motion vision'', inFitting the Mind to theWorld: Adaptation and After-effects in High-level Vision Eds CWGClifford,G Rhodes (Oxford: Oxford University Press) pp 47 ^ 82

DunlapW P, Cortina J M,Vaslow J B, Burke M J, 1996 `̀ Meta-analysis of experiments with matchedgroups or repeated measures designs'' Psychological Methods 1 170 ^ 177

Ekman P, Friesen W V, 1976 Pictures of Facial Affect (Palo Alto, CA: Consulting PsychologistPress)

Fox C J, Barton J J S, 2007 `̀ What is adapted in face adaptation? The neural representations ofexpression in the human visual system'' Brain Research 1127 80 ^ 89

Galper R E, 1970 `̀ Recognition of faces in photographic negative'' Psychonomic Science 19 207 ^ 208Groner R,Walder F, Groner M, 1984 `̀ Looking at faces: local and global aspects of scanpaths'', in

Theoretical and Applied Aspects of Eye Movement Research Eds A Gale, F Johnson (Amsterdam:Elsevier) pp 523 ^ 533

Hayes T, Morrone M C, Burr D C, 1986 `̀ Recognition of positive and negative bandpass-filteredimages'' Perception 15 595 ^ 602

Johnston A, Hill H, Carman N, 1992 `̀ Recognising faces: Effects of lighting direction, inversion,and brightness reversal'' Perception 21 365 ^ 375

Kappas A, Hess U, Barr C L, Kleck R E, 1994 `̀Angle of regard: the effect of vertical viewingangle on the perception of facial expressions'' Journal of Nonverbal Behavior 18 263 ^ 280

Kemp R, Pike G, White P, Musselman A, 1996 `̀ Perception and recognition of normal andnegative faces: the role of shape from shading and pigmentation cues'' Perception 25 37 ^ 52

Effect of photographic negation on face expression aftereffects 7

Lyons M J, Campbell R, Plante A, Coleman M, Kamachi M, Akamatsu S, 2000 `̀ The Noh maskeffect: vertical viewpoint dependence of facial expression perception'' Proceedings of the RoyalSociety of London, Series B 267 2239 ^ 2245

Montoya P, Campos J J, Schandry R, 2005 `̀ See red? Turn pale? Unveiling emotions throughcardiovascular and hemodynamic changes'' Spanish Journal of Psychology 8 79 ^ 85

Rhodes G, Robbins R, Jacquet E, McKone E, Jeffery L, Clifford C WG, 2005 `̀Adaptation and faceperception: how aftereffects implicate norm-based coding of faces'', in Fitting the Mind tothe World: Adaptation and After-effects in High-level Vision Eds C W G Clifford, G Rhodes(Oxford: Oxford University Press) pp 213 ^ 240

Russell R, Sinha P, Biederman I, Nederhouser M, 2006 `̀ Is pigmentation important for facerecognition? Evidence from contrast negation'' Perception 35 749 ^ 759

Santos I M, Young AW, 2008 `̀ Effects of inversion and negation on social inferences from faces''Perception 37 1061 ^ 1078

Vuong Q C, Peissig J J, Harrison M C, Tarr M J, 2005 `̀ The role of surface pigmentation forrecognition revealed by contrast reversal in faces and Greebles'' Vision Research 45 1213 ^ 1223

Walker-Smith G J, Gale A G, Findlay J M, 1977 `̀ Eye-movement strategies involved in faceperception'' Perception 6 313 ^ 326

Watt R J, Andrews D P, 1981 `̀APE: Adaptive Probit Estimation of psychometric functions'' CurrentPsychological Review 1 205 ^ 214

Webster M A, Kaping D, Mizokami Y, Dumahel P, 2004 `̀Adaptation to natural face categories''Nature 428 558 ^ 561

White M, 2001 `̀ Effect of photographic negation on matching the expressions and identities offaces'' Perception 30 969 ^ 981

Wichmann F A, Hill N J, 2001 `̀ The psychometric function: I. Fitting, sampling, and goodnessof fit'' Perception & Psychophysics 63 1293 ^ 1313

Xu H, Dayan P, Lipkin R M, Qian N, 2008 `̀Adaptation across the cortical hierarchy: Low-levelcurve adaptation affects high-level facial-expression judgments'' Journal of Neuroscience 283374 ^ 3383

Yarbus A L, 1967 Eye Movements and Vision (New York: Plenum Press)

ß 2009 a Pion publication

8 C P Benton

ISSN 0301-0066 (print)

This article is an advance online publication. It will not change in content under normal circumstancesbut will be given full volume, issue, and page numbers in the final PDF version, which will be madeavailable shortly before production of the printed version.

Conditions of use. This article may be downloaded from the Perception website for personal researchby members of subscribing organisations. Authors are entitled to distribute their own article (in printedform or by e-mail) to up to 50 people. This PDF may not be placed on any website (or other onlinedistribution system) without permission of the publisher.

www.perceptionweb.com

ISSN 1468-4233 (electronic)