Exploring Particle-Based Caricature Generations

Somnuk Phon-Amnuaisuk

Perceptions and Simulation of Intelligent Behaviours,Universiti Tunku Abdul Rahman, Petaling Jaya, Malaysia

somnuk@utar.edu.my

Abstract. Computer generated caricatures are commonly created us-ing either line drawing or image warping technique. Two main paradigmsemployed to automate the caricature generation process are variance ex-aggeration which exaggerates facial components that deviate from norms;and example-based generation which exaggerates facial components ac-cording to provided templates. In this paper, we explore a novel applica-tion of an interactive particle-based technique for generating a caricatureof a given face. This does not require prior examples but relies on users’feedback to explore the caricature face space. In this approach, facialfeature points are represented as particles and their movements are usedto incrementally warp a given face until the desired exaggerations areachieved. We have shown that the proposed approach could successfullyprovide an interactive means for generating good quality facial carica-tures.

Key words: Automated caricature generation, Particle-based system

1 Background

The term particle refers to a small discrete object. It has been used to describedifferent computing paradigms, for example, the Particle Swarm Optimisation(PSO) system and a physics model of particles in computer graphics. PSO isattributed to Kennedy and Eberhart [5], and has been successfully employedto solve optimisation problem. In this paradigm, each particle in a swarm isencoded as a plausible solution and each particle moves over the search spacein search for a better solution. The basic mechanisms of PSO are based on theexploitation of the swarm’s group knowledge and of each individual particle’spersonal knowledge. It is a heuristic search which does not rely on the guidancegradient of the search landscape.

In this paper, we explore the warping technique to automate the generationof a caricature from a given celebrity face. In brief, the given face is marked withcoordinates corresponding to predefined points (face model). The face modelcomprises of 84 points which represent 84 particles, these particles move in a2D space and the particles’ positions are used to warp the image. However, ateach time step, there could be many possible next states from the current state(i.e., the particles could move to many different combinations). Here, the system

explores the caricature face space in a depth first search fashion, by generatingsix possible next states (the branching factor is arbitrarily decided to be six here).The user interactively identifies one or more faces that she likes and the systemgenerates a new six possible next states from the chosen image (or the averageof the chosen images). The user terminates the generation process when thedesired caricature is produced. Details of this implementation will be discussedin section 3.

This paper is organised into the following sections. Section 2 gives an overviewof related works. Section 3 discusses our proposed concept and gives the details ofthe techniques behind it. Section 4 provides the output of the proposed apprach.Finally, the conclusion and further research are presented in section 5.

2 Literature Review

In principle, caricatures express concepts rather than physical similarities. Thefigure of Venus of Willendorf (see Figure 1, left)1 could be argued as a carica-ture of a woman with an exaggeration to her fertility. Similarly, the concept offemininity is exaggerated in the sculpture of the Hindu goddess named Parvathi(see Figure 1, middle). The figure exaggerates her femininity through the pos-ture and body characteristics such as a narrow waist, full breasts, and a widehip. This kind of conceptual exaggeration is universal in most cultures, if not all.It is suggested by Ramachandran [13] that this is the peak shift effect in whichthe brain favours an exaggerated concept more than a non-exaggerated version.

The art of facial caricature is a much-neglected gem despite the fact thatexpressing a concept in an exaggerated manner is common in all communica-tion modes. Caricaturists do not really receive recognition in the art scene. Yourcolleague may raise an eyebrow if you want to touch this area. Nevertheless,the science of caricature has received some attention from the researchers inthe facial recognition community in recent years. From the literature, an articleabout Rules for Drawing Caricatures was published in 1791 by Francis Grose(see discussion in [2]). It is generally accepted that caricatures are exaggerationsof distinct features. There are a few accepted scientific hypotheses that explainwhy a distorted image could still be recognised. Perkins suggests that a cari-cature is recognised because we recognise selected attributes that are commonbetween the caricature and the real person. The other common explanation ofcaricatures is that a successful caricature exaggerates features that are differentfrom the norm features. Ramachandran explains that this results from the peakshift effect hypothesis [13]. This school of thought is also backed up by Leopoldet al.’s findings that the face-recognizing neurons seem to strongly respond tothe caricature version of the same faces [7]. At present, we still do not fullyunderstand this issue. An understanding of the cognitive process behind facialrecognition is important and could be beneficial to many applications, e.g., artand entertainment, face recognition, and facial animation. It is interesting to

1 All images are retrieved from http://en.wikpedia.org

note that the area of facial animation is closely related to caricature generationin terms of representation and processing techniques.

Fig. 1. Exaggerating concepts, (left) Venus of Willendorf: the concept of fertility isexaggerated using oversized breasts and abdomen; (middle) Parvati: her femininityis exaggerated through her posture and properties such as a narrow waist and fullbreasts; (right) Ancient Pompeiian graffiti caricature of a politician: take note of theexaggerated nose and chin.

Brennan [2] was one of the pioneers who applied the computer to gener-ate caricatures. Her methodology was based on interactive amplification of thefeatures that were different from the averaged norms. The caricature generatorprogram altered the line drawing of a given face by exaggerating the metricdifferences between the subject and the prototype norm. In her experiments,the averaged norms were taken from a group of male population from Aspen,Colorado, USA., as well as some accepted anatomical ideals after Leonardo daVinci. It is interesting to note that she also reported that good caricature resultsfrequently came from any two pictures which just seem different e.g., ElizabethTaylor and John F. Kennedy in her experiment [2].

Caricature generation method by Brennan could be described as semi-automated.Users’ control played a crucial role in controlling the caricature generation pro-cess. In order to automate the process, the system must possess knowledge aboutthe common features on a face and how to exaggerate those features. The appli-cation of artificial intelligence (AI) techniques in line drawing was investigatedin Librande’s example-based character drawing [10], where examples of draw-ings were analysed and corresponding interpolation functions were learned. Thegeneration of line drawing caricatures was also investigated in PICASSO [6] andimplemented in a robot that would draw the face of a customer into a shrimprice cracker [15]. Another work that followed the line drawing approach was thefacial sketch from Chen et al. [3]. Later, Liang et al. [8] expanded facial sketchtechniques in [3] to line-drawing caricatures.

A deformation-based approach is another popular approach employed to gen-erate caricatures. Morphing/warping was employed by Akleman [1]. Note thatthis is from a different perspective as compared to Brennan’s line drawing ap-

proach, though the deformation was also based on moving reference points inthe 2D space. Akleman’s approach to caricature generation was semi-automatedwhere user interactions were very important in the generation process.

The process employed in computer generated caricatures above share thesame tactic . In essence, the caricature was generated by exaggerating the at-tributes that deviated from the prototypes. We have seen two main computergenerated caricature genre: (i) line drawing; and (ii) pixels morphing. Our workin this report explores an interactive particle-based approach which could bedescribed as semi-automated pixels morphing. The details of the system areintroduced in the next section.

3 An Interactive Particle-based Caricature Generation

It is our interest here to explore the interactive caricature generation of a givenface. Here, the face is modified by the particle movements where each particlerepresents a reference point in a facial feature model (will be discussed later inthis section). Our approach interactively and incrementally modifies the imageusing the information from the particles’ movements. In each step, a user selectsone or more caricatures produced by the system. If more than one images are se-lected, the average of all the images will be used. The position and velocity of theparticles would be employed in the next iteration. This can be seen as a depth-first search strategy with a modified tree expansion tactic since the informationfrom different child nodes are allowed to be combined before branching onto thenext level. Next, the two important theoretical backgrounds in this study arediscussed: (i) the representation of a facial feature model using particles, and (ii)the warping of an image when the particles move.

3.1 Representing a face model using particles

MPEG4 has a standardised facial feature points with the aim to support facialanimation with a low bit rate. MPEG4 defines 84 facial feature points in its facialmodel as well as face animation parameters. Instead of sending the whole anima-tion frame by frame, only control information to instruct facial model would besent over the network. Although MPEG4 has suggested facial feature points in itsspecification, apparently, it is not adopted in most caricature generation works.Researchers usually devise their own face model where variations in referencefeature points in their models are common. This is understandable. Althoughcaricature generation and facial animation share many common grounds, theyare not exactly the same. In this work, we also propose our own facial model.The design of the face model is to support facial deformation. This involves thedeformation of components such as mouth, nose, eyes, eyebrows, etc. Figure 2displays the facial feature points in our face model. The deformation of a facecould be obtained by warping the image from the current facial points to newfacial points.

Fig. 2. A conceptual illustration showing that particles movements alter each triangu-lar mesh and this results in a warped image. From left to right: (i) Mona Lisa’s face, reddots (gray in BW) show corresponding particle positions on her face; (ii) Triangularmesh is formed using the particles’ positions as vertices; and (iii) Her facial featuresare interactively warped as particles traverse the facial space. In this example, she hasa bigger smile.

In this paper, the concept of particles is applied to our problem domain. Thecoordinates in a 2D space are represented as particles. Let a vector of particlesf = (x11, ..., xij) represents a facial model where feature points xij ∈ R denotesthe position of particle i in a j dimension. Particles at time point t, f(t) couldbe transformed across time to t + 1 as a new vector f(t + 1) as follow:

xij(t + 1) = xij(t) + vij(t + 1)

where vij is the velocity of particle i in dimension j.

vij(t + 1) = wvij(t) + c1r1j(pij(t) − xij(t)) + c2r2j(gij(t) − xij(t))

where w is the inertia weight that seeds particular movement units (e.g., eyes,nose, mouth, etc). The vij is the velocity of a particle i in a j dimension. Thec1r1j and c2r2j are weight parameters that combine the influence of the localbest position pij , global best position gij and the current velocity to determinethe velocities of the particles in the next step.

3.2 Generating Caricature using Particles’ Positions:

Allowing particles to move freely in a 2D space would generate all kinds of fa-cial deformations. Unfortunately, many of the deformations obtained in this waywould be useless. It is likely that some kind of control mechanisms must beincorporated to control particle movements. In our implementation, 48 primi-tive movements are created as seed-movements for face shape, eyes, eye brown,nose, and lips. These seed movements are either expanding/shrinking particles’movement from a reference point, or vertically/horizontally push/pull particlesin certain degrees and directions. As it is our interest to explore interactive

Fig. 3. Examples of the applications of seed-movements to face shape, mouth and eyes(column two, three and four respectively). The original image marked with particles’positions is shown on the left top row.

feedback from users, these seeds provide search-guides. During the caricaturegeneration process, movements of the particles derived from different primitiveseed-movements could be combined and this helps generate diversity while curb-ing exploration at the same time. Figure 3 provides some examples of the seedingand Table 1 shows the flow of the caricature generation process.

Warping Techniques In this paper, the warping technique is used to deformfacial components to generate the caricature of a given image. Firstly, a trian-gular mesh is formed using the coordinates from the facial feature points. Forany triangle in the mesh, three vertices u1,u2,u3 of an original triangle couldbe warped to a new triangle formed by the vertices w1,w2,w3 such that

wi = Mui + b

where M is a 2×2 matrix and b is a vector. It is known that there is a uniquelyaffine transformation that maps u1,u2,u3 to w1,w2,w3 respectively and anypoint u in the triangle formed by u1,u2,u3 would map to a point w in thetriangle formed by w1,w2,w3 if u = c1u1+c2u2+c33, w = c1w1+c2w2+c3w3

and c1 + c2 + c3 = 1.

4 Results and Discussion

This subsection provides an overview of the caricature generation process in ourapproach. We work with the famous Mona Lisa face in this example. The aimof this exercise is to give readers a walk through a typical generation process inour approach.

Table 1. Caricature Generation

Outline of important conceptual steps

// create a particle vector f for an input imagef ← initParticle(image);0 ← t;while not Termination Criterion(f(t)) do

// determine particles velocity (see sub-section 3.1)

vpso ← interactiveSession(f(t));// generate six variations of a given face based on PSO and primitive

// seed movements

f(t)[6] ← applyPSO(f(t),vseed);// a user guides the search by choosing one or more preferred images

f(t) ← interactiveSession(f(t + 1)[6]);t ← t + 1;

endwhile

1. The first step is to prepare an input image. All images used are downloadedfrom public domains via Google. The position of particles are manuallymarked to each face.

2. For each iteration the particles move to new points as described in subsection3.1. In this implementation, the branching factor is set at six.

3. Users then select one or more generated caricatures and further explore thesearch space in that direction. If more than one caricatures are selected, thenthe positions and the velocities of the particles would be averaged and thesearch would continue from the averaged output.

4. Although the branching factor is fixed at six, users could repeatedly generatevarious output before making their choices.

5. The process continues until the desired caricature effects are achieved.

Figure 4 shows examples of various smiles generated by the system. Figure 5shows the caricature of sixteen celebrities. As the evaluation function is subjec-tive, we provide an insight on how the search space is guided in generating theseoutput. Our strategy used in creating these caricatures could be summarised inthe following simple heuristics: (i) examine the face and think about the impres-sion you have of the face; (ii) exaggerate that part; (iii) modify the neighboringcomponents to agree with the exaggeration you intend to achieve, for example,a bigger smile may go well with slightly bigger nose and slightly smaller eyes.Explore those face space. Repeat steps i, ii, iii until the desired distortion isobtained.

The issue of computational aesthetics in evolutionary art has been addressedby many researchers before [11], [4], [9]. It is a challenge to find features thatcould capture the aesthetic dimension. Choices of these features, at a fine gran-ularity level, such as sound frequency spectra or histograms of color could throwaway many semantics (i.e., semantics could emerge from a hierarchical com-position of lower level units). At a coarse granularity such as universal I-V-I

Fig. 4. Examples of plausible smiles produced from warping different degrees of particlemovements

progressions in music, or fractal parameters, it could be too high-level and tooabstract. For faces, it has been suggested that pleasing faces are those whichare symmetrical. Although many males are also attracted to pouty lips andsometimes non-symmetrical female smiles, many pleasing visual experiences docoincide with symmetry. Unfortunately, caricatures are also about distortion. Itis difficult to propose an objective fitness functions for the task.

5 Conclusion and Future Work

We have shown that an interactive particle-based approach could be used to gen-erate convincing caricatures. The particles movements in a 2D space are guidedby interactive fitness evaluation to create a caricature of a given celebrity face.The particle-based approach supports both the mutation and breeding concepts.Mutation can be seen as particles’ movements in the face space and breeding canbe seen as the averaged velocity update of many leaf nodes during the searchtree expansion step. This guarantees that the particles have a means to explorethe face space and useful information could be shared among particles (i.e.,good solutions breed). Future work could be extended in the following areas: (i)investigation of the automated aesthetic evaluation criteria, (ii) devising moreprimitive seed movements to explore various areas of the face space, (iii) breed-ing inherited facial features from parents, and (iv) animating facial emotionalexpressions. There are many interesting applications in the emerging creativeindustries that could benefit from this line of research.

Fig. 5. Caricature examples produced by our approach. All celebrity faces are retrievedfrom public domain from the Internet. No special treatment other than removing thebackground and some smudging on the shirts are applied to the input images.

References

1. Akleman, E.: Making Caricatures with Morphing In Proceedings of the InternationalConference on Computer Graphics and Interactive Techniques ACM SIGGRAPH

97 Visual Proceedings: The art and interdisciplinary programs of SIGGRAPH ’97,ISBN:0-89791-921-1, pp 145. Los Angeles, California, United States. 1997

2. Brennan, S.E.: Caricature Generator: The Dynamic Exaggeration of Faces by Com-puter. Leonardo 18 (3), pp 170-178, MIT Press, 1985.

3. Chen, H., and Xu, Y.Q., and Shum, H.Y., and Zhu, S.C., and Zheng, N.N.: Example-based Facial Sketch Generation with Non-parametric Sampling. In Proceedings ofthe Eighth International Conference on Computer Vision (ICCV01), pp 433-438,Vancouver, Canada, July 7-14, 2001.

4. den Heijer, E., and Eiben, A.E.: Comparing Aesthetic Measures for EvolutionaryArt. In Proceedings of the 9th European Event on Evolutionary and Biologically In-spired Music, Sound, Art and Design (EvoMusart), EvoApplication 2010, Istanbul,Turkey,April 7-9, 2010.

5. Eberhart, R. C. and Kennedy, J., A New Optimizer using Particle Swarm Theory.In Proceedings of the Sixth International Symposium on Micromachine and HumanScience, Nagoya, Japan. pp. 39-43, 1995.

6. Koshimizu, H., and Tominaga, M., and Fujiwara, T.,and Murakami, K.: On KANSEIFacial Image Processing for Computerized Facial Caricaturing System PICASSO. InProceedings of 1999 IEEE International Conference on Systems, Man, and Cyber-netics pp. 294-299, Vol. 6 ISBN: 0-7803-5731-0 Tokyo, Japan. 1999

7. Leopold, D., and Bondar, I., and Giese, M.: Norm-based Face Encoding by SingleNeurons in the Monkey Inferotemporal Cortex. Nature, 442 (7102):572-575, 2006

8. Liang, L., and Chen, H., and Xu, Y.Q. and Shum, H.Y.: Example-based Carica-ture Generation with Exaggeration. In Proceedings of the 10th Pacific Conference onComputer Graphics and Applications pp 386-389, 2002.

9. Li, Y., and Hu, C.J.: Aesthetic Learning in an Interactive Evolutionary Art Sys-tem. In Proceedings of the 9th European Event on Evolutionary and Biologically In-spired Music, Sound, Art and Design (EvoMusart), EvoApplication 2010, Istanbul,Turkey,April 7-9, 2010.

10. Librande, S.E.: Example-based Character Drawing, MSc in Visual Studies Thesis,Massachusetts Institute of Technology. U.S.A.

11. Machado, P. and Romero, J. and Manaris, B.: Experiments in ComputationalAesthetics: An Iterative Approach to Stylistic Change in Evolutionary Art, The Artof Artificial Evolution, Natural Computing Series, Part V, pp 381-415, 2008.

12. Mauro, R. and Kubovy, M.: Caricature and Face Recognition, Memory and Cog-nition, 20, pp 433-440, 1992.

13. Ramachandran, V.S.: A Brief Tour of Human Consciousness. PI Press, New York.2004.

14. Staake, B.: The Complete Book of Caricature North Light Books, Cincinnati, Ohio,1991.

15. Tokuda, N., and Hoshino, T., and Watanabe, T., and Funahashi, T., and Fujiwara,T., and Koshimizu, H.: Facial Caricaturing Robot COOPER Exhibited at EXPO2005and Its Improvements. In Proceedings of the 2007 International Conference on Ma-chine Vision Applications (MVA2007 IAPR). pp 512-515, May 16-18, 2007. Tokyo,Japan.

16. Tversky, B., and Baratz, D.: Memory for Faces: Are Caricatures Better than Pho-tographs, Memory & Cognition. 13 (1): 45-49, 1985.

17. Yang, T.T., and Lai, S.H.: A Learning-based System for Generating Exaggera-tive Caricature from Face Images with Expression. In Proceedings of the 2010 IEEEInternational Conference on Acoustics Speech and Signal Processing (ICASSP). pp2138-2141 March 14-19, 2010. Dallas, Texas ISSN: 1520-6149 ISBN: 978-1-4244-4295-9