70 Joni Brenner and Arlene Archer

dependent porous uncertain incomplete and unresolved The argument is made through the use of multiple modes including the visual written and spatial Configurations of these modes construct difference variously through juxtaposition comparison and ambiguity in order to unsettle and question certain assumptions and perceived dominant ideas Kresss notion that argument foregrounds difference produces ambiguity and so opens the space for reconsideration for a shift in values and attitudes and for an extension of thought and investigation seems well aligned with our concepshytion of how artworks and exhibitions construct argument

REFERENCES

Andrew D 2011 The artists sensibility and multimodality-Classrooms as works of art Unpublished doctoral thesis Johannesburg University of the Witwatersrand

Andrews R 2010 Argumentation in higher education Improving practice through theory and research New York Routledge

Brenner] amp Burroughs E 2011 Conclusion In BrennerJ Burroughs E amp NeI K (eds) Life of bone Art meets science Johannesburg Wits University Press 155-160

Brenner J Burroughs E amp Glencross D 2011 Obsessions and impulses Art meets science In Brenner J Burroughs E amp Nel K (eds) Life of bone Art meets science Johannesburg Wits University Press 9-34

Brenner J Burroughs E amp Nel K (eds) 2011 Life of bone Art meets science Johannesburg Wits University Press

Burroughs E 2011 Of words and skulls Joni Brenner In Brenner J Burroughs E amp Nel K (eds) Life of bone Art meets science Johannesburg Wits University Press 87-103

Bunn D 2002 Dark Designs Joni Brenner infra-red (Exhibition catalogue) Johannesburg Gertrude Posel Gallery Wits University

Davis W 2009 The wayfinders Why ancient wisdom matters in the modern world Toronto House of Anansi Press 210

Dewey J 1929 The quest for certainty A study of the relation of knowledge and action New York Minton Balch amp Company 310

Fairbrother T 1989 Skulls In Garrels G (ed) The Work of Andy Warhol Seattle WA Dia Art Foundation and Bay Press 93-115

Jung C amp Wilhelm R 1931 The secret of the golden lower London Kegan Paul Trench Trubner 78

Kemp M 2010 A second renaissance New Scientist May 844 Kress G 1989 Texture and meaning In Andrews R (ed) Narrative and argument

Milton Keynes Open University Press 9-21 Nel K 2011 Matter out of place Four new pieces In Brenner J Burroughs E amp

Nel K (eds) Life of bone Art meets science Johannesburg Wits University Press 119-134

Ratele K 2011 Part of the story In Brenner J Burroughs E amp Nel K (eds) Life of bone Art meets science Johannesburg Wits University Press 67-86

5 Teaching Visual Narratives U sing a Social Semiotic Framework The Case of Manga

Cheng- Wen Huang

The dominance of the visual in our communication landscape has resulted in recognition of the importance of the visual in literacy curricula A number of theorists have proposed frameworks to analyse and discuss images Kress and Van Leeuwen (1996) for example have proposed a comprehensive frameshywork for analysing still single-framed images Thibault (2000) and OHalloshyran (2004) have posited ways of looking at moving images and Martinec and Salway (2005) and Unsworth (2006) have looked at describing image-writing interactions Although there has been some research into understanding still sequential images (Baldry amp Thibault 2006 Lim 2007 Matthiessen 2007) there remains a need for a comprehensive framework and a metalanguage which can be adapted for classroom use Students tend to struggle with vague generalities rather than insightful analyses (Archer 2010 206) if there are no frameworks to act as guidelines and metalanguages to act as tools of analysis no sets of grammars to describe semiotic resources and how they function to construct meaning This chapter extends Kress and Van Leeuwens (1996) framework for single images by combining it with Matthiessens (2007) work on rhetorical relations in images In doing so it produces a framework and a metalanguage which can be used to analyse and describe visual narratives such as comics for students at high school or tertiary level

Comics are increasingly being drawn into progressive visual literacy media and popular culture curricula as the pedagogical efficacy of using popular cultural texts in schooling becomes more apparent to educators (Allen amp lngulsrud 2003 Alvermann amp Heron 2001 Norton amp Vandershyheyden 2004 Rubinstein-Avila amp Schwartz 2006) This chapter proposes a framework and a metalanguage for analysing a particular genre of comics manga that is growing in popularity among teenagers and university stushydents Manga refers to a genre of comics from Japan which has attained international popularity Brienza (2009) notes that in the United States manga sales have grown at an exceptional growth rate of 350 from $60 million in 2002 to $210 million in 2007 Though it is true that sales began to experience a decline in late 2008 owing to the economic downturn and scanlations (free online translated scans of the original which were believed to attract new readers) manga remains a popular international comic genre

72 Cheng-Wen Huang

Its popularity has led to a trend in manga-style comics in the West such as la nouvelle manga in France and Amerimanga in the United States In the United Kingdom SelfMadeHero a publishing house has taken advantage of the appeal of manga and has produced a series of Shakespeare plays in manga-style-Romeo and Juliet (Leong 2007) Hamlet (Vieceli 2007) and A midsummer nights dream (Brown 2008) to name a few In South Africa most students gain access to these comics by downloading the scanlations offline though manga can be purchased from high-quality bookshops

The aim of this chapter is to propose a Hallidayan framework that offers a way of understanding and describing how semiotic resources in manga communicate ideas and experiences of the world convey interpersonal meanings and how these meanings are communicated to the reader as a coherent whole The chapter draws on examples of generic resources from the popular manga Naruto by Masashi Kishimoto (2003) Naruto is a ninja action fantasy that centres on the story of Uzumaki Naruto an adoshylescent ninja and his quest to become the lead ninja of his village

THEORETICAL FRAMEWORK

The theoretical approach that underlies the proposed framework is multishymodal social semiotics This theoretical approach assumes that texts are comshyposed of a combination of semiotic resources and that these resources are always socially situated produced in a particular cultural social and historishycal context (Jewitt 2009 Kress 2010) Semiotic resources describe

the actions materials and artefacts we use for communicative purposes whether produced physiologically-for example with our vocal appashyratus the muscles we use to make facial expressions and gestures-or technologically-for example with pen and ink or computer hardware and software-together with the ways in which these resources can be organized (Van Leeuwen 2005285)

In other words they are resources for meaning-making In multimodal social semiotics it is assumed that different semiotic resources have different comshymunicative potential Adopting a multimodal social semiotic view of text thus means recognising the communicative functions of semiotic resources and situating their use and form in relation to context

Hallidays (1978) meta functional principle provides the basis for a mulshytimodal social semiotic approach to text analysis This principle is based on the view that language is a semiotic system which realises particular social functions These social functions include communicating ideas and expeshyriences of the world (ideational metafunction) conveying social relations (interpersonal metafunction) and organising the grammar in a coherent manner so that it can be communicated as a text (textual metafunction)

Teaching Visual Narratives Using a Social Semiotic Framework 73

The term metafunction describes that part of the system of a language that has evolved to perform the function in question (Halliday amp Hasan 198544) The ideational interpersonal and textual metafunctions can be found in all languages irrespective of social use and context Kress and Van Leeuwen (1996) extend HallidayS meta functional principle to the study of images and propose that images like language perform three communishycative functions They rename the metafunctions the representational the interactive and the compositional respectively

It is important to note that in Hallidays conception of the ideational metafunction he identifies this as being a combination of two metafuncshytions the experiential and the logical The experiential metafunction is concerned with the features that can be thought of as representing the real world as it is apprehended in our experience whereas the logical metafunction is concerned with the relationship between one process and another or one participant and another that share the same position in the text (Halliday amp Hasan 198519 45) In other words the experiential metafunction is concerned with how experience is represented and the logshyical meta function is concerned witb the relations of causal and temporal interdependency (Baldry amp Thibault 200622) According to Halliday the logical meta function is tbe dosest in meaning to the experiential and there is a lot of give-and-take between the two (200318) Because both metashyfunctions are needed to contribute towards communicating experiences of the world he collectively calls the two meta functions the ideational metashyfunction Although in Kress and Van Leeuwens (1996) conception of the representational meta function they do not make a distinction between tbe experiential and the logical metafunction foregrounding the logical metashyfunction is pertinent when exploring visual narratives because temporality and causality are key properties of narratives

Narrative according to Bordwell and Thompson is a chain of events in cause-effect relationship occurring in time and space (200160) Similarly Baldry and Thibault write that narratives signal a temporal succession of events and show how some aspect of a situation or a participant in a narshyrative changes as a result of the transition from an earlier moment to some later moment (italics in original 200613) Single images can only illustrate one moment in time which means they are limited in terms of expressing causality and temporality Sequences of still images on the other hand are able to illustrate linked moments in time and are therefore capable of articshyulating causality and temporality As Baldry and Thibault write [tJhe very notion of sequence implies a time-based chronological ordering of events in a narrative andor cause-effect structuring (200644) Since sequence is key to visual narratives a framework for analysing manga needs to have the grammar to explain the development of seq uences of images in these texts For this reason the proposed framework foregrounds the logical metashyfunction and draws on Matthiessens (2007) work on rhetorical relations in images to account for the meanings made here

Jo= _bullbullbullbullbull

~-- ~

~

social experiences through props location and sound

74 Cheng-Wen Huang

In sum the framework proposed in this chapter involves four metafuncshytions the experiential the logical the interactive and the compositional The usefulness of this framework for teaching visual narratives lies in the multi- dimensional view it offers for analysis The experiential metafunction highshylights the resources that communicate experiences of the world in a narrative The logical metafunction explicates how a narrative is developed through sequences of images The interactive metafunction explains the resources that realise social interactions In other words it provides an understanding of the power relations and attitudes that are constructed between the image and the viewer The compositional metafunction assists in understanding the composition of the image as a whole While integrating Kress and Van Leeuwen (1996) and Matthiessens (2007) work to produce a framework for analysing manga I replace some technical terms within their frameworks with film vocabularies both because the genre of manga draws on cinematic visual techniques (McCloud 1994 Rommens 2000) and in order to renshyder the metalanguage more accessible to students Thesen writes that rather than propose an entirely new set of vocabularies a practical metalanguage should serve as a way of verbalising what you know in relation to other ways of knowing (2001143) As film vocabularies are not only taught in visual literacy classes in school but can also be found in everyday language it seems appropriate to replace technical terms with a vocabulary borrowed from film Below I outline the framework and metalanguage in detail

EXPERIENTIAL METAFUNCTION WAYS OF PORTRAYING HUMAN EXPERIENCE

Halliday proposes that representations which produce particular meanings are always seen as the expression of some kind of a process (198518) The experiential metafunction is primarily concerned with how the world is categorised according to social experience In visual narratives resources such as props costume location colour and sound (in the form of onoshymatopoeia in written text) all contribute towards creating a representation of some kind of a social experience as can be seen in Figure 51

Figure 51 is a representation of a morning routine Panels in manga are read from right to left top to bottom In the first frame the backdrop of the narrative is established by the represented resources in the background on the right A village setting is evoked by the hut-like structures In parshyticular the kanji (Chinese character) ~ at the bottom centre of the frame points towards an Asian-oriented narrative world In the foreground on the left props such as a bed and a pillow allow the reader to recognise the present location as a bedroom The beam of light shining through the window and the sun and the clouds outside signify that it is morning The boys arms outstretched mouth wide open accompanied with the sound

76 Cheng- Wen Huang

yaaawn suggest that he has just woken up In the second frame the represented resources include a carton of milk and a glass These props are a signifier for breakfast In the third frame the boy can be seen sitting at a table Background props such as a cupboard pots of plants and posters on the wall that reads Do your best on the one and Ninja on the other convey a change in location The words munch munch indicate that the boy is eating breakfast Props such as the milk the glass the plate the sandwich and the sound effect of the onomatopoeias glug glug and munch munch all add to the construction of a breakfast atmosphere In the last row of frames the boy can be seen in a change of clothing standing in front of a mirror This suggests that he is now getting reading to leave the house The headband in the last frame is an important prop in Naruta it is a signifier of the ninja theme The social experience of a morning routine is thus created through the use of props location and sound effects Manga in particular is known to pace the narrative paying attention to small world details in order to re-create everyday social experiences (McCloud 19942006)

As seen in Figure 51 embodied action such as body posture gesture facial expression also contributes to the relaying of a particular social

experience The boys outstretched arms and wide-open mouth in Frame 1 convey that he has just woken up In Frame 3 the boys half-awake state is portrayed through his slumped body posture and half-dosed eyes The emphasis that manga places on embodied action to evoke social experishyences has resulted in the development of highly conventionalised images for conveying emotions which are particular to the visual culture of Japan For example in Figure 510 (right image) manga readers will be able to recshyognise that the character who is enlarged and depicted in an exaggerated super deformed state is a convention for expressing an extreme burst of emotion This is a technique often used for comic relief

To summarise the experiential metafunction is concerned with the repshyresented resources and how they function to depict a particular event into categories of social experience Semiotic resources which are able to realise the experiences include embodied actions (facial expressions body posture gesture) and resources which establish setting (costume props location) The orchestration of these resources is akin to what is known as mise en scene in film

LOGICAL METAFUNCTION WAYS OF MOVING THE VISUAL NARRATIVE FORWARD

It is generally accepted that narratives are characterised by time and cause and effect In visual narratives this is realised through placing frames in sequence and a causal relationship is created between the frame before and the frame after The logical metafunction is crucial when looking at visual

Teaching Visual Narratives Using a Social Semiotic Framework 77

narratives as it is concerned with the relationship between one process and another (Halliday 198545) According to Baldry and Thibault the logical metafunction is realised by recursive structures which add one element to another so as to build up more complex chain-like structures (200622) This means that the logical metafunction is about how elements of a text link together to form a coherent whole It can also be understood as how elements of a text push a narrative forward and thus aligns with Matthiesshysens notion of rhetorical relations According to Matthiessen rhetorical relations are concerned with the development of sequences of passages in a text (200733) He posits two ways in which images may be developed expansion and projection

Expansion refers to the augmentation of an image There are three difshyferent levels in which an image can be augmented elaboration extension and enhancement Elaboration refers to the restatement of an image To elaborate something is to build onto something where the foundation is still the same This can be understood as depicting the same image again but in greater detail or context For example an image may be elaborated through the means of zooming in or out That is a smaller part of an image can be focused on where although the context of the image is minimised the represented element is afforded greater detail Alternatively by zooming out greater context is disclosed but there is less focus on the represented element For example Frame 1 of Figure 52 presents a close-up image of

Frame 1

TWTlOtII( C8IlIgt

GtAfmlshy1116 THe

MOHI1oINSlD lJIIIIGE5

OFIU 1lpound ~OF ~VIUJISE- Frame 2

1OIJl ~

Figure 52 Elaboration by means of zooming in and out (Kishimoto 200310)

78 Cheng-Wen Huang

an old man smoking his pipe The old mans annoyance and unease at being disrupted are conveyed through his facial expression and the drop of sweat rolling down his forehead The sense of shock is also suggested by the four lines above his head The close-up view not only divulges his emotions but also what he is doing writing calligraphy While the close-up in Frame 1 does well in conveying the mans emotions and actiolls it gives the reader

information with regard to the context of the situation-where the man is situated and who the people are calling out to him Frame 2 discloses this information by zooming out of the close-up The wide-shot in Frame 2 provides information with regard to the context of situation but in doing so affords less focus and detail to the represented elements

Elaboration works with the same image elaborating it either by zoomshying in or zooming out Extension on the other hand propels the narrative forward by providing a new image The new image extends the existing image by providing additional information Although the new image is new in the sense that it has not been seen before it is still related to the previous image For instance in film terms extension can be realised through panning (camera moving sideways from a fixed position) tilting (camera moving up or down from a fixed position) or tracking (camera follows an object or participant) Figure 53 is an example of extension The top image establishes the time of the day (night) but it does not proshyvide details with regard to the setting The bottom frame provides this information by lowering the camera eye This is an example of extension through tilting

Enhancement is the augmentation of an image through a change in time and space This change in time and space pushes the narrative forward In language-based narratives this typically happens at the beginning ofchapters or sections within a chapter Since enhancement requires a change either in time or space I propose using the word transition In images the transition from one image to another can occur through techniques such as flashback or flashforward and split frames which evoke the notion of meanwhile Any image in which a temporal or spatial change has occurred in relation to a previous image can be seen as a transition Figure 54 is an example of a narrative being propelled forward through transition Reading from right to left the narrative is initially situated at a traditional Japanese restaurant in the evening The narrator exits this scene with an external birds eye view of the restaurant the perspective of an omniscient narrator In the next frame puffs of cloud indicate that there is a shift in time It is no longer evening but daytime The last frame takes the reader into a classroom Students are sitting slouched at their desks These three frames clearly indicate a shift in time and space This approach to shifting between scenes is typical of manga storytelling

Another way of pushing the narrative forward is through dialogue whether it is internal dialogue (thought) or external dialogue (speech) Matshythiessen (2007) refers to this process as projection Projection in comics is

Teaching Visual Narratives Using a Social Semiotic Framework 79

tilting an image (Kishimoto 200324) Figure 53 Extension

usually captured in frames Figure 55 is an example of projection through voice-over narration The image depicts two people having a discussion In Frame 1 the man sitting on the right can be identified through his costume as a ninja The headband the dark suit and the vest are characteristics of ninja iconography in Naruto The ninja has his hands up posed to ask the boy on the left a question The boy makes a response in Frame 1 and proceeds to answer the question in Frame 2 He elaborates on his answer and in doing so his explanation is carried into Frame 3 In Frame 3 the

80

Figure 54 A shift in time and space through transition (Kishimoto 200318)

Frame 2 Frame 1

Frame 4 Frame 3

Figure 55 Projection through a voiceover (Kishimoto 200317)

boys dialogue is overlapped with an image that is set in a different time and place The image would have been otherwise out of context in this sequence of images but the boys dialogue serves as a continuity element propelling the narrative forward

In sum the logical meta function is concerned with the logical progresshysion of a narrative how the story is projected forward from frame to frame

Teaching Visual Narratives Using a Social Semiotic Framework 81

Drawing from Matthiessen (2007) I have identified two approaches in which this can occur expansion and projection Expansion propels the narshyrative forward through elaboration extension and transition The narrative in these cases is developed through the images Projection on the other hand propels the narrative forward through verbal components that is through speech or thought

INTERACTIVE METAFUNCTION INTERACTIONS WITH THE READER

The interactive meta function is concerned with the interpersonal relationshyship between the image and the reader Specifically it is concerned with the social relations and the evaluative orientations between these two intershyactants Kress and Van Leeuwen (1996) identify three particular categoshyries which express the interactive metafunction contact social distance and attitude These categories are useful when looking as visual narratives because they provide an understanding of how semiotic resources function to elicit certain emotions from readers

Contact is concerned with the presence or absence of gaze from a represhysented participant In film terms this can be regarded as viewer identificashytion which is established through camera positioning Depending on who the director wishes the viewer to identify with the camera can be placed in a number of positions constructing a particular point of view I use the term point of view in this chapter instead of contact when referring to viewer identification because it is commonly used in both film and literature A first person point of view is established when the camera takes the position of the represented participant This position allows the viewer maximum identification with the represented participant Figure 56 (rop image) is an example of a subjective point of view where the depicted images simulate the vision of an eye opening The image begins with a totally black frame which imitates the vision of the closed eye A horizontal central split in the next image represents the eye partially opening This eye widens the entire image is clear in the last frame In this representation there is a strong interpersonal engagement between the reader and the character as the reader adopts the position of the character and views the world at the exact moment as the character

Third person point of view is established when the camera represents the unfolding narrative from a third person in the narrative Viewer identishyfication with the represented participant is less strong from a third person perspective The omniscient point of view presents a narrative unfolding from gods view or a birds eye view This is a point of view that does not belong to anyone in the narrative Viewer identification is absent from this viewpoint For example in Figure 56 the middle and bottom images are of the same location The point of view of the middle image is that of a third

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

72 Cheng-Wen Huang

Its popularity has led to a trend in manga-style comics in the West such as la nouvelle manga in France and Amerimanga in the United States In the United Kingdom SelfMadeHero a publishing house has taken advantage of the appeal of manga and has produced a series of Shakespeare plays in manga-style-Romeo and Juliet (Leong 2007) Hamlet (Vieceli 2007) and A midsummer nights dream (Brown 2008) to name a few In South Africa most students gain access to these comics by downloading the scanlations offline though manga can be purchased from high-quality bookshops

The aim of this chapter is to propose a Hallidayan framework that offers a way of understanding and describing how semiotic resources in manga communicate ideas and experiences of the world convey interpersonal meanings and how these meanings are communicated to the reader as a coherent whole The chapter draws on examples of generic resources from the popular manga Naruto by Masashi Kishimoto (2003) Naruto is a ninja action fantasy that centres on the story of Uzumaki Naruto an adoshylescent ninja and his quest to become the lead ninja of his village

THEORETICAL FRAMEWORK

The theoretical approach that underlies the proposed framework is multishymodal social semiotics This theoretical approach assumes that texts are comshyposed of a combination of semiotic resources and that these resources are always socially situated produced in a particular cultural social and historishycal context (Jewitt 2009 Kress 2010) Semiotic resources describe

the actions materials and artefacts we use for communicative purposes whether produced physiologically-for example with our vocal appashyratus the muscles we use to make facial expressions and gestures-or technologically-for example with pen and ink or computer hardware and software-together with the ways in which these resources can be organized (Van Leeuwen 2005285)

In other words they are resources for meaning-making In multimodal social semiotics it is assumed that different semiotic resources have different comshymunicative potential Adopting a multimodal social semiotic view of text thus means recognising the communicative functions of semiotic resources and situating their use and form in relation to context

Hallidays (1978) meta functional principle provides the basis for a mulshytimodal social semiotic approach to text analysis This principle is based on the view that language is a semiotic system which realises particular social functions These social functions include communicating ideas and expeshyriences of the world (ideational metafunction) conveying social relations (interpersonal metafunction) and organising the grammar in a coherent manner so that it can be communicated as a text (textual metafunction)

Teaching Visual Narratives Using a Social Semiotic Framework 73

The term metafunction describes that part of the system of a language that has evolved to perform the function in question (Halliday amp Hasan 198544) The ideational interpersonal and textual metafunctions can be found in all languages irrespective of social use and context Kress and Van Leeuwen (1996) extend HallidayS meta functional principle to the study of images and propose that images like language perform three communishycative functions They rename the metafunctions the representational the interactive and the compositional respectively

It is important to note that in Hallidays conception of the ideational metafunction he identifies this as being a combination of two metafuncshytions the experiential and the logical The experiential metafunction is concerned with the features that can be thought of as representing the real world as it is apprehended in our experience whereas the logical metafunction is concerned with the relationship between one process and another or one participant and another that share the same position in the text (Halliday amp Hasan 198519 45) In other words the experiential metafunction is concerned with how experience is represented and the logshyical meta function is concerned witb the relations of causal and temporal interdependency (Baldry amp Thibault 200622) According to Halliday the logical meta function is tbe dosest in meaning to the experiential and there is a lot of give-and-take between the two (200318) Because both metashyfunctions are needed to contribute towards communicating experiences of the world he collectively calls the two meta functions the ideational metashyfunction Although in Kress and Van Leeuwens (1996) conception of the representational meta function they do not make a distinction between tbe experiential and the logical metafunction foregrounding the logical metashyfunction is pertinent when exploring visual narratives because temporality and causality are key properties of narratives

Narrative according to Bordwell and Thompson is a chain of events in cause-effect relationship occurring in time and space (200160) Similarly Baldry and Thibault write that narratives signal a temporal succession of events and show how some aspect of a situation or a participant in a narshyrative changes as a result of the transition from an earlier moment to some later moment (italics in original 200613) Single images can only illustrate one moment in time which means they are limited in terms of expressing causality and temporality Sequences of still images on the other hand are able to illustrate linked moments in time and are therefore capable of articshyulating causality and temporality As Baldry and Thibault write [tJhe very notion of sequence implies a time-based chronological ordering of events in a narrative andor cause-effect structuring (200644) Since sequence is key to visual narratives a framework for analysing manga needs to have the grammar to explain the development of seq uences of images in these texts For this reason the proposed framework foregrounds the logical metashyfunction and draws on Matthiessens (2007) work on rhetorical relations in images to account for the meanings made here

Jo= _bullbullbullbullbull

~-- ~

~

social experiences through props location and sound

74 Cheng-Wen Huang

In sum the framework proposed in this chapter involves four metafuncshytions the experiential the logical the interactive and the compositional The usefulness of this framework for teaching visual narratives lies in the multi- dimensional view it offers for analysis The experiential metafunction highshylights the resources that communicate experiences of the world in a narrative The logical metafunction explicates how a narrative is developed through sequences of images The interactive metafunction explains the resources that realise social interactions In other words it provides an understanding of the power relations and attitudes that are constructed between the image and the viewer The compositional metafunction assists in understanding the composition of the image as a whole While integrating Kress and Van Leeuwen (1996) and Matthiessens (2007) work to produce a framework for analysing manga I replace some technical terms within their frameworks with film vocabularies both because the genre of manga draws on cinematic visual techniques (McCloud 1994 Rommens 2000) and in order to renshyder the metalanguage more accessible to students Thesen writes that rather than propose an entirely new set of vocabularies a practical metalanguage should serve as a way of verbalising what you know in relation to other ways of knowing (2001143) As film vocabularies are not only taught in visual literacy classes in school but can also be found in everyday language it seems appropriate to replace technical terms with a vocabulary borrowed from film Below I outline the framework and metalanguage in detail

EXPERIENTIAL METAFUNCTION WAYS OF PORTRAYING HUMAN EXPERIENCE

Halliday proposes that representations which produce particular meanings are always seen as the expression of some kind of a process (198518) The experiential metafunction is primarily concerned with how the world is categorised according to social experience In visual narratives resources such as props costume location colour and sound (in the form of onoshymatopoeia in written text) all contribute towards creating a representation of some kind of a social experience as can be seen in Figure 51

Figure 51 is a representation of a morning routine Panels in manga are read from right to left top to bottom In the first frame the backdrop of the narrative is established by the represented resources in the background on the right A village setting is evoked by the hut-like structures In parshyticular the kanji (Chinese character) ~ at the bottom centre of the frame points towards an Asian-oriented narrative world In the foreground on the left props such as a bed and a pillow allow the reader to recognise the present location as a bedroom The beam of light shining through the window and the sun and the clouds outside signify that it is morning The boys arms outstretched mouth wide open accompanied with the sound

76 Cheng- Wen Huang

yaaawn suggest that he has just woken up In the second frame the represented resources include a carton of milk and a glass These props are a signifier for breakfast In the third frame the boy can be seen sitting at a table Background props such as a cupboard pots of plants and posters on the wall that reads Do your best on the one and Ninja on the other convey a change in location The words munch munch indicate that the boy is eating breakfast Props such as the milk the glass the plate the sandwich and the sound effect of the onomatopoeias glug glug and munch munch all add to the construction of a breakfast atmosphere In the last row of frames the boy can be seen in a change of clothing standing in front of a mirror This suggests that he is now getting reading to leave the house The headband in the last frame is an important prop in Naruta it is a signifier of the ninja theme The social experience of a morning routine is thus created through the use of props location and sound effects Manga in particular is known to pace the narrative paying attention to small world details in order to re-create everyday social experiences (McCloud 19942006)

As seen in Figure 51 embodied action such as body posture gesture facial expression also contributes to the relaying of a particular social

experience The boys outstretched arms and wide-open mouth in Frame 1 convey that he has just woken up In Frame 3 the boys half-awake state is portrayed through his slumped body posture and half-dosed eyes The emphasis that manga places on embodied action to evoke social experishyences has resulted in the development of highly conventionalised images for conveying emotions which are particular to the visual culture of Japan For example in Figure 510 (right image) manga readers will be able to recshyognise that the character who is enlarged and depicted in an exaggerated super deformed state is a convention for expressing an extreme burst of emotion This is a technique often used for comic relief

To summarise the experiential metafunction is concerned with the repshyresented resources and how they function to depict a particular event into categories of social experience Semiotic resources which are able to realise the experiences include embodied actions (facial expressions body posture gesture) and resources which establish setting (costume props location) The orchestration of these resources is akin to what is known as mise en scene in film

LOGICAL METAFUNCTION WAYS OF MOVING THE VISUAL NARRATIVE FORWARD

It is generally accepted that narratives are characterised by time and cause and effect In visual narratives this is realised through placing frames in sequence and a causal relationship is created between the frame before and the frame after The logical metafunction is crucial when looking at visual

Teaching Visual Narratives Using a Social Semiotic Framework 77

narratives as it is concerned with the relationship between one process and another (Halliday 198545) According to Baldry and Thibault the logical metafunction is realised by recursive structures which add one element to another so as to build up more complex chain-like structures (200622) This means that the logical metafunction is about how elements of a text link together to form a coherent whole It can also be understood as how elements of a text push a narrative forward and thus aligns with Matthiesshysens notion of rhetorical relations According to Matthiessen rhetorical relations are concerned with the development of sequences of passages in a text (200733) He posits two ways in which images may be developed expansion and projection

Expansion refers to the augmentation of an image There are three difshyferent levels in which an image can be augmented elaboration extension and enhancement Elaboration refers to the restatement of an image To elaborate something is to build onto something where the foundation is still the same This can be understood as depicting the same image again but in greater detail or context For example an image may be elaborated through the means of zooming in or out That is a smaller part of an image can be focused on where although the context of the image is minimised the represented element is afforded greater detail Alternatively by zooming out greater context is disclosed but there is less focus on the represented element For example Frame 1 of Figure 52 presents a close-up image of

Frame 1

TWTlOtII( C8IlIgt

GtAfmlshy1116 THe

MOHI1oINSlD lJIIIIGE5

OFIU 1lpound ~OF ~VIUJISE- Frame 2

1OIJl ~

Figure 52 Elaboration by means of zooming in and out (Kishimoto 200310)

78 Cheng-Wen Huang

an old man smoking his pipe The old mans annoyance and unease at being disrupted are conveyed through his facial expression and the drop of sweat rolling down his forehead The sense of shock is also suggested by the four lines above his head The close-up view not only divulges his emotions but also what he is doing writing calligraphy While the close-up in Frame 1 does well in conveying the mans emotions and actiolls it gives the reader

information with regard to the context of the situation-where the man is situated and who the people are calling out to him Frame 2 discloses this information by zooming out of the close-up The wide-shot in Frame 2 provides information with regard to the context of situation but in doing so affords less focus and detail to the represented elements

Elaboration works with the same image elaborating it either by zoomshying in or zooming out Extension on the other hand propels the narrative forward by providing a new image The new image extends the existing image by providing additional information Although the new image is new in the sense that it has not been seen before it is still related to the previous image For instance in film terms extension can be realised through panning (camera moving sideways from a fixed position) tilting (camera moving up or down from a fixed position) or tracking (camera follows an object or participant) Figure 53 is an example of extension The top image establishes the time of the day (night) but it does not proshyvide details with regard to the setting The bottom frame provides this information by lowering the camera eye This is an example of extension through tilting

Enhancement is the augmentation of an image through a change in time and space This change in time and space pushes the narrative forward In language-based narratives this typically happens at the beginning ofchapters or sections within a chapter Since enhancement requires a change either in time or space I propose using the word transition In images the transition from one image to another can occur through techniques such as flashback or flashforward and split frames which evoke the notion of meanwhile Any image in which a temporal or spatial change has occurred in relation to a previous image can be seen as a transition Figure 54 is an example of a narrative being propelled forward through transition Reading from right to left the narrative is initially situated at a traditional Japanese restaurant in the evening The narrator exits this scene with an external birds eye view of the restaurant the perspective of an omniscient narrator In the next frame puffs of cloud indicate that there is a shift in time It is no longer evening but daytime The last frame takes the reader into a classroom Students are sitting slouched at their desks These three frames clearly indicate a shift in time and space This approach to shifting between scenes is typical of manga storytelling

Another way of pushing the narrative forward is through dialogue whether it is internal dialogue (thought) or external dialogue (speech) Matshythiessen (2007) refers to this process as projection Projection in comics is

Teaching Visual Narratives Using a Social Semiotic Framework 79

tilting an image (Kishimoto 200324) Figure 53 Extension

usually captured in frames Figure 55 is an example of projection through voice-over narration The image depicts two people having a discussion In Frame 1 the man sitting on the right can be identified through his costume as a ninja The headband the dark suit and the vest are characteristics of ninja iconography in Naruto The ninja has his hands up posed to ask the boy on the left a question The boy makes a response in Frame 1 and proceeds to answer the question in Frame 2 He elaborates on his answer and in doing so his explanation is carried into Frame 3 In Frame 3 the

80

Figure 54 A shift in time and space through transition (Kishimoto 200318)

Frame 2 Frame 1

Frame 4 Frame 3

Figure 55 Projection through a voiceover (Kishimoto 200317)

boys dialogue is overlapped with an image that is set in a different time and place The image would have been otherwise out of context in this sequence of images but the boys dialogue serves as a continuity element propelling the narrative forward

In sum the logical meta function is concerned with the logical progresshysion of a narrative how the story is projected forward from frame to frame

Teaching Visual Narratives Using a Social Semiotic Framework 81

Drawing from Matthiessen (2007) I have identified two approaches in which this can occur expansion and projection Expansion propels the narshyrative forward through elaboration extension and transition The narrative in these cases is developed through the images Projection on the other hand propels the narrative forward through verbal components that is through speech or thought

INTERACTIVE METAFUNCTION INTERACTIONS WITH THE READER

The interactive meta function is concerned with the interpersonal relationshyship between the image and the reader Specifically it is concerned with the social relations and the evaluative orientations between these two intershyactants Kress and Van Leeuwen (1996) identify three particular categoshyries which express the interactive metafunction contact social distance and attitude These categories are useful when looking as visual narratives because they provide an understanding of how semiotic resources function to elicit certain emotions from readers

Contact is concerned with the presence or absence of gaze from a represhysented participant In film terms this can be regarded as viewer identificashytion which is established through camera positioning Depending on who the director wishes the viewer to identify with the camera can be placed in a number of positions constructing a particular point of view I use the term point of view in this chapter instead of contact when referring to viewer identification because it is commonly used in both film and literature A first person point of view is established when the camera takes the position of the represented participant This position allows the viewer maximum identification with the represented participant Figure 56 (rop image) is an example of a subjective point of view where the depicted images simulate the vision of an eye opening The image begins with a totally black frame which imitates the vision of the closed eye A horizontal central split in the next image represents the eye partially opening This eye widens the entire image is clear in the last frame In this representation there is a strong interpersonal engagement between the reader and the character as the reader adopts the position of the character and views the world at the exact moment as the character

Third person point of view is established when the camera represents the unfolding narrative from a third person in the narrative Viewer identishyfication with the represented participant is less strong from a third person perspective The omniscient point of view presents a narrative unfolding from gods view or a birds eye view This is a point of view that does not belong to anyone in the narrative Viewer identification is absent from this viewpoint For example in Figure 56 the middle and bottom images are of the same location The point of view of the middle image is that of a third

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

Jo= _bullbullbullbullbull

~-- ~

~

social experiences through props location and sound

74 Cheng-Wen Huang

In sum the framework proposed in this chapter involves four metafuncshytions the experiential the logical the interactive and the compositional The usefulness of this framework for teaching visual narratives lies in the multi- dimensional view it offers for analysis The experiential metafunction highshylights the resources that communicate experiences of the world in a narrative The logical metafunction explicates how a narrative is developed through sequences of images The interactive metafunction explains the resources that realise social interactions In other words it provides an understanding of the power relations and attitudes that are constructed between the image and the viewer The compositional metafunction assists in understanding the composition of the image as a whole While integrating Kress and Van Leeuwen (1996) and Matthiessens (2007) work to produce a framework for analysing manga I replace some technical terms within their frameworks with film vocabularies both because the genre of manga draws on cinematic visual techniques (McCloud 1994 Rommens 2000) and in order to renshyder the metalanguage more accessible to students Thesen writes that rather than propose an entirely new set of vocabularies a practical metalanguage should serve as a way of verbalising what you know in relation to other ways of knowing (2001143) As film vocabularies are not only taught in visual literacy classes in school but can also be found in everyday language it seems appropriate to replace technical terms with a vocabulary borrowed from film Below I outline the framework and metalanguage in detail

EXPERIENTIAL METAFUNCTION WAYS OF PORTRAYING HUMAN EXPERIENCE

Halliday proposes that representations which produce particular meanings are always seen as the expression of some kind of a process (198518) The experiential metafunction is primarily concerned with how the world is categorised according to social experience In visual narratives resources such as props costume location colour and sound (in the form of onoshymatopoeia in written text) all contribute towards creating a representation of some kind of a social experience as can be seen in Figure 51

Figure 51 is a representation of a morning routine Panels in manga are read from right to left top to bottom In the first frame the backdrop of the narrative is established by the represented resources in the background on the right A village setting is evoked by the hut-like structures In parshyticular the kanji (Chinese character) ~ at the bottom centre of the frame points towards an Asian-oriented narrative world In the foreground on the left props such as a bed and a pillow allow the reader to recognise the present location as a bedroom The beam of light shining through the window and the sun and the clouds outside signify that it is morning The boys arms outstretched mouth wide open accompanied with the sound

76 Cheng- Wen Huang

yaaawn suggest that he has just woken up In the second frame the represented resources include a carton of milk and a glass These props are a signifier for breakfast In the third frame the boy can be seen sitting at a table Background props such as a cupboard pots of plants and posters on the wall that reads Do your best on the one and Ninja on the other convey a change in location The words munch munch indicate that the boy is eating breakfast Props such as the milk the glass the plate the sandwich and the sound effect of the onomatopoeias glug glug and munch munch all add to the construction of a breakfast atmosphere In the last row of frames the boy can be seen in a change of clothing standing in front of a mirror This suggests that he is now getting reading to leave the house The headband in the last frame is an important prop in Naruta it is a signifier of the ninja theme The social experience of a morning routine is thus created through the use of props location and sound effects Manga in particular is known to pace the narrative paying attention to small world details in order to re-create everyday social experiences (McCloud 19942006)

As seen in Figure 51 embodied action such as body posture gesture facial expression also contributes to the relaying of a particular social

experience The boys outstretched arms and wide-open mouth in Frame 1 convey that he has just woken up In Frame 3 the boys half-awake state is portrayed through his slumped body posture and half-dosed eyes The emphasis that manga places on embodied action to evoke social experishyences has resulted in the development of highly conventionalised images for conveying emotions which are particular to the visual culture of Japan For example in Figure 510 (right image) manga readers will be able to recshyognise that the character who is enlarged and depicted in an exaggerated super deformed state is a convention for expressing an extreme burst of emotion This is a technique often used for comic relief

To summarise the experiential metafunction is concerned with the repshyresented resources and how they function to depict a particular event into categories of social experience Semiotic resources which are able to realise the experiences include embodied actions (facial expressions body posture gesture) and resources which establish setting (costume props location) The orchestration of these resources is akin to what is known as mise en scene in film

LOGICAL METAFUNCTION WAYS OF MOVING THE VISUAL NARRATIVE FORWARD

It is generally accepted that narratives are characterised by time and cause and effect In visual narratives this is realised through placing frames in sequence and a causal relationship is created between the frame before and the frame after The logical metafunction is crucial when looking at visual

Teaching Visual Narratives Using a Social Semiotic Framework 77

narratives as it is concerned with the relationship between one process and another (Halliday 198545) According to Baldry and Thibault the logical metafunction is realised by recursive structures which add one element to another so as to build up more complex chain-like structures (200622) This means that the logical metafunction is about how elements of a text link together to form a coherent whole It can also be understood as how elements of a text push a narrative forward and thus aligns with Matthiesshysens notion of rhetorical relations According to Matthiessen rhetorical relations are concerned with the development of sequences of passages in a text (200733) He posits two ways in which images may be developed expansion and projection

Expansion refers to the augmentation of an image There are three difshyferent levels in which an image can be augmented elaboration extension and enhancement Elaboration refers to the restatement of an image To elaborate something is to build onto something where the foundation is still the same This can be understood as depicting the same image again but in greater detail or context For example an image may be elaborated through the means of zooming in or out That is a smaller part of an image can be focused on where although the context of the image is minimised the represented element is afforded greater detail Alternatively by zooming out greater context is disclosed but there is less focus on the represented element For example Frame 1 of Figure 52 presents a close-up image of

Frame 1

TWTlOtII( C8IlIgt

GtAfmlshy1116 THe

MOHI1oINSlD lJIIIIGE5

OFIU 1lpound ~OF ~VIUJISE- Frame 2

1OIJl ~

Figure 52 Elaboration by means of zooming in and out (Kishimoto 200310)

78 Cheng-Wen Huang

an old man smoking his pipe The old mans annoyance and unease at being disrupted are conveyed through his facial expression and the drop of sweat rolling down his forehead The sense of shock is also suggested by the four lines above his head The close-up view not only divulges his emotions but also what he is doing writing calligraphy While the close-up in Frame 1 does well in conveying the mans emotions and actiolls it gives the reader

information with regard to the context of the situation-where the man is situated and who the people are calling out to him Frame 2 discloses this information by zooming out of the close-up The wide-shot in Frame 2 provides information with regard to the context of situation but in doing so affords less focus and detail to the represented elements

Elaboration works with the same image elaborating it either by zoomshying in or zooming out Extension on the other hand propels the narrative forward by providing a new image The new image extends the existing image by providing additional information Although the new image is new in the sense that it has not been seen before it is still related to the previous image For instance in film terms extension can be realised through panning (camera moving sideways from a fixed position) tilting (camera moving up or down from a fixed position) or tracking (camera follows an object or participant) Figure 53 is an example of extension The top image establishes the time of the day (night) but it does not proshyvide details with regard to the setting The bottom frame provides this information by lowering the camera eye This is an example of extension through tilting

Enhancement is the augmentation of an image through a change in time and space This change in time and space pushes the narrative forward In language-based narratives this typically happens at the beginning ofchapters or sections within a chapter Since enhancement requires a change either in time or space I propose using the word transition In images the transition from one image to another can occur through techniques such as flashback or flashforward and split frames which evoke the notion of meanwhile Any image in which a temporal or spatial change has occurred in relation to a previous image can be seen as a transition Figure 54 is an example of a narrative being propelled forward through transition Reading from right to left the narrative is initially situated at a traditional Japanese restaurant in the evening The narrator exits this scene with an external birds eye view of the restaurant the perspective of an omniscient narrator In the next frame puffs of cloud indicate that there is a shift in time It is no longer evening but daytime The last frame takes the reader into a classroom Students are sitting slouched at their desks These three frames clearly indicate a shift in time and space This approach to shifting between scenes is typical of manga storytelling

Another way of pushing the narrative forward is through dialogue whether it is internal dialogue (thought) or external dialogue (speech) Matshythiessen (2007) refers to this process as projection Projection in comics is

Teaching Visual Narratives Using a Social Semiotic Framework 79

tilting an image (Kishimoto 200324) Figure 53 Extension

usually captured in frames Figure 55 is an example of projection through voice-over narration The image depicts two people having a discussion In Frame 1 the man sitting on the right can be identified through his costume as a ninja The headband the dark suit and the vest are characteristics of ninja iconography in Naruto The ninja has his hands up posed to ask the boy on the left a question The boy makes a response in Frame 1 and proceeds to answer the question in Frame 2 He elaborates on his answer and in doing so his explanation is carried into Frame 3 In Frame 3 the

80

Figure 54 A shift in time and space through transition (Kishimoto 200318)

Frame 2 Frame 1

Frame 4 Frame 3

Figure 55 Projection through a voiceover (Kishimoto 200317)

boys dialogue is overlapped with an image that is set in a different time and place The image would have been otherwise out of context in this sequence of images but the boys dialogue serves as a continuity element propelling the narrative forward

In sum the logical meta function is concerned with the logical progresshysion of a narrative how the story is projected forward from frame to frame

Teaching Visual Narratives Using a Social Semiotic Framework 81

Drawing from Matthiessen (2007) I have identified two approaches in which this can occur expansion and projection Expansion propels the narshyrative forward through elaboration extension and transition The narrative in these cases is developed through the images Projection on the other hand propels the narrative forward through verbal components that is through speech or thought

INTERACTIVE METAFUNCTION INTERACTIONS WITH THE READER

The interactive meta function is concerned with the interpersonal relationshyship between the image and the reader Specifically it is concerned with the social relations and the evaluative orientations between these two intershyactants Kress and Van Leeuwen (1996) identify three particular categoshyries which express the interactive metafunction contact social distance and attitude These categories are useful when looking as visual narratives because they provide an understanding of how semiotic resources function to elicit certain emotions from readers

Contact is concerned with the presence or absence of gaze from a represhysented participant In film terms this can be regarded as viewer identificashytion which is established through camera positioning Depending on who the director wishes the viewer to identify with the camera can be placed in a number of positions constructing a particular point of view I use the term point of view in this chapter instead of contact when referring to viewer identification because it is commonly used in both film and literature A first person point of view is established when the camera takes the position of the represented participant This position allows the viewer maximum identification with the represented participant Figure 56 (rop image) is an example of a subjective point of view where the depicted images simulate the vision of an eye opening The image begins with a totally black frame which imitates the vision of the closed eye A horizontal central split in the next image represents the eye partially opening This eye widens the entire image is clear in the last frame In this representation there is a strong interpersonal engagement between the reader and the character as the reader adopts the position of the character and views the world at the exact moment as the character

Third person point of view is established when the camera represents the unfolding narrative from a third person in the narrative Viewer identishyfication with the represented participant is less strong from a third person perspective The omniscient point of view presents a narrative unfolding from gods view or a birds eye view This is a point of view that does not belong to anyone in the narrative Viewer identification is absent from this viewpoint For example in Figure 56 the middle and bottom images are of the same location The point of view of the middle image is that of a third

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

76 Cheng- Wen Huang

yaaawn suggest that he has just woken up In the second frame the represented resources include a carton of milk and a glass These props are a signifier for breakfast In the third frame the boy can be seen sitting at a table Background props such as a cupboard pots of plants and posters on the wall that reads Do your best on the one and Ninja on the other convey a change in location The words munch munch indicate that the boy is eating breakfast Props such as the milk the glass the plate the sandwich and the sound effect of the onomatopoeias glug glug and munch munch all add to the construction of a breakfast atmosphere In the last row of frames the boy can be seen in a change of clothing standing in front of a mirror This suggests that he is now getting reading to leave the house The headband in the last frame is an important prop in Naruta it is a signifier of the ninja theme The social experience of a morning routine is thus created through the use of props location and sound effects Manga in particular is known to pace the narrative paying attention to small world details in order to re-create everyday social experiences (McCloud 19942006)

As seen in Figure 51 embodied action such as body posture gesture facial expression also contributes to the relaying of a particular social

experience The boys outstretched arms and wide-open mouth in Frame 1 convey that he has just woken up In Frame 3 the boys half-awake state is portrayed through his slumped body posture and half-dosed eyes The emphasis that manga places on embodied action to evoke social experishyences has resulted in the development of highly conventionalised images for conveying emotions which are particular to the visual culture of Japan For example in Figure 510 (right image) manga readers will be able to recshyognise that the character who is enlarged and depicted in an exaggerated super deformed state is a convention for expressing an extreme burst of emotion This is a technique often used for comic relief

To summarise the experiential metafunction is concerned with the repshyresented resources and how they function to depict a particular event into categories of social experience Semiotic resources which are able to realise the experiences include embodied actions (facial expressions body posture gesture) and resources which establish setting (costume props location) The orchestration of these resources is akin to what is known as mise en scene in film

LOGICAL METAFUNCTION WAYS OF MOVING THE VISUAL NARRATIVE FORWARD

It is generally accepted that narratives are characterised by time and cause and effect In visual narratives this is realised through placing frames in sequence and a causal relationship is created between the frame before and the frame after The logical metafunction is crucial when looking at visual

Teaching Visual Narratives Using a Social Semiotic Framework 77

narratives as it is concerned with the relationship between one process and another (Halliday 198545) According to Baldry and Thibault the logical metafunction is realised by recursive structures which add one element to another so as to build up more complex chain-like structures (200622) This means that the logical metafunction is about how elements of a text link together to form a coherent whole It can also be understood as how elements of a text push a narrative forward and thus aligns with Matthiesshysens notion of rhetorical relations According to Matthiessen rhetorical relations are concerned with the development of sequences of passages in a text (200733) He posits two ways in which images may be developed expansion and projection

Expansion refers to the augmentation of an image There are three difshyferent levels in which an image can be augmented elaboration extension and enhancement Elaboration refers to the restatement of an image To elaborate something is to build onto something where the foundation is still the same This can be understood as depicting the same image again but in greater detail or context For example an image may be elaborated through the means of zooming in or out That is a smaller part of an image can be focused on where although the context of the image is minimised the represented element is afforded greater detail Alternatively by zooming out greater context is disclosed but there is less focus on the represented element For example Frame 1 of Figure 52 presents a close-up image of

Frame 1

TWTlOtII( C8IlIgt

GtAfmlshy1116 THe

MOHI1oINSlD lJIIIIGE5

OFIU 1lpound ~OF ~VIUJISE- Frame 2

1OIJl ~

Figure 52 Elaboration by means of zooming in and out (Kishimoto 200310)

78 Cheng-Wen Huang

an old man smoking his pipe The old mans annoyance and unease at being disrupted are conveyed through his facial expression and the drop of sweat rolling down his forehead The sense of shock is also suggested by the four lines above his head The close-up view not only divulges his emotions but also what he is doing writing calligraphy While the close-up in Frame 1 does well in conveying the mans emotions and actiolls it gives the reader

information with regard to the context of the situation-where the man is situated and who the people are calling out to him Frame 2 discloses this information by zooming out of the close-up The wide-shot in Frame 2 provides information with regard to the context of situation but in doing so affords less focus and detail to the represented elements

Elaboration works with the same image elaborating it either by zoomshying in or zooming out Extension on the other hand propels the narrative forward by providing a new image The new image extends the existing image by providing additional information Although the new image is new in the sense that it has not been seen before it is still related to the previous image For instance in film terms extension can be realised through panning (camera moving sideways from a fixed position) tilting (camera moving up or down from a fixed position) or tracking (camera follows an object or participant) Figure 53 is an example of extension The top image establishes the time of the day (night) but it does not proshyvide details with regard to the setting The bottom frame provides this information by lowering the camera eye This is an example of extension through tilting

Enhancement is the augmentation of an image through a change in time and space This change in time and space pushes the narrative forward In language-based narratives this typically happens at the beginning ofchapters or sections within a chapter Since enhancement requires a change either in time or space I propose using the word transition In images the transition from one image to another can occur through techniques such as flashback or flashforward and split frames which evoke the notion of meanwhile Any image in which a temporal or spatial change has occurred in relation to a previous image can be seen as a transition Figure 54 is an example of a narrative being propelled forward through transition Reading from right to left the narrative is initially situated at a traditional Japanese restaurant in the evening The narrator exits this scene with an external birds eye view of the restaurant the perspective of an omniscient narrator In the next frame puffs of cloud indicate that there is a shift in time It is no longer evening but daytime The last frame takes the reader into a classroom Students are sitting slouched at their desks These three frames clearly indicate a shift in time and space This approach to shifting between scenes is typical of manga storytelling

Another way of pushing the narrative forward is through dialogue whether it is internal dialogue (thought) or external dialogue (speech) Matshythiessen (2007) refers to this process as projection Projection in comics is

Teaching Visual Narratives Using a Social Semiotic Framework 79

tilting an image (Kishimoto 200324) Figure 53 Extension

usually captured in frames Figure 55 is an example of projection through voice-over narration The image depicts two people having a discussion In Frame 1 the man sitting on the right can be identified through his costume as a ninja The headband the dark suit and the vest are characteristics of ninja iconography in Naruto The ninja has his hands up posed to ask the boy on the left a question The boy makes a response in Frame 1 and proceeds to answer the question in Frame 2 He elaborates on his answer and in doing so his explanation is carried into Frame 3 In Frame 3 the

80

Figure 54 A shift in time and space through transition (Kishimoto 200318)

Frame 2 Frame 1

Frame 4 Frame 3

Figure 55 Projection through a voiceover (Kishimoto 200317)

boys dialogue is overlapped with an image that is set in a different time and place The image would have been otherwise out of context in this sequence of images but the boys dialogue serves as a continuity element propelling the narrative forward

In sum the logical meta function is concerned with the logical progresshysion of a narrative how the story is projected forward from frame to frame

Teaching Visual Narratives Using a Social Semiotic Framework 81

Drawing from Matthiessen (2007) I have identified two approaches in which this can occur expansion and projection Expansion propels the narshyrative forward through elaboration extension and transition The narrative in these cases is developed through the images Projection on the other hand propels the narrative forward through verbal components that is through speech or thought

INTERACTIVE METAFUNCTION INTERACTIONS WITH THE READER

The interactive meta function is concerned with the interpersonal relationshyship between the image and the reader Specifically it is concerned with the social relations and the evaluative orientations between these two intershyactants Kress and Van Leeuwen (1996) identify three particular categoshyries which express the interactive metafunction contact social distance and attitude These categories are useful when looking as visual narratives because they provide an understanding of how semiotic resources function to elicit certain emotions from readers

Contact is concerned with the presence or absence of gaze from a represhysented participant In film terms this can be regarded as viewer identificashytion which is established through camera positioning Depending on who the director wishes the viewer to identify with the camera can be placed in a number of positions constructing a particular point of view I use the term point of view in this chapter instead of contact when referring to viewer identification because it is commonly used in both film and literature A first person point of view is established when the camera takes the position of the represented participant This position allows the viewer maximum identification with the represented participant Figure 56 (rop image) is an example of a subjective point of view where the depicted images simulate the vision of an eye opening The image begins with a totally black frame which imitates the vision of the closed eye A horizontal central split in the next image represents the eye partially opening This eye widens the entire image is clear in the last frame In this representation there is a strong interpersonal engagement between the reader and the character as the reader adopts the position of the character and views the world at the exact moment as the character

Third person point of view is established when the camera represents the unfolding narrative from a third person in the narrative Viewer identishyfication with the represented participant is less strong from a third person perspective The omniscient point of view presents a narrative unfolding from gods view or a birds eye view This is a point of view that does not belong to anyone in the narrative Viewer identification is absent from this viewpoint For example in Figure 56 the middle and bottom images are of the same location The point of view of the middle image is that of a third

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

78 Cheng-Wen Huang

an old man smoking his pipe The old mans annoyance and unease at being disrupted are conveyed through his facial expression and the drop of sweat rolling down his forehead The sense of shock is also suggested by the four lines above his head The close-up view not only divulges his emotions but also what he is doing writing calligraphy While the close-up in Frame 1 does well in conveying the mans emotions and actiolls it gives the reader

information with regard to the context of the situation-where the man is situated and who the people are calling out to him Frame 2 discloses this information by zooming out of the close-up The wide-shot in Frame 2 provides information with regard to the context of situation but in doing so affords less focus and detail to the represented elements

Elaboration works with the same image elaborating it either by zoomshying in or zooming out Extension on the other hand propels the narrative forward by providing a new image The new image extends the existing image by providing additional information Although the new image is new in the sense that it has not been seen before it is still related to the previous image For instance in film terms extension can be realised through panning (camera moving sideways from a fixed position) tilting (camera moving up or down from a fixed position) or tracking (camera follows an object or participant) Figure 53 is an example of extension The top image establishes the time of the day (night) but it does not proshyvide details with regard to the setting The bottom frame provides this information by lowering the camera eye This is an example of extension through tilting

Enhancement is the augmentation of an image through a change in time and space This change in time and space pushes the narrative forward In language-based narratives this typically happens at the beginning ofchapters or sections within a chapter Since enhancement requires a change either in time or space I propose using the word transition In images the transition from one image to another can occur through techniques such as flashback or flashforward and split frames which evoke the notion of meanwhile Any image in which a temporal or spatial change has occurred in relation to a previous image can be seen as a transition Figure 54 is an example of a narrative being propelled forward through transition Reading from right to left the narrative is initially situated at a traditional Japanese restaurant in the evening The narrator exits this scene with an external birds eye view of the restaurant the perspective of an omniscient narrator In the next frame puffs of cloud indicate that there is a shift in time It is no longer evening but daytime The last frame takes the reader into a classroom Students are sitting slouched at their desks These three frames clearly indicate a shift in time and space This approach to shifting between scenes is typical of manga storytelling

Another way of pushing the narrative forward is through dialogue whether it is internal dialogue (thought) or external dialogue (speech) Matshythiessen (2007) refers to this process as projection Projection in comics is

Teaching Visual Narratives Using a Social Semiotic Framework 79

tilting an image (Kishimoto 200324) Figure 53 Extension

usually captured in frames Figure 55 is an example of projection through voice-over narration The image depicts two people having a discussion In Frame 1 the man sitting on the right can be identified through his costume as a ninja The headband the dark suit and the vest are characteristics of ninja iconography in Naruto The ninja has his hands up posed to ask the boy on the left a question The boy makes a response in Frame 1 and proceeds to answer the question in Frame 2 He elaborates on his answer and in doing so his explanation is carried into Frame 3 In Frame 3 the

80

Figure 54 A shift in time and space through transition (Kishimoto 200318)

Frame 2 Frame 1

Frame 4 Frame 3

Figure 55 Projection through a voiceover (Kishimoto 200317)

boys dialogue is overlapped with an image that is set in a different time and place The image would have been otherwise out of context in this sequence of images but the boys dialogue serves as a continuity element propelling the narrative forward

In sum the logical meta function is concerned with the logical progresshysion of a narrative how the story is projected forward from frame to frame

Teaching Visual Narratives Using a Social Semiotic Framework 81

Drawing from Matthiessen (2007) I have identified two approaches in which this can occur expansion and projection Expansion propels the narshyrative forward through elaboration extension and transition The narrative in these cases is developed through the images Projection on the other hand propels the narrative forward through verbal components that is through speech or thought

INTERACTIVE METAFUNCTION INTERACTIONS WITH THE READER

The interactive meta function is concerned with the interpersonal relationshyship between the image and the reader Specifically it is concerned with the social relations and the evaluative orientations between these two intershyactants Kress and Van Leeuwen (1996) identify three particular categoshyries which express the interactive metafunction contact social distance and attitude These categories are useful when looking as visual narratives because they provide an understanding of how semiotic resources function to elicit certain emotions from readers

Contact is concerned with the presence or absence of gaze from a represhysented participant In film terms this can be regarded as viewer identificashytion which is established through camera positioning Depending on who the director wishes the viewer to identify with the camera can be placed in a number of positions constructing a particular point of view I use the term point of view in this chapter instead of contact when referring to viewer identification because it is commonly used in both film and literature A first person point of view is established when the camera takes the position of the represented participant This position allows the viewer maximum identification with the represented participant Figure 56 (rop image) is an example of a subjective point of view where the depicted images simulate the vision of an eye opening The image begins with a totally black frame which imitates the vision of the closed eye A horizontal central split in the next image represents the eye partially opening This eye widens the entire image is clear in the last frame In this representation there is a strong interpersonal engagement between the reader and the character as the reader adopts the position of the character and views the world at the exact moment as the character

Third person point of view is established when the camera represents the unfolding narrative from a third person in the narrative Viewer identishyfication with the represented participant is less strong from a third person perspective The omniscient point of view presents a narrative unfolding from gods view or a birds eye view This is a point of view that does not belong to anyone in the narrative Viewer identification is absent from this viewpoint For example in Figure 56 the middle and bottom images are of the same location The point of view of the middle image is that of a third

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

80

Figure 54 A shift in time and space through transition (Kishimoto 200318)

Frame 2 Frame 1

Frame 4 Frame 3

Figure 55 Projection through a voiceover (Kishimoto 200317)

boys dialogue is overlapped with an image that is set in a different time and place The image would have been otherwise out of context in this sequence of images but the boys dialogue serves as a continuity element propelling the narrative forward

In sum the logical meta function is concerned with the logical progresshysion of a narrative how the story is projected forward from frame to frame

Teaching Visual Narratives Using a Social Semiotic Framework 81

Drawing from Matthiessen (2007) I have identified two approaches in which this can occur expansion and projection Expansion propels the narshyrative forward through elaboration extension and transition The narrative in these cases is developed through the images Projection on the other hand propels the narrative forward through verbal components that is through speech or thought

INTERACTIVE METAFUNCTION INTERACTIONS WITH THE READER

The interactive meta function is concerned with the interpersonal relationshyship between the image and the reader Specifically it is concerned with the social relations and the evaluative orientations between these two intershyactants Kress and Van Leeuwen (1996) identify three particular categoshyries which express the interactive metafunction contact social distance and attitude These categories are useful when looking as visual narratives because they provide an understanding of how semiotic resources function to elicit certain emotions from readers

Contact is concerned with the presence or absence of gaze from a represhysented participant In film terms this can be regarded as viewer identificashytion which is established through camera positioning Depending on who the director wishes the viewer to identify with the camera can be placed in a number of positions constructing a particular point of view I use the term point of view in this chapter instead of contact when referring to viewer identification because it is commonly used in both film and literature A first person point of view is established when the camera takes the position of the represented participant This position allows the viewer maximum identification with the represented participant Figure 56 (rop image) is an example of a subjective point of view where the depicted images simulate the vision of an eye opening The image begins with a totally black frame which imitates the vision of the closed eye A horizontal central split in the next image represents the eye partially opening This eye widens the entire image is clear in the last frame In this representation there is a strong interpersonal engagement between the reader and the character as the reader adopts the position of the character and views the world at the exact moment as the character

Third person point of view is established when the camera represents the unfolding narrative from a third person in the narrative Viewer identishyfication with the represented participant is less strong from a third person perspective The omniscient point of view presents a narrative unfolding from gods view or a birds eye view This is a point of view that does not belong to anyone in the narrative Viewer identification is absent from this viewpoint For example in Figure 56 the middle and bottom images are of the same location The point of view of the middle image is that of a third

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

shy

Teaching Visual Narratives Using a Social Semiotic Framework 83

person It is as if someone within the narrative is looking at the restaurant from the outside In the bottom image the restaurant is viewed from above from a birds eye view This is not a position anyone in the narrative can occupy thus it can be said that the point of view comes from that of the omniscient narrator

An image is not only always seen from a particular point of view but also from a particular distance The term social distance refers to the proximity between the subject depicted and the audience Kress and Van Leeuwen (1996) identify three types of proximity intimatepersonal social and impersonal In film social distance is realised through the type of shot used an intimatepersonal distance through a close-up shot a social disshytance through a medium shot and an impersonal distance through a long shot The type of shot used not only reveals the social distance but also establishes how much information is disclosed in an image For example a long-distance shot places a distance between the character and the reader but provides the reader more information about the characters environshyment In film terms a long shot which establishes the scene is referred to as the establishing shot

Another resource that is closely related to point of view and social distance is the angle from which an image is depicted The term attitude (Kress amp Van Leeuwen 1996) refers to the interpersonal attitude specifically the power and the level of involvement realised through the positioning of camera angles Relationships of power are constructed through the use of vertical angles (low medium high) For example a low angle depicting the represented participant looking down at the viewer can convey power over the viewer In contrast a high angle which depicts the viewer gazing down at the represented participant can afford the viewer power An angle which is placed at eye level can establish a feeling of equality

The level of involvement which the viewer has with the represented world can be constructed through the use of horizontal angles (frontal or oblique) frontal shots create maximum engagement as the viewer is directshyly confronted with the world of the represented In contrast oblique shots suggest detachment as the viewer looks at the world of the represented from the side It is important to note that these affordances are not cast in stone rather as Jewitt and Oyama write They are an attempt to describe a meanshying potential a field of possible meanings which needs to be activated by the producers and viewers of images (2001135) In other words dependshying on their usage these angles have the potential to mean otherwise in difshyferent contexts

The human form is an emotionally expressive resource that invites the reader to take on particular evaluative stances Attitude is realised not only through camera angles but also through facial expressions body postures and gestures Often these resources work together to convey emotion or attitude This is especially effective when the interaction between the repshyresented element and the reader is in the form of direct address Figure 57

Figure 56 First person point of view (top) third person point of view (middle) birds eye view (bottom) (Kishimoto 200358 16 18)

I --I I -I ~ ~bull

It - _

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

Lines and shapes as semiotic resources

84 Cheng-Wen Huang

AJYOJ Qt$tCN ~m

STtfIDTRlOCS )tVfffJRflf

Figure 57 The index finger demands interaction from the viewer (Kishimoto 200312)

is an example of point of view working with gesture and facial expresshysion inviting the reader to engage directly in the narrative The slightly low angle from a first person perspective the index figure pointed directly at the reader as well as the fierce facial expression of the represented character all work together to create a feeling of powerlessness in the reader The resources demand a first-person interaction and thus invite the reader to actively participate in the narrative world

In manga facial expressions body posture and gestures tend to be exagshygerated This has the effect of amplifying the readers sense of participation as heshe comes to identify with the characters their actions and emotions As McCloud points out Humans love humans They cant get enough of themselves They crave the company of humans they value the opinion of humans and they love hearing stories about humans (200660) It is thus not surprising that the human form is most effective in drawing emotional responses from the reader

Other semiotic resources which elicit interpersonal meanings in visual narratives include lines and shapes In Figure 57 the sharp lines extending from the speech frames are volume lines which portray the loudness of the dialogue The lines or the loudness of the speech support the dialogue facial expression and gesture in creating a frightening impression Lines and shapes can also work together to signify emotions such as excitement anger or tension For instance in Figure 58 the dotted lines which frame the exclamation mark convey tension and alarm The puff of smoke suggests that the ninja is letting off steam The symbols used here are particular to manga iconography

This section has examined how point of view social distance and attitude realise interpersonal meanings in manga Other semiotic resources which are powerful in conveying interpersonal meanings include facial expression body posture gesture lines and shapes The meanings are never communishycated through a single resource alone but rather through a combined readshying of all the semiotic resources

Teaching Visual Narratives Using a Social Semiotic Framework 85

Figure 58

- z bull 11 1

llllllVlV 200329)

COMPOSITIONAL METAFUNCTION LAYOUT AS SEMIOTIC RESOURCE

Composition is the way in which the representational and interactive eleshyments are made to relate to each other the ways they are integrated into a meaningful whole (Kress amp Van Leeuwen 1996181) Layout is a key aspect of compositional meaning Kress and Van Leeuwen identify three inter-related systems which are concerned with the layout of a text inforshymation value salience and framing

Information value describes the value attributed to the different posishytions of a page For example the centre is generally regarded as having more value than the margin and in the West the left side of a page has the meaning of the Given the familiar and the right side of the page has the value of the New the unfamiliar (Kress amp Van Leeuwen 1996) This differs greatly to the reading practices of the East where the reading path of comics is from right to left top to bottom In their comparative analysis of British and Japanese ad vertisements Jewitt and Oyama (2001) note that the information value of GivenlNew attributed to the left and right side of the page respectively seem to be reversed that is the Given appears on the right and the New appears on the left This is the case in manga For examshyple in Figure 59 a new character who appears out of nowhere in Frame 2 is introduced into the narrative As the new element in the narrative he is placed on the left side of the page while the cloaked man having been introduced earlier in the narrative is placed on the right The positioning of the characters in these frames is most important to the narrative flow The move from Frame 3 to 4 is a case of extension through projection-a case of projection through speech The ninjas body posture along with the sound effect SHF in Frame 3 communicates the idea of someone taking

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

86 Cheng- Wen Huang

a deep breath and prepares the reader for some outburst The outburst is released in Frame 4 as the explosive words matching the action burst through the speech frame on the right Thus despite frames dividing the two images the body posture and the projection allow a continuous flow in the reading path In this case the success of the projection is very much dependent on the positioning of the represented participants and these parshyticipants have clearly been positioned in line with the right-to-left reading path This example demonstrates that information value is deeply rooted in the reading practices of a culture However one should also bear in mind that signifying systems will have relations of homology with other cultural systems whether religious philosophical or practical (Kress amp Van Leeushywen 1996199) which suggests that meanings attributed to positions on a page may overlap in certain cases especially when practices are increasingly merging as a result of a globalised context

Besides the layout within an image the layout of a page as a whole is important in establishing narrative flow For example scene transitions and climax points are more effective when placed in certain positions on a page Page turning is also an important transitional device that is effective in establishing scene changes and achieving moments of climax

The notion of salience applies to the layout of a text where the represhysented elements are given visual prominence through particular techniques For instance foregrounding and back grounding can be achieved through the allocation of space Elements at the forefront tend to be larger in size than those in the background The larger the size of the represented element

Frame 2 Frame 1

Frame 3Frame 4

Figure 59 Extension through projection (Kishimoto 200311)

Teaching Visual Narratives Using a Social Semiotic Framework 87

the more prominent it is as in Frame 3 of Figure 59 Compared to the cloaked man on the right who is partially framed from over the shoulder the ninja on the left holds the readers attention more due to his size and more complete representation Of course the central positioning of the man also plays a role in foregrounding the ninja Other semiotic resources which realshyise salience include colour saturation CentreMargin positioning and blurshyring or sharpening the represented through focus In sum salience applies to the degree to which an element draws attention to itself due to its size its place in the foreground or its overlapping of other elements its colour its tonal values its sharpness or definition and other features (Kress amp Van Leeuwen 1996225)

Framing describes the use of elements such as border lines Frames can function to link or detach elements of a text signifying that they belong or do not belong together in some sense through its presence or absence (Kress amp Van Leeuwen 1996225) This is an essential compositional device for manga and comic art in generaL Sequential frames create a flow in the narrative by segmenting one moment followed by another This creates the illusion of the passage of time and the notion of cause and effect As Ryan writes

The reader (for the eye movement amounts to an act of reading) conshystructs a story line by assuming that similar shapes on different frames represent common referents (objects characters or setting) by intershypreting spatial relations as temporal sequence (adjacent frames represhysent subsequent moments) and by inferring causal relations between the states depicted in the frames (2004 141)

The actual shape of the frames also has meaning-making potentials Accordshying to Baldry and Thibault frames function as a metacomment which helps to signify how the world depicted in the frame should be interpreted (200610) such as in the case with the images on Figure 510 In the image

Frames as a metacomment on the narrative world (Kishimoto 200331)

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

88 Cheng- Wen Huang

on the left the diagonal frames simulate the action happening at that moment The sharp edges formed as a result of the diagonal frames create the illusion of daggers slashing through the page Even the frames of speech bubbles can make a metacomment about the dialogue framed within it In Figure 510 (right image) the jagged framing of the speech bubble reflects the anger intensity and volume of the voice

PEDAGOGICAL IMPLICATIONS A METALANGUAGE OF MANGA TO INTERROGATE OTHER VISUAL NARRATIVES

This chapter has presented a multimodal social semiotic framework for anashylysing manga The strength of this framework lies in the multidimensional view of text it provides The experiential metafunction reveals how ideas and experiences of the world can be conveyed through semiotic resources such as setting (costume props location) and embodied action (facial expresshysions body posture gesture) resources also known as mise en scene in film

logical metafunction explains how narrative can be propelled forshyward through image (expansion) and text (projection) The interactive meta function describes how attitude and power relations are conveyed through resources such as camera angles facial expressions body postures and gestures The compositional metafunction reveals how layout affects meaning of texts Although the framework presented here is for the analysis of manga specifically there is the possibility of extending this framework to interrogate other visual narratives of similar nature

Culture is central to the development of semiotic resources and their use (Huang amp Archer 2012 Kress 2003) The strength of multimodal social semiotics as a theoretical approach is that it foregrounds the need to undershystand the social realities that affect text-making practices

In the compositional metafunction for example I have pointed out how the information value of a given text is dependent on the context of situashytion and the context of culture from which the text emerges In manga the value of GivenlNew that is attributed to the left and right side of a page is reversed because it is affected by the reading direction of the East The New London Group write that a practical metalanguage is one which can identify and explain difference between texts and relate these to the conshytexts of culture and situation in which they seem to work (200024) The strength of building a metalanguage from a social semiotic approach is that it grounds uses of semiotic resources to context

Throughout this chapter I have pointed to the similarities between conshyventions used in manga and conventions used in film What this points towards is the overlapping nature of genres and conventions Kress writes The mixing of genre has to be a reality simply as an effect of our ordishynary normal social lives and our ordinary normal use of language constant change has to be seen as entirely normal as an effect of a social theory

Teaching Visual Narratives Using a Social Semiotic Framework 89

(200387) He argues that the mixing of genres is unavoidable because as in any social process there will always be new situations which requires new genres New genres however are never entirely new but are built on and adapted from previous genres they are changed in particular ways to suit the new context of use A metalanguage of analysis which is able to identify and discuss different forms of meaning can assist in the transforshymation process by creating a dialogue between old and new genres It can help students to recontextualise meanings and apply what they know in relation to other ways of knowing (Thesen 2001143) This points to the possibility of using a metalanguage of manga to examine and produce other visual narratives of this kind

Alien K amp Ingulsrud JE 2003 Manga literacy Popular culture and the reading habits of Japanese college students Journal of Adolescent amp Adult literacy 46 (8)674-683

Alvermann DE amp Heron AH 2001 Literacy identity work Playing to learn with popular media Journal of Adolescent amp Adult Literacy 45118-122

Archer A 2010 Multimodal texts in higher education and the implications for writing pedagogy English in Education 44 (3)201-213

Baldry AP amp Thibault P] 2006 Multimodal transcription and text analysis A multimedia toolkit and coursebook London and New York Equinox

Bordwell D amp Thompson K 2001 Film art An introduction 6th edition New York McGraw Hill

Brienza G 2009 Books not comics Publishing fields globalization and Japanese manga in the United States Publication Research Quarterly 25 1 01-117

Brown K 2008 Manga Shakespeare A midnight summers dream UK SelfMadeHero Halliday MAK 1978 Language as social semiotic The social interpretation

language and meaning London Arnold Halliday MAK 1985 An introduction to functional grammar London Arnold Halliday MAK 2003 On language and linguistics Volume 3 in the collected

works ofMAK Halliday Webster J (ed) London and New York Continuum Halliday MAK amp Hasan R 1985 Language context and text Aspects of lanshy

guage in a social-semiotic perspective Belmont Victoria Deakin University Huang C amp Archer A 2012 Uncovering the multimodal literacy practices in readshy

ing manga and the implications for pedagogy In Williams B amp Zenger A (cds) New media literacies and participatory tlOPular culture across borders New York and London Routledge 44-60

Jewitt c 2009 The Routledge handbook ofmultimodal analysis London and New York Routledge

Jewitt c amp Oyama R 2001 Visual meaning A social semiotic approach In Van Leeuwen T amp Jewitt C (eds) The handbook ofvisual analysis London SAGE 134-156

Kishimoto M 2003 Naruto Volume 1 San Francisco VIZ Media Kress G 2003 Literacy in the new media age London Routledge Kress G 2010 Multimodality A social semiotic approach to contemporary comshy

munication London and New York Routledge Kress G amp Van Leeuwen T 1996 Reading images The grammar of visual design

London Routledge Leong S 2007 Manga Shakespeare Romeo and Juliet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero

90 Cheng- Wen Huang

Lim VE 2007 The visual semantics stratum Making meaning in sequential images In Royce TD amp Bowcher WL (eds) New directions in the analysis of multishymodal discourse Mahwah NJ L Erlbaum Associates 195-214

Martinec R amp Salway A 2005 A system for image-text relations in new (and old) media Visual Communication 4 (3)337-371

Matthiessen CMIM 2007 The multimodal page A systemic functional exploshyration In Royce TD amp Bowcher WL (eds) New directions in the analysis of multimodal discourse Mahwah NJ L Erlbaum Associates 1-62

McCloud S 1994 Understanding comics The invisible art New York HarperPerennia McCloud S 2006 Making comics Storytelling secrets of comics manga and

graphic novels New York London Toronto and Sydney Harper Norton B amp Vanderheyden K 2005 Comic book culture and second language

learners In Norton B amp Toohey K (eds) Critical pedagogies and language learning Cambridge England Cambridge University Press 201-222

OHalloran KL 2004 Visual semiosis in film In OHalloran KL (ed) Multishymodal discourse analysis London and New York Continuum 109-130

Rommens A 2000 Manga story-tellingshowing Image [amp] Narrative Online Magazine of the Visual Narrative 1 (1) Available wwwimageandnarrativebe narratolo$yaarnoudrommenshtm [2008 8 May]

Rubinstein-Avila E amp Schwartz A 2006 Understanding the manga hype Uncovshyering the multimodality of comic-book literacies Journal of Adolescent amp Adult Literacy 50 (1)40-49

Ryan M (ed) 2004 Narrative across media The language of storytelling Lincoln University of Nebraska Press

The New London Group 2000 A pedagogy of multiliteracies Designing social futures In Cope B amp Kalantzis M (eds) Multiliteracies Literacy learning and the design of social futures London and New York Routledge 9-37

Thesen L 2001 Modes literacies and power A university case study Language and Education 15 (2 amp 3)132-145

Thibault P 2000 The multimodal transcription of a television advertisement Theory and practice In Baldry A (ed) Multimodality and multimediality in the distance learning age Campobasso Italy Palladino Editore 311-385

Unsworth L 2006 Towards a metalanguage for multiliteracies education Describshying the meaning-making resources of language-image interaction English Teachshying Practice and Critique 5 (1 )55-76

Van Leeuwen T 2005 Introducing social semiotics London and New York Routledge

Vieceli E 2007 Manga Shakespeare Hamlet UK SelfMadeHero