Rappresentare la multimodalità: approcci, problematiche, prospettive

Daniela Veronesi

Free University of Bozen-Bolzano, Language Study Unit/Faculty of Education

daniela.veronesi@unibz.it

Bolzano, Language Documentation 5, December 15, 2014

"Wir verstehen Interaktion als multimodale Hervorbringung aller Beteiligten und gehen davon aus, dass alle den Interaktionsbeteiligten zur Gestaltung der Interaktion zur Verfügung stehenden Modalitäten theoretisch zunächst einmal gleichwertig sind“ (Schmitt 2007: 21) "Eine multimodale Vorstellung von Interaktion macht es nötig, bei der konkreten Vollzugsanalyse (…) immer gleichzeitig alle Beteiligten im Auge zu behalten und nach deren spezifischen Beiträgen für das Zustandekommen der Interaktionsordnung zu fragen." (Schmitt 2007: 21) „A central question raised by the study of human interaction is how different modalities—talk, gaze, gesture—are integrated so as to form coherent courses of action. In their review of the literature Jones and LeBaron (2002) argued that until quite recently, “verbal and non-verbal messages” were “studied separately, as though they were independent rather than co-occuring and related phenomena (p. 499). […] In contrast, current work on multimodality focusses on questions of “integration” by putting at the forefront the question of how different modalities are integrated so as to form coherent courses of action. The promise of multimodality as an analytical orientation is, then, to explicate the formal structures of activity within which different modalities are integrated […]. This involves examining interaction not for gesture, gaze, or particular features of the talk per se but for the activities that the participants understand themselves to engage in. this, in turn, requires a shift away from a model in which the phenomena are characterized as messages and meanings towards one more squarely fouces on action”. (Sidnell 2006: 379-380) Convenzioni di trascrizione (cfr. Jefferson 2004) TEXT especially loud sound (relative to sorrounding talk) ºtextº especially soft sound (relative to sorrounding talk) text, text some form for stress (via pitch and/or amplitude) >text<, <text> faster speech, slower speech tex- cut off te:xt sound prolongation . falling, conclusive intonation , heard as unfinished intonation, slightly rising ?; ! rising intonation (question); animated utterance .h; h inbreath; outbreath ↑ ↓ shift into higher or lower pitch (.); (0.3), (1) short pause (less than 0.2); longer pause (duration) [ overlapping talk Convenzioni di trascrizione per azioni visibili * * start and end of participant's gaze Φ Φ start and end of participant's gesture . . . . . action preparation - - - - action continuation , , , , , action retraction _____ frozen gesture - -> continuation of action in the next line >- - beginning of action before current line

1. PRATICHE DI TRASCRIZIONE: ORIENTAMENTO VISIVO 1.1. Goodwin (1981)

1.2. Rossano, Brown & Levinson (2009)

.  .  .  .  .  .  .  .  .  .   fase  in  cui  l’interlocutore  comincia  a  portare  lo  sguardo     verso   il  parlante _________          sguardo  rivolto  all’interlocutore X   punto  preciso  in  cui  avviene  l’allineamento

 

1.3. Goodwin & Goodwin (1996)

2. PRATICHE DI TRASCRIZIONE: GESTUALITA' 2.1. Streeck (2009)

2.2. Kendon (2004)

 2.3. Lymer, Lindwall & Ivarsson (2011)

2.4. PItsch (2007)

2.5. Schmitt (2005)

2.6. Goodwin (2013)

 3. PRATICHE DI TRASCRIZIONE: MOVIMENTO NELLO SPAZIO 3.1. Broth & Mondada (2013)

4. PRATICHE DI TRASCRIZIONE: RAPPRESENTARE LA MULTIMODALITA' IN CONTESTI PERFORMATIVI 4.1. Keevallik (2010)

 

4.2. Black (2014)

4.3. Weeks (1996)

4.4. Veronesi (2014)

b.

 c.

((text)) comment; description of action * * start and end of conductor's gaze or body movement Φ Φ start and end of conductor’s gesture (right hand, rh) ∆ ∆ start and end of conductor's gesture (left hand, lh) Ω Ω start and end of conductor's gestures (both hands) ppp pointing gesture SUSTAIN Conduction directive . . . . . action preparation - - - - action continuation , , , , , action retraction - -> continuation of action in the next line >- - beginning of action before current line dee da sung expressions ♩ music activity (music playing) BIBLIOGRAFIA (selezione) Atkinson, J.M. & Heritage, J. (eds.). Structures of Social Action. Studies in Conversation Analysis.

Cambridge, Cambridge University Press. Auer, P. & Di Luzio, A. (eds.). The Contextualization of Language. Amsterdam/Philadelphia,

Benjamins. Black (2014). The intersubjective space-time of a Zulu choir/HIV support group in global perspective.

Social Semiotics 24(4): 381-401. Bonaiuto, M. & Maricchiolo, F. (2003). La comunicazione non verbale. Roma, Carocci. Broth, M. & Mondada L. (2013). Walking away: The embodied achievement of activity closings in

mobile interaction. Journal of Pragmatics 47: 41-58. Fele, G. (2007). L'analisi della conversazione. Bologna, Il Mulino. Gavioli, L. (1999). Alcuni meccanismi di base dell'analisi della conversazione. In R. Galatolo & G.

Pallotti (eds.). La conversazione. Milano, Cortina, 38-61. Goffman, E. (1981). Forms of talk. Philadelphia, University of Philadelphia Press. Goodwin, C. (1981). Conversational Organization. Interaction Between Speakers and Hearers. New

York, Academic Press. Goodwin, C. (2013). The co-operative, transformative organization of human action and knowledge.

Journal of Pragmatics 46(1), 8-23, special issue Conversation Analytic Studies of Multimodal

Interaction. Goodwin, C. & Goodwin M.H. (1996). Seeing as a Situated Activity: Formulating Planes. In Y.

Engestroem & D. Middleton (eds.). Cognition and Communication at Work. Cambridge, Cambridge University Press, 61-95.

Goodwin M. H. & Goodwin, C. (1986). Gesture and coparticipation in the activity of searching for a word. Semiotica, 62(1/2): 51-75. Haviland, J. B. (2011). Musical spaces. In C. Goodwin, J. Streeck & C. LeBaron (eds.). Embodied

Interaction. Language and Body in the Material World. Cambridge, Cambridge University Press, 298-304.

Jefferson, G. (2004). Glossary of transcript symbols with an introduction. In G. Lerner (ed.). Conversation Analysis: studies from the first generation. Amsterdam/Philadelphia, Benjamins, 13-23.

Keevallik (2010). Bodily Quoting in Dance Correction. Research on Language and Social Interaction 43(4): 401-426.

Kendon, A. (2004). Gestures: Visible Action as Utterance. Cambridge, Cambridge University Press. Knoblauch, H. (ed.) 2006. Video Analysis: Methodology and methods. Qualitative audiovisual data in

sociology. Frankfurt, Lang. Lindwall, O. & Ekström, A. (2012). Instruction-in-Interaction: The Teaching and Learning of a Manual

Skill. Human Studies 35(1), 27-49. Lomax, H. & Casey, H. 1998. Recording Social Life: Reflexivity and Video Methodology. Sociological

Research Online 3 (2) (http://www.socresonline.org.uk/socresonline/3/2/1.html). Lymer, G., Lindwall, O. & Ivarsson, J. (2011). Space and discourse interleaved: intertextuality and

interpretation in the education of architects. Social Semiotics 21(2): 197-217. Mondada, L. (2007). Multimodal resources for turn-taking: pointing and the emergence of possible

next speakers. Discourse Studies 9 (2): 195-226. Mondada, L. (2009). Commentary: transcript variations and the indexicality of transcribing practices.

Discourse Studies 9(6): 809-821. Müller, C. & Posner, R. (eds.) (2003). The Semantics and Pragmatics of Everyday Gestures. The

Berlin Conference. Berlin, Weidler. Pitsch, K. (2007). Koordinierung von parallelen Aktivitäten: Zum Anfertigen von Mitschriften im

Schulunterricht. In Schmitt (ed.), 411-446. Rossano, F., Brown, P., & Levinson, S. C. (2009). Gaze, questioning and culture. In J. Sidnell (ed.),

Conversation analysis: Comparative perspectives. Cambridge, Cambridge University Press, 187-249.

Sacks, H. (1992). Lectures on Conversation (edited by Gail Jefferson). Basil Blackwell, Oxford. Sacks, H., Schegloff, E. & Jefferson, G. (1974). A Simplest Systematics of the Organisation of Turn-

Taking for Conversation. Language 50: 696-735. Schegloff, E.A., Jefferson, G. & Sacks, H. (1977). The preference for self-correction in the

organization of repair in conversation. Language 53: 361-382. Schmitt, R. (2005). Zur multimodalen Struktur von turn-taking. Gesprächsforschung-Online-Zeitschrift

zur verbalen Interaktion 6: 17-61 (www.gespraechsforschung-ozs.de). Schmitt, R. (ed.) (2007). Koordination. Analysen zur multimodalen Organisation. Tübingen, Narr. Sidnell, J. (2006). Coordinating Gesture, Talk and Gaze in Reenactments. Research on Language

and Social Interaction 39(4): 377-409. Streeck, J. (2009). Gesturecraft. The manu-facture of meaning. Amsterdam, Benjamin. Streeck, J. & Henderson, D. (2010). Das Handwerk des Hip-Hop: Freestyle als körperliche Praxis. In

C. Wulf & W. Fischer-Lichte (eds.). Gesten: Inszenierung, Aufführung und Praxis. München, Wilhelm Fink, 62-92.

Veronesi, D. (2014). Correction sequences and semiotic resources in ensemble music workshops: the case of Conduction®". Social Semiotics 24(4): 468-494.

Weeks, P. (1996). A Rehearsal of a Beethoven Passage. An Analysis of Correction Talk. ROLSI 29(3): 247-290.

Weeks, P. (2002). Performative Error-Correction in Music: A Problem for Ethnomethodological Description. Human Studies 25: 359-385.