From icono-linguistic unity to semiotic continuity. An alternative description of semiotic repertoire of comics

Hubert Kowalewski

Department of Cognitive LinguisticsMaria Curie-Skłodowska University

Lublin, Poland

This a pre-print version. The final, definitive version of this paper has been published in International Journal of Comic Art, Vol. 17, No.1, (Spring/Summer 2015) (www.ijoca.com)

Text & Figures 4-7 © Hubert KowalewskiFigure 1 © Dave McKean. The figure constitutes a “fair use.”Figure 2 © Kelley Jones & Mike Dringenberg. The figure constitutes a “fair use.”Figures 3 © Dylan Horrocks. Used with permission.

1

1. Proem

The physical characteristics of any medium define ways of expressing concepts

through signs. Similar notions are expressed differently and spoken language, written

language, comics, films, etc. John Lyons notices that onomatopoeic words do not always

resemble the concepts they denote, because similarity “always dependent upon properties of

the medium in which the form is manifest” (1977: 105). Thus, the word “cuckoo” is

onomatopoeic only when it is spoken, but not when it is written.

The physical environment of comics medium, a two dimensional surface on which

static images are drawn (painted, computer generated, etc.), grants opportunities and imposes

restrictions on how meanings can expressed and how signs are organized into higher-level

structures. Most semiotic characterizations of comics recognize the synergy between

linguistic and figurative signs (the latter are often called “pictorial” or “iconic”). In other

words, most definitions of comics emphasize a dichotomy and interaction between two types

of semiotic components; this approach can be called “icono-linguistic unity” (unité iconico-

linguistique; Bernard Toussaint’s term, 1976: 82)

This article has a two-fold goal. The critical part demonstrates the inadequacy of

icono-linguistic unity as a framework for describing the semiotic complexity of comics. A

number of representative formulations of icono-linguistic unity are discussed in order to

reveal their internal inconsistencies and demonstrate how icono-linguistic unity fails to

account for signs like special lettering, onomatopoeias and speech balloons. The second part

proposes an alternative characterization of comics called “semiotic continuity”: a descriptive

framework that overcomes the difficulties shown in the critical part by offering a more

comprehensive and fine-grained description of signs used in comics.1

2

2. Semiotic definitions of comics

As already mentioned, traditional semiotics-based definitions revolve around two

broad categories of signs. Bernard Toussaint postulates icono-linguistic unity, referring

explicitly to linguistic and iconic signs. Another incarnation of this approach can be found in

Randy Duncan and Matthew J. Smith’s The Power of Comics, where the authors write about

the “interdependence of pictorial and linguistic elements” (2009: 138). Robert C. Harvey

emphasized the synergetic fusion of “text and image” (cf. 1994: 8-9), introducing the notion

of text, rather language. Michał Błażejczyk, in turn, cautiously claims that “the main

components [of comics] are graphic elements that are not words, however words can play a

significant role”2 (2001; trans. H.K.). Scott McCloud in his seminal Understanding Comics,

is of the opinion that comics narratives consist of “juxtaposed pictorial and other images in

deliberate sequence, intended to convey information and/or to produce an aesthetic response

in the viewer” (1994: 9). This definition draws a puzzling distinction between “pictorial

images” and “other images,” but excludes language. The most original approach is adopted

by Neil Cohn, who insists that while “comics themselves are not a language,” they “are

written in visual languages the same way that novels or magazines are written in English” (cf.

2014: 2). In other words, comics narratives are viewed as are texts created by means of a

“visual language.”

The list could continue, but other definitions employ around categories already

evoked in the five examples quoted above. All definitions of comics emphasize its

“graphicness” (this aspect is implicitly present even in Cohn’s visual language), which is the

most obvious property of signs used in this medium. At the same time, all definitions

acknowledge, more or less explicitly, that there are various types of graphic signs. Let us take

a closer look at the recurring category of a linguistic sign. At this point, a crucial distinction

3

between language and linguistic expression should be made. The distinction dates back to

Ferdinand de Saussure and his ground-breaking Course in General Linguistic (1966 [1916]).

The linguist differentiates language (langue, i.e. the conceptual system of signs) from

speaking (parole, i.e. the act of producing utterances be means of language). Since language

is conceptual in nature, or as Saussure would say it is “basically psychological” (1966: 15), it

cannot be identified with any physical sign used in actual comics narratives, just like a

sentence cannot be identified with the language used to produce it. Therefore, it is reasonable

to talk about “linguistic elements” (as Duncan and Smith do), “words” (Błażejczyk), or

simply “text” (Harvey), rather than language itself. All of these terms are valid as long as they

refer to “graphic utterances” created by means of a linguistic system.

At this point, certain tension appear in some of the definitions quoted above. If

linguistic elements appear in comics in the form of graphic signs, what is the rationale for

distinguishing linguistic graphic signs from other “graphic elements” in Błażejczyk’s

definition, “pictorial elements” in Duncan and Smith’s, and “images” in Harvey’s? Are

linguistic graphic sign instances of McCloud’s “pictorial images,” or “other images”? The

criterion for these distinctions seems to be “pictoriality,” i.e. the fact that pictorial graphic

signs represent visible entities, as opposed to non-pictorial signs, which represent entities or

concepts that cannot be seen. This interpretation is suggested by McCloud when the scholar

discusses the paradox of speech balloon: even though “[speech] balloons don’t exist in the

same plane of reality as [the] pictures,” in comics they “float about like physical objects”

(2006: 142). By drawing speech balloons more subtly, the artist may suggest that the balloon

“[is] not really here, this is where the picture ends” (2006: 142). If “the picture ends” when

the entity represented in the image is not visible in reality, pictoriality seems to consist in

representing visible object with visual signs. In more technical terms, pictoriality is

4

synonymous with iconicity, mentioned explicitly in Toussaint’s icono-linguistic unity.

Iconicity in the context of semiotics is often associated with American philosopher

Charles S. Peirce. Peirce proposed an elaborate taxonomy of sign, out of which only three

types are used extensively in the modern study of signs. The first type is iconic signs, “which

serve to convey idea of the things they represent simply by imitating them” (1998: 5). The

second type is indexical signs, “which show something about things on account of their being

physically connected with them” (1998: 5), but may also “[stand] for its object by virtue of a

real connection with it, or because [they] force the mind to attend to this object” (1998: 14).

The third type is symbolic signs, “which have become associated with their meaning by

usage” (1998: 5), “law” (1998: 292), “habit” (1998: 461), or “rule” (1992: 226). In modern

terminology, iconic signs rely on similarity between the form and the meaning, indexical

signs depend on contiguity (physical or conceptual association) between the form and the

meaning, while symbols are conventional.3

If pictoriality is just another term for iconicity, the definitions of comics proposed by

Toussaint, Duncan and Smith, and Harvey hinge on the dichotomy between iconic signs (in

which the form resembles the meaning), and linguistic signs (which are “products” of

operationalization of the conceptual system of language). As Section 3 will demonstrate, all

variants of icono-linguistic unity are problematic due to the fact that comics involves signs

that cannot be unequivocally classified as either iconic, or linguistic.

Błażejczyk’s definition avoids this problem by employing linguistic signs (“words”)

and a broad umbrella term “graphic elements which are not words.” Błażejczyk is right in

acknowledging the fact that within comics even linguistic signs are graphic, but why should

one type graphic signs in particular be mentioned explicitly in his definition? Distinguishing

words, rather than other kinds of signs, seems somewhat arbitrary and unjustified. Why not

5

distinguish between, for example, iconic signs and graphic elements which are not iconic?

The fact that linguistic signs enjoy a privileged status in Błażejczyk’s definition suggests that

the author is committed to a weak version of icono-linguistic unity. The weak version does

not explicitly mention the dichotomy of iconicity and linguage, but it does organize the

semiotic repertoire of comics into graphic linguistic signs and graphic non-linguistic signs.

This approach is devoid of the shortcomings of the strong version advocated by Toussaint,

Duncan and Smith, and Harvey, since it covers signs which are neither linguistic, nor iconic,

but nonetheless it lacks descriptive accuracy. The semiotic complexity of comics cannot be

described comprehensively, let alone exhaustively, in terms of the all-embracing category

“graphic sign” and its special subclass “linguistic sign.”

McCloud’s definition, which features iconic (“pictorial”) images and “other images,”

is even more cautious than Błażejczyk’s, because it eliminates the category of language

altogether. Consequently, McCloud does not fall into the trap of assigning a special status to

linguistic signs, but unfortunately he privileges iconic signs instead. While the significance of

this type of signs in comics cannot be denied, the semiotic richness of the medium is not

reducible to a dichotomy and interaction between any two categories. McCloud’s definition is

therefore as inaccurate and incomplete as Błażejczyk’s.

We are now left with the most original approach: Neil Cohn’s visual language. The

theory focuses on psychosemiotic mechanisms of comics comprehension. Cohn argues that

language is “a collection of mental patterns organizing the expression of concepts for

expression in some sensorial modality” (2008: 4). It is somewhat controversial to what extent

the metaphor of language describes the rules of creating and interpreting comics,4 but if

limitations of the metaphor are borne in mind, Cohn’s enterprise is indeed impressive and

promising. The author emphasizes the necessity of providing a formal description of the

6

structure of visual language, which could be called “visual grammar,” which is a somewhat

mechanical system of rules or templates for organizing graphic signs on a two-dimensional

surface. However, relatively little attention is devoted to the symbolic nature of a graphic

sign. Even though in “Japanese Visual Language” Cohn mentions Peirce’s typology, he fails

to discuss in more detail how signs in comics are analyzed in terms of this typology. In other

words, the definitions by Toussaint, Duncan and Smith, Harvey, Błażejczyk, and McCloud

explore the cognitive strategies of associating the form and the meaning, which is where

Peirce’s iconicity, indexicality, and conventionality is particularly relevant. Cohn, in turn,

explores systemic ways of relating signs to each other, and therefore this approach is slightly

incompatible the above definitions and the overall goal of this article. For this reason, visual

language will not be a major reference point in the sections to follow.

3. Icono-linguistic fallacy

The key problem of Błażejczyk’s and McCloud’s definitions (“graphic elements” plus

“words,” and “pictorial images” plus “other images” respectively) is the umbrella terms that

fail to account for the cornucopia of signs in comics. This issue will be addressed in detail in

the Section 4, which offers a comprehensive framework allowing for more fine-grained

distinctions. This section focuses on the shortcomings of the strong version of icono-

linguistic unity in the definitions by Toussaint (“iconic signs” plus “linguistic signs”), Duncan

and Smith (“pictorial elements” plus “linguistic elements”), and Harvey (“text” plus

“image”). More specifically, the section analyses three categories of signs that escape this

rigid dichotomy: special lettering, onomatopoeias, and speech balloons.

3.1 Special lettering

I use the term “special lettering” to refer to a text (a graphic representation of an

7

linguistic utterance) which is in some salient and meaningful way different from a “standard”

text used in the story. Two examples of special lettering are shown in Figures 1 and 2.

Fig. 1. Arkham Asylum: A Serious House on Serious Earth (Morrison and McKean,

1989: 21)

Fig. 2. The Sandman: Season of Mists #3 (Gaiman, Jones, and Dringenberg, 2011: 11)

8

Technically, every instance of a grapheme in a text is unique, and no two letters are

identical. Nevertheless, in most cases small, inconspicuous differences are accidental and

hardly ever noticed. Special lettering covers only what Gregory Bateson aphoristically calls

“a difference which makes a difference” (2000: 459), that is a variation which is informative

and meaningful. Bateson comments on the importance of difference in creation of meaning

while discussing the map-making process in the context of information theory:

‘What is it in the territory that gets onto the map?’ … [If] the territory were

uniform, nothing would get onto the map except its boundaries, which are the

points at which it ceases to be uniform against some larger matrix. What gets

onto the map, in fact, is difference, be it a difference in altitude, a difference in

vegetation, a difference in population structure, difference in surface, or

whatever. Differences are the things that get onto a map. (2000: 457)

According to Bateson, maps show differences, because differences constitute information and

are instrumental in creation of meaning:

This piece of paper is different than the wood of this lectern. There are many

differences between them – of colour, texture, shape, etc. … Of this infinitude,

we select a very limited number which become information. In fact, what we

mean by information – the elementary unit of information – is a difference

which makes a difference. (2000: 459)

From a more psychological point of view, one may argue that there is a cognitive pressure in

humans to interpret salient differences as meaningful. Artists from all fields frequently make

use to this conceptual mechanism to in their works.

From this perspective, the differences between Batman’s font and Joker’s font in

Figure 1, as well as Order’s font and Chaos’s font in Figure 2, are salient, informative, and

9

meaningful. The red in Joker’s font (as opposed to the white in Batman’s) is associated, as we

may guess, with blood, which in turn corresponds to the protagonist’s violent behavior; the

“scribbled” texture plus uneven vertical and horizontal alignment denote Joker’s mental

disorder and instability, et cetera.5 Within the framework of icono-linguistic unity, the

question that naturally arises at this point is whether these meaningful aspects of Joker’s font

are iconic or linguistic in nature. It could be argued that there is a highly abstract resemblance

between the disorder in texture and alignment of the font, and the disorder in the psyche of

Joker (i.e. the font “resembles” Joker’s mind on a high level abstraction). This would mean

that the disorder in the font are indeed iconic relative to Joker’s mental illness.6 Redness,

however, is not related with blood and violence through similarity, but through contiguity,

whereby a property of blood is used to prompt the whole concept. Moreover there is a cause-

and-effect relationship between blood and violence, which does not count as iconicity either.

Thus, the color of the font is not iconic, but indexical relative to blood and violence.

The texture, the alignment, and the color of Joker’s font cannot be classified as

linguistic, because these properties of text are considered to be accidental, asemantic, and

non-symbolic. In fact, they are analogous to such extra-linguistic aspects as quality of voice

in speech. Voice quality can be indexical in the sense of indicating something about the

speaker, but it is not used deliberately with the intention of expressing linguistic concepts.

According to Taylor

when indexical signs are generated by animals or humans, the signs are

produced independently of a person’s (or an animal’s) intentions. If you are

drunk, you do not choose to slur your speech – you can’t help it. If you have a

sore throat, you cannot but speak hoarsely. To this extent, indexical aspects of

language are incidental to the conceptual content that the speaker intends to

10

convey. (2000: 49)

Even when Taylor admits that “[the] indexical status of a hoarse voice gets subverted

once I consciously try to communicate something at variance with the sign’s indexical value,”

for instance “when I attempt to put on hoarse voice in order to support the claim that I an

suffering from a throat infection” (2002: 49), the author never claims that such incidental

properties of speech become linguistic in nature. Hence, for Taylor properties like hoarseness

of voice in speech and, by analogy, color, texture, and alignment in writing, fall outside the

field of linguistics.7 Duncan and Smith are equally reluctant to classify these properties as

linguistic. Instead, they use the term “paralanguage” to refer to the aspects of text that convey

“volume, emphasis, rate, vocal quality, etc.” (2009: 145).

Despite the fact that the borderline between the linguistic and the non-linguistic is

fuzzy and difficult to delineate, special lettering can hardly be considered a product of

language alone. Special lettering may involve iconic elements, but some aspects, like the

color of Joker’s font, may display indexicality. This kinds of elements effectively escape the

icono-linguistic unity supported by Toussaint, Duncan and Smith, and Harvey.

3.2 Onomatopoeias

Onomatopoeias do not “fit in” nicely into icono-linguistic dichotomy for slightly

different reasons than special lettering. An example of a comics onomatopoeia is “tok tok

tok” in Figure 3.

11

Fig. 3 Sam Zabel and the Magic Pen (Horrocks, 2011)

Obviously, onomatopoeias in comics should be distinguished from onomatopoeic

words in language, i.e. the words whose phonological form resembles the meaning (e.g.

“screech”). The latter are conventional elements of lexicon, they belong to specific word

classes (“screech” is a verb), and undergo various grammatical processes, like inflection

(“screeches,” “screeched,” “screeching”) and derivation (“screechy,” “screecher,”

“screechiness,” etc.). Furthermore, they are produced by humans (machines capable of

producing computer-generated speech count as highly non-prototypical language users at

best). Even in the extended and metaphorical sense advocated by Cohn, where any system of

rules for expressing syntagmatically complex meanings could pass for a “language” (cf.

Section 2), only animate being can be language users. Comics onomatopoeias, in turn, do not

belong to the lexicon of a language, they are not subject to grammatical rules, and they

symbolize sounds produced by not only humans, but also inanimate objects and animals.

Moreover, comics onomatopoeias do not have to be conventional, and if they do become

conventionalized to some extent (Duncan and Smith mention “the infamous ‘Pow!’ and

12

‘Zowie!’ of the Batman television series” (2009: 156)), this is not a defining feature of this

kind of signs. For all of these reasons, “tok tok tok” in Figure 3 cannot be classified as a

linguistic sign.

Technically, comics onomatopoeias are not purely iconic either. Iconic signs are

founded on similarity, while letters that constitute “tok tok tok” do not resemble the sound of

tapping with a stick against the floor. The lack of iconicity between the graphic form and the

auditory concept follows naturally from Lyons’s observation concerning medium-dependence

of iconicity (cf. Section 1): iconicity typically occurs between entities of the same sensorial

modality. Thus, it is possible to produce a sound which is iconic relative to another sounds, or

a visual image that is iconic relative to a visible entity, but it is much more difficult to

produce a sound that resembles something visible, or an image that resembles a sound.8

Instead, comics onomatopoeias seek to approximate sounds by means of a conventional

writing system. In order to achieve this, a “proxy” sound is created: this sound resembles the

sound simulated by the onomatopoeia and can be represented alphabetically. In Figure 3, the

proxy sound /tɑk tɑk tɑk/9 resembles the sound of tapping and can be rendered alphabetically as

“tok tok tok.”

The differences between an prototypical iconic sign and a comics onomatopoeia is

sketched in Figure 4.

13

Fig. 4 Differences between iconic sign and onomatopoeia

In the figure, the elements of signs are arranged on three planes: semantic (meaning

expressed), phonological (acoustic representation of this meaning), and graphic (visual

representation of the acoustic representation). The broken arrows stand for similarity

relations, pointing from the source of similarity. The solid line between the phonological form

/tɑk tɑk tɑk/ and the graphic form of the onomatopoeia stands for the conventional relation

between linguistic signs and letters that represent them.

The similarity relation between the semantic structure, i.e. the sound expressed by the

onomatopoeia, and the proxy phonological form is an iconic moment in the onomatopoeia.

This proxy sound is not identical with the semantic structure, because it is hardly ever the

case that sounds produced by inanimate objects, animals, etc. are identical with sounds used

14

in speech. The sound /tɑk tɑk tɑk/ is not acoustically the same as the sounds of a stick tapping

against a floor, but they are nonetheless bound together by partial similarity. Analogically, the

relation between the proxy phonological structure and the graphic form can be called an

alphabetic moment, because the writing system of a language is used to render the proxy

sound visually.

Onomatopoeias are also shaped by indexicality. The indexical component is

particularly conspicuous when we take into account how onomatopoeias function inside the

panel. In Figure 3 “tok tok tok” is located close to the source of the sound expressed by the

onomatopoeia. This spatial proximity is meaningful, as it indicates a significant aspect of the

sound, i.e. the source. The proximity, however, cannot be explained in terms of iconicity or

language. Instead, it is an clear instance of what Peirce defines as indexicality: a “real

connection,” which “forces the mind to attend to [the] object” (1998: 14; cf. Section 2),

which in this case is the source of the sound.

Comics onomatopoeias are signs that rely on iconic, indexical, and linguistic (or

better said, alphabetic) moments, but cannot be classified as purely iconic, indexical, or

linguistic. The incomplete framework of icono-linguistic unity mentions only two out of the

three constituents and does not provide tools for describing interrelations between them.

3.3 Speech balloons

“The relationship of word balloons to the drawings that surround them has always

been an uneasy one,” admits Scott McCloud (2006: 142). The scholar asserts that “[speech]

balloons don’t exist in the same plane of reality as [the] pictures” (2006: 142). Toeplitz agrees

that when we look at texts and pictures in comics, “from the first glance we have the

impression that there are two spheres of reality, which are not sufficiently bound together”10

(2006; trans. H.K.). Subjective impressions of the scholars are difficult to debate with, but

15

Toeplitz’s claim that “the meeting of text and image, which takes place in a comics narrative”

is “distant and essentially difficult”11 (both quotations from 1985: 99) seems somewhat

exaggerated. By definition, “[a] sign is a thing which serves to convey knowledge of some

other thing” (Peirce, 1996: 13) which means that the form of a sign is never identical with the

meaning of the sign. If images and texts in comics do belong to different realities, does this

mean that the manner in which an image is not the entity depicted is different from the

manner in which a speech balloon is not the actual utterance? This leads us to a slippery and

dangerous region of metaphysics: in order to solve this puzzle we would need to investigate

the modes of being of different entities and the mode of something not being something else.

This seems to be a false direction. Modern semiotics tends to be a phenomenological

endeavor: it is all about expressing subjective concepts as conceived by humans, rather than

representing objective reality. A more “phenomenological” and semiotics-oriented path is

taken by Will Eisner, for whom speech balloons try to “capture and make visible an ethereal

element: sound” (2008; quoted in McCloud, 2006: 142). This, perhaps, is the source of

tension mentioned by McCloud and Toeplitz. A speech balloon cuts across semiotic

modalities; it represents an acoustic entity by means of a non-acoustic entity. Where does this

place this kind of sign in the icono-linguistic dichotomy?

As already noted in Section 3.2, iconicity is difficult to achieve if the form and the

content belong to different spheres of perception, like in this case. Even though Peirce’s

metaphoric icons (cf. fn. 7) leave some hope for cross-modal iconicity, speech balloons are

not iconic in this way. In fact, even if actual propagation of sound waves were visualized in

the form of air vibrations spreading concentrically in all directions, the visualization would

bear no resemblance, not even highly abstract, to any kind of speech balloons used in comics.

It would be equally difficult to argue that speech balloons are linguistic. A balloon as

16

such does not stand for the content of a linguistic expression; rather, it is a “container” for an

expression represented by the text inside it. In other words, a speech balloon indicates the fact

that an utterance took place, but it does not stand for the content of the utterance. Unlike in

the case of comics onomatopoeias, the link between the form and the meaning does not

involve any linguistic moment. Consequently, speech balloons are neither iconic, nor

linguistic signs, because they lack the conceptual foundation of either type (similarity and

language respectively).

Obviously, this does not mean that semiotics is helpless in explaining this kind of

signs. Firstly, speech balloons tend to be spatially close to the speaker, and the tail of the

balloon indicates her even more explicitly. Both spatial proximity and the pointing tail are

instances of Peirce’s “real connection,” which directs reader’s attention to the speaker in the

story. This means that speech balloons are in fact highly indexical. Secondly, indexicality is

reinforced by the association with puffs of steam that come out of speaker’s mouth while

talking, for instance, on a cold day. This is a possible, although unverifiable, motivation for

the form of a speech balloon and one suggested by McCloud (cf. 2006: 142). If this

association does motivate the form of the sign, the relation between puffs of steam and

utterances is that of spatial and temporal co-occurrence, which is another sort of indexical

real connection.

Iconicity, indexicality, and convention are also responsible for salient and meaningful

differences between speech balloons. Gene Kannenberg, Jr. lists some of the most popular

variations:

For thought, the balloon’s edge is scalloped and the tail is replaced by a series

of small circles, or bubbles. Whispering is indicted either by forming the

balloon with a dashed line or by using smaller-than-normal text. To convey

17

shouting, a balloon’s edge is spiked … similarly, electronic speech can be

simulated by a jagged-edge or geometrically shaped shaped balloon.

(Kannenberg 2001: 174; quoted in Duncan and Smith, 2009: 156)

By the same token, absence of speech balloon around a character’s utterance may be

meaningful, like in the case of Joker in Figure 1. These variations are shaped by different

factors, and a detailed analysis of all variants mentioned by Kannenberg is beyond the scope

of this article. Suffice it to say that most (if not all) of them can be explained in terms of

iconicity, indexicality, and conventionality.

Apart from a strong indexical component, speech balloons are also conventional.

There is no contradiction in claiming that a sign dovetails conventionality and indexicality.

We have already seen a similar situation in onomatopoeias (Section 3.2), which involve both

similarity and conventional relations between sounds, proxy phonological forms, and letters

of an alphabet. Here, speech balloons are conventionalized indices, because, in the words of

Peirce, the “have become associated with their meaning by usage” and “habit” on the part of

comics readers (cf. Section 2).

4. Semiotic continuity of comics

A comprehensive framework for describing the semiotic complexity of comics should

be capable of explaining the properties of the signs discusses in Section 3 and overcome the

explanatory limitations of icono-linguistic unity. It should also avoid umbrella terms like

“graphic elements” or “other images,” because they conflate crucial distinctions between

different types of signs. In particular, traditional approaches systematically fail to recognize

the role if indexicality. At the same time, the alternative approach should not reduce the

semiotic repertoire of the medium to a clear-cut, straight-forward typology, which may be

desirable form the methodological point of view, but fails to recognize the complex nature of

18

signs in comics. For instance, it may be tempting to classify various elements of comics as

pure icons, pure indices, and pure conventional signs, but special lettering, onomatopoeias,

and speech balloons (as well as other elements not covered in this article) are not

unambiguous instances of these classes. The alternative framework should therefore

acknowledge mixture of various properties within one signs.

Writings of Charles S. Peirce confirm that the categories proposed by the philosopher

are not mutually exclusive and the typology makes allowances for such composite signs. For

example, Peirce believes that maps are never fully iconic, because in order to be useful, they

have to include indexical elements like “a mark of a known locality, and the scale of miles,

and the points of compass” (1998: 8). Consequently, the “experience of the world we live in

renders the map something more than a mere icon and confers upon it the added characters of

an index” (1998: 8; original emphasis). Peirce also describes a “perfect sign” as one in which

“[iconic, indexical and symbolic characters] are blended as equally as possible” (1932: 172;

quoted in Łozowski, 2006: 139). The quotations suggest that signs should be viewed not as

clearly delineated types, but as multifaceted constructs displaying iconic, indexical, and

conventional properties in varying proportions. This is the main reason for naming the

alternative framework “semiotic continuity;” the term suggests a seamless, yet structured

combination of iconicity, indexicality, and conventionality instead of a uniformity of two

constituents implied by “icono-linguistic unity.”

Essentially, an analysis in terms of semiotic continuity consists in unraveling each of

the three properties that may constitute a sign (iconicity, indexicality, conventionality) and

describing interactions between them. This is not to say that all properties are evident in all

signs; for instance, speech balloons do not seem to have the iconic component. Nonetheless, a

comprehensive analysis should account for all three properties, even if only by recognizing

19

the absence of one of them. Describing the interaction between the three components

amounts to mapping out the structure of the symbolic link between the form and the meaning.

If an iconic moment is discovered, the question that we may ask is “what resembles what?” If

an indexical moment is recognized, the question is “what is associated with what?” The

subsections below offer analyses of the signs from Section 3 which follow these guidelines.

Some elements of the alternative approach where already mentioned in the case studies, but

case studies in the following subsections are more systematic and comprehensive.

4.1 Joker’s font and speech balloon

The network of semiotic relations that motivate Joker’s font in Figure 1 is sketched in

Figure 5.

Fig. 5 Semiotic relations in Joker’s font

The box with “You heard him, folks! Hit the trail!” represents the linguistic

component, i.e. the meaning conveyed by language. The solid line that connects it to the box

with the graphic sign stands for the conventional relation between words and their meanings.

20

Two broken arrows stand for relationships of iconicity. On the one hand, there is a highly

abstract metaphoric similarity between lack of order in the texture and alignment of the text,

and Joker’s mental illness. On the other hand, there is a metaphoric similarity (broken arrow)

between the fact that Joker is no longer confined in the asylum and the lack of speech balloon

around his words (the abstract concept of lack of confinement is the common denominator of

these elements).12 The solid arrow between the color of the font and blood stands for

indexical relation, since there is a real connection between redness and blood, i.e. a property

and the entity that displays this property. The concepts of blood and disorder are indexically

associated with the concept of psychopathic violence, which, presumably, is the general

notion meant to be expressed by the font.

As already mention, this kind of analysis does not seek to answer the question “Is

Joker’s font iconic, indexical, or a conventional?”, since it does not try to reduce the sign to

any particular class. Instead, the analysis tries to answer the question “How do iconicity,

indexicality, and conventionality connect the form to the meaning?” In other words, the goal

of the analysis is to map out the components of the symbolic relation behind the sign.

4.2 “Tok tok tok”

The network of motivating relations in the onomatopoeia “tok tok tok” in Figure 3 is

sketched in Figure 6.

21

Fig. 6 Semiotic relations in “tok tok tok”

Essentially, the onomatopoeia is shaped by four factors. Firstly, let us integrate Figure

4 with the overall network. The quality of the sound of tapping is approximated by the proxy

phonological form /tɑk tɑk tɑk/; these elements are linked by similarity. Since /tɑk tɑk tɑk/

consists of sounds belonging to a linguistic system, it can be represented by conventional

letters corresponding to these sounds; the conventional link is shown as the solid line.

Secondly, the location of “tok tok tok” in the panel is close to the source of the sound; this

indexical relation is represented by the solid arrow. Thirdly, these are two similarity relations

that shape two aspect of the onomatopoeia. Both of these relations are instances of what

Dirven and Verspoor call “iconicity of quantity,” where a quantitive aspect of a concept

(intensity, size, number of repetitions, etc.) is projected on a quantitive aspect of the form

22

used to symbolize the concept (size, number of elements, etc.) (cf. 2004: 8). In this case, the

number of repetitions of “tok” corresponds to the number of taps, and the size of the

onomatopoeia corresponds to relative loudness of the sound. Comparing the size of a “tok” to

the size of the words in the speech balloon, we may infer that the taps are about as loud as the

words spoken by the protagonists and quieter than the word “SILENCE!”, which was

probably shouted.

Similarly to the analysis of Joker’s font and speech balloon from the previous

subsection, the this analysis does not seek to reduce the onomatopoeia to an iconic sign. Even

though iconic moments are numerous in “tok tok tok,” the indexical and conventional

constituents should not be ignored, because they are crucial in how the sign expresses the

concept behind it.

4.3 Speech balloon

Our analysis involves a generic speech balloon, and all conclusions will be valid a

prototypical sign of this type. Nonetheless, it should be borne in mind that speech balloons

may be modified in many ways. If the modifications are salient, they probably carry semantic

content, and the analysis should be extended to incorporate them. Several variants are listed

by Kannenberg quoted in section 3.3, and another example (absence of speech balloon around

Joker’s text in Figure 1) is covered in Section 3.1.

The relations motivating a generic speech balloon is sketched in Figure 7.

23

Fig. 7 Semiotic relations in a speech balloon

The figure includes the association with puffs of steam coming from speaker’s mouth,

which counts as an instance of indexical “real connection.” The network also contains spatial

proximity and balloon’s tail, both of which explicitly “point to” the speaker, and are therefore

indexical. Nevertheless, despite the three indexical connections, a speech balloon is not a

pure index. Proximity and tail are aspects of a speech balloon that help readers to associate

the balloon with a character speaker, but they do not explicitly indicate that the signs stands

for something that the character said. This meaning is suggested more overtly by the

association with puffs of steam, but the association it may not be obvious for all readers, and

it is unlikely that the association is evoked every time a reader sees this kind of sign. People

familiar with conventions used in comics narratives will recognize a speech balloon as

something that symbolizes an utterance, even if the sign is “taken out” of the panel and drawn

on an empty sheet of paper, where no indexical prompts guide associations with a protagonist

in the story. This suggests that a speech balloon is recognized as a conventional sign of an

utterance within comics medium, and the indexical aspects merely help to related the

24

utterance with the speaker. This conventional aspects is visualized as the solid line from the

box UTTERANCE OR THOUGHT.

5. Conclusion

The aim of this article is to outline the main assumptions underlying the framework of

semiotic continuity in comics and to demonstrate its strengths compared to different variants

of icono-linguistic unity. At this point, the formulation of the paradigm is neither exhaustive,

nor definitive, and several matters require further investigation. Firstly, for the sake of

completeness, other elements of the semiotic environment of comics should be analyzed

within the alternative framework. Speedlines, arrows, visual metaphors and metonymies,

shape and layout of panels, and any other symbolic (e.g. meaningful) elements deserve a

more details study. Also, more attention should be given to what Duncan and Smith call

“hermeneutic images,” which “are not not part of the world of the story, but instead comment

on the story and influence how readers interpret it” (2009: 155). Hermeneutic images are

juxtaposed with diegetic images, which “are used to depict the world of the story” (ibid.).

Examples include are the captions “Order:” and “Chaos:” in Figure 2. What is the place of

hermeneutic images within the framework of semiotic continuity? Can this type of signs be

described in terms of iconicity, indexicality, and conventionality? Are there any significant

differences between hermeneutic and diegetic signs?

Secondly, the analyses conducted in this article focus on explaining various cognitive

strategies that forge the symbolic link between the form and the meaning of a sign, but the

interaction between these strategies is discussed tentatively. We do know, for instance, that

the indexical association between puffs of steam and speech balloons can be

conventionalized, and that comics onomatopoeias involve iconic and conventional

connections between the sound, the proxy phonological form, and the graphic sign. What is

25

not clear is how the content of Joker’s utterance in Figure 1 and the concepts of blood,

violence, and disorder merge seamlessly into a uniform sign, even though Figure 5 suggests

several discrete associations. Are there any rules and restrictions specifying how iconic,

indexical, and conventional properties blend into a single structure? Is the process chaotic and

purely opportunistic, or does it involve hidden patterns and regularities?

The quest for answers to these questions is somewhat similar to Neil Cohn’s effort to

define the grammar of visual language (cf. Section 1). However, the “grammar” of semiotic

continuity would have to describe the mechanisms of sign formation from a more semantic

perspective; it should specify how various strategies of linking the form to the meaning

(iconicity vs. indexicality vs. conventionality) dovetail to produce a coherent whole.13 The

grammar of visual language is more form-oriented: it defines how various formal elements of

the comics medium (like panels) are combined and concatenated to give rise to a coherent

narrative.

The organic interaction between different types of signs is a fundamental assumption

underlying the framework of semiotic continuity. Peirce believes that this kind of interaction

is essential to all processes of interpretation, and captures it in a simile:

[Though] a man may be said to be composed of living tissues, yet portions of

his nails, teeth hair, and bones, which are most necessary to him, have ceased

to undergo the metabolic processes which constitute life, and there are liquids

in his body which are not alive. Now, we may like the indices … to the hard

parts of the body, and the likenesses [i.e. icons] … to the blood: the one holds

us stiffly up to the realities, the other with its swift changes supplies the

nutrient for the main body of thought. (1998: 10)

26

Endnotes

1 I use the term “comics” in order to denote a medium or a semiotic mode of storytelling. Graphic novels and

comic books, despite many differences between these genres, are not discussed separately, because they function

within the same medium.

2 “[Komiks to] seria statycznych obrazków ułożonych obok siebie, stanowiących spójną całość narracyjną i

znaczeniową, której głównym składnikiem są elementy graficzne nie będące słowami, aczkolwiek tekst może w

niej odgrywać istotną rolę.”

3 Unless stated otherwise, I will use the term “conventional sign” to refer to Peirce’s symbols, and reserve the

term “symbol” to what Langacker defines as “the pairing between a semantic structure and a phonological

structure, such that one is able to evoke the other” (2008: 5). It is worth noting that Langacker’s definition does

not stipulate whether the pairing is constituted by iconicity, indexicality, or convention.

4 This definition covers verbal languages and Cohn’s visual language, but also architecture, industrial design,

film-making, computer games and computer interface design, mathematical equations, road signs, fine arts, etc.

If Cohn’s definition of language is to be brought to its logical conclusion, the entirety of human culture would

be a kind of language, and linguistics would have to effectively replace general semiotics and anthropology.

Cohn downplays the fact that even though in speech, fine arts, computer games, etc. signs are combined into

complex structures in a rule-governed fashion, in each of these fields concepts are expressed differently through

idiosyncratic strategies of symbolization. These strategies require a different descriptive framework, which

cannot be seen as mere extension of linguistics.

5 A more detailed and formalized analysis of the figures will be shown in Section 4.1.

6 Peirce proposed a subdivision of iconic signs in which a sign based on such a highly abstract similarity is

called “metaphor”: “[iconic signs] which partake the simple qualities … are images; those which represent the

relations … of the parts of one thing by analogous relations in their own parts, are diagrams; those which

represent the representative character of the [concept] by representing a parallelism in something else, are

metaphors” (1998: 274; original emphasis).

7 For completeness, it should be mentioned that linguists recognize the problems with determining the

“linguisticness” of various aspects of an utterance. Langacker, whose discussion is limited to speech,

demonstrates the difficulties in defining the phonological form of a linguistic sign: “Interpreting it broadly

enough to include any kind of sound implies that even ringing of a dinner bell is a linguistic symbol to be

27

described in the grammar of a language. To avoid this anomaly, we can require that the sounds in question be

produced by the human vocal apparatus. … But … what about clicks that some people use to direct a horse or

call a cat?” (1989: 61).

8 Iconicity across modalities can be sometimes achieved via more abstract diagrammatic and metaphoric

similarities (cf. fn. 7). There is evidence that some aspects of onomatopoeias are shaped by this kind of abstract

iconicity, but this does not apply to the quality of sound itself. For a more detailed analysis, see Section 4.2.

9 I use International Phonetic Alphabet to transcribe the proxy sound in order to emphasize its acoustic nature.

10 “Otóż już na pierwszy rzut oka mamy tu wrażenie istnienia dwóch nie dość ze sobą związanych sfer

rzeczywistości.”

11 “Uświadomienie sobie tej oczywistości umysławia nam jednak, jak dalekie i w istocie trudne jest spotkanie

tekstu z rysunkiem, które dokonuje się w ramach opowiadania komiksowego.”

12 Since other characters do have speech balloons around their texts, this can be treated as a salient meaningful

difference in Bateson’s understanding (cf. Section 3.1) and should be included in the analysis.

13 One methodological toolkit that may be used to explain these syntagmatic intricacies is Conceptual Blending

Theory devised by Gilles Fauconnier and Mark Turner (cf. e.g. 2007).

28

References

Bateson, Gregory. 2000. Steps to an Ecology of Mind. Chicago: University of Chicago Press.

Błażejczyk, M. 2001. “Koń jaki jest każdy widzi, czyli o definicji komiksu.” Zeszyty

komiksowe. Available at <http://www.zeszytykomiksowe.org/strona.php?

strona=definicja_kom>. Accessed September 3, 2014.

Cohn, Neil. 2014. “Comics, Linguistics, and Visual Language: The Past and Future of a

Field.” New York: Palgrave Macmillan. Available at

<http://visuallanguagelab.com/P/NC_Comics&Linguistics.pdf>. Accessed September

3, 2014.

Cohn, Neil. 2008. “Japanese Visual Language. The Structure of Manga.” In Manga: The

Essential Reader, edited by. Toni Johnson-Woods. New York: Continuum Books.

Dirven, René, and Marjolijn Verspoor. 2004. Cognitive Exploration of Language and

Linguistics. Amsterdam: John Benjamins.

Duncan, R., and M. J. Smith. 2009. The Power of Comics. History, Form and Culture. New

York: Continuum.

Eisner, Will. 2008. Comics and Sequential Art: Principles and Practices from the Legendary

Cartoonist. New York: W. W. Norton & Company.

Fauconnier, Gilles, and Mark Turner. 2007. “Conceptual Integration Networks.” In The

Cognitive Linguistics Reader, edited by Vyvyan Evans, Jörg Zinken, and Benjamin

Bergen, pp. 360-419. London-Oakland: Equinox.

Gaiman, Neil, Kelley Jones, and Mike Dringenberg. 2011. The Sandman: Season of Mists.

Vol. 3. New York: Vertigo.

Harvey, Robert C. 1994. The Art of the Funnies: An Aesthetic History. Jackson: University

Press of Mississippi.

Horrocks, Dylan. 2011. “Sam Zabel and the Magic Pen.” Available at

<http://hicksvillecomics.com/magicpen/the-magic-pen-page-54/>. Accessed

29

September 3, 2014.

Kannenberg, Gene, Jr. 2001. “Graphic Text, Graphic Context: Interpreting Custom Fonts and

Hands in Contemporary Comics.” In Illuminating Letters: Typography and Literary

Interpretation, edited by Paul Gutjahr and Megan L. Benton, pp. 163–192. Amherst:

University of Massachusetts Press.

Langacker, Ronald W. 2008. Cognitive Grammar. A Basic Introduction. New York: Oxford

University Press.

Langacker, Ronald W. 1989. Foundations of Cognitive Grammar. Vol. 1. Stanford: Stanford

University Press.

Lyons, John. 1977. Semantics. Vol. 1. Cambridge: Cambridge University Press.

Łozowski, Przemysław. 2006. “Podobieństwo jako przejaw niedowolności (niearbitralności)

znaku językowego.” In Kognitywistyka 2. Podobieństwo, edited by Henryk Kardela,

Zbysław Muszyński, and Maciej Rajewski. Kognitywistyka 2. Podobieństwo, pp. 131–

141. Lublin: Wydawnictwo Uniwersytet Marii Curie-Skłodowskiej.

McCloud, Scott. 2006. Making Comics: Storytelling Secrets of Comics, Manga and Graphic

Novels. New York: Harper.

McCloud, Scott. 1994. Understanding Comics: The Invisible Art. New York: William

Morrow Paperbacks.

Morrison, Grant, and Dave McKean. 1989. Arkham Asylum: A Serious House on Serious

Earth. New York: DC Comics.

Peirce, Charles S. 1932. Collected Papers of Charles Sanders Peirce 2. Elements of Logic.

Cambridge: Harvard University Press.

Peirce, Charles S. 1992. The Essential Peirce. Selected Philosophical Writings. Vol. 1.

Bloomington and Indianapolis: Indiana University Press.

Peirce, Charles S. 1998. The Essential Peirce. Selected Philosophical Writings. Vol. 2.

Bloomington and Indianapolis: Indiana University Press.

Saussure, Ferdinand de. 1966 [1916]. Course in General Linguistics, translated by Wade

30

Baskin. New York, Toronto and London: McGraw-Hill.

Taylor, John R. 2002. Cognitive Grammar. Oxford: Oxford University Press.

Toeplitz, Krzysztof Teodor. 1985. Sztuka komiksu: próba definicji nowego gatunku

artystycznego. Warszawa: Czytelnik.

Toussaint, Bernard. 1976. “Idéographie et bande desinée.” In Communications 24, pp. 81–93.

31