Semiotic approach to mobile screen communication: How we are instigated to voluntarily make descisions

Oscar Bastiaens. Master in Media Innovation, NHTV Breda University of Applied Sciences. July 5th 2013.

2

3

Semiotic approach to mobile screen communication: how we are instigated to voluntarily make decisions. Master thesis O.S.P. Bastiaens, student at NHTV Breda University of Applied Sciences, Master in Media Innovation. Supervisor, Associate Professor Hans Bouwknegt

July 2013

4

PREFACE After graduating from by bachelor I thought I’ve found the subject of my dreams;

transmedia storytelling. However, this master has provided me with so much more

inspiring theories, thoughts and lecturers, that this could only get better. Now, in my line

of interests, semiotics has managed to take a high position and helped me to deepen the

knowledge, insight and understanding of many theories, structures and narratives that

I’ve researched before. For this, I owe Hans Bouwknegt a great deal of thanks.

Not only has he introduced me in this line of theory, he has been supervising me

while writing this thesis and challenged me each and every time to think again

and go beyond the obvious and aesthetics. I couldn’t have wished for a better,

and fore mostly inspiring, supervisor for these last weeks.

Besides Hans I would also like to thank my very good friend and business

partner Joris Niessen for all his time, thoughts and ideas that all helped getting

my mind straight.

John van den Elst, a big thanks for helping me with all my other documents and

remaining realistic in projections and ideas.

My mother and father, Karen Rokx and Sjef Bastiaens who have listened to my

endless gibberish on semiotics and signs, thanks for all those hours willing to

listen to me.

My fellow classmates with whom I’ve had a fantastic year and all our

brainstorming sessions to get inspired once more and of course all our leisure

time well spent.

Of course Ling Teo for helping me out on marking up the document.

Finally many thanks to all of those who I’ve forgotten to mention here.

5

MANAGEMENT SUMMARY

Whereas users nowadays seem to be glued to their mobile devices and in particular

smartphones, the common communication has become more visual than before. This

thesis deals with the question of how in-application use can be steered by means of

symbolic icons and with that, indicate how a preferred reading comes about.

Furthermore it researches how these are best encoded and in return decoded by the

user. Semiotics offers a broad pallet of theories and structures that are able to strip off

all excesses of aesthetic abundance and grasp the essential elements that make up for

the successes and failures, therefore this line of research will be primary for this

research.

First of all the symbol and icon are defined in order to understand and create a larger

comprehension on their operation. Peirce (2006) here has already defined a large part

of these terms as he defined later on in his description of semiosis. However, by

combining this theory with the connotative and denotative features as described by

Stuart Hall (1973) and therefore incorporating the encoding and decoding in

formulating the correct definition, the conclusion can be made that symbols are those

visuals that are connotative, dynamic objects, whereas icons are denotative immediate

visual objects.

In order to understand how these symbols and icons are processed by the human brain

a thorough research is needed into the processing stages the user is experiencing,

consciously as unconsciously. This thesis finds that the slight use of symbols in the

visual expressions used is due to the fact that the recognition and understanding of the

complete message is made easier for the user. Since the combination of both symbol and

icon is common, a proposition to further refer to symbolic icons instead of symbols and

icons is presented.

When the entirety of symbolic icon definitions and processing has been reviewed, this

thesis examines the navigational aspects of mobile application usage. For this the

Saussurian model of associative relations is used to demonstrate how the linear,

syntagmatic reading comes about. However, this model is not concerned with the digital

implications yet, therefore Bouwknegt’s (2011) models of digital associations is used to

generate a larger understanding and ensure a valuable addition to the existing model of

Saussure.

6

The profundity axis is added to illustrate the fact that each sign, and thus symbolic icon,

in itself contains a subliminal reference to what lies beyond. Furthermore, does this new

model contain a so-called genetic indication for applications, which illustrates their

freedom of choice, and therefore influences the process of reaching the preferred

reading.

Finally, the findings are practically illustrated by means of two case studies,

demonstrating the effectiveness of the new model in current mobile screen

communication and narrative.

7

TABLE OF CONTENTS

PREFACE 4

MANAGEMENT SUMMARY 5

INTRODUCTION 8

CHAPTER ONE – DEFINING SYMBOL AND ICON 11 THE THREE TYPES OF SIGN VEHICLES 12 CONNOTATIVE AND DENOTATIVE 15

CHAPTER TWO: SYMBOL AND ICON PROCESSING 18 HEURISTIC-SYSTEMATIC MODEL OF INFORMATION PROCESSING 18 LIMITED CAPACITY MODEL 20

CHAPTER THREE – ASSOCIATIVE RELATIONS 22 PARADIGMATIC RELATIONS 23 SYNTAGMATIC RELATIONS 24

CHAPTER FOUR – MODEL CONTEMPLATION 26 A DATABASE OF POSSIBILITIES 26 ADDITION TO THE MODEL 29 STATIC VS. DYNAMIC 33

CHAPTER FIVE – MODEL BROUGHT TO PRACTICE 35 CASE STUDY – NARRATIVES 35 BUILDING A TRANSMEDIA WORLD 36 CASE STUDY – APPLICATION 40

CHAPTER SIX - CONCLUSION 41

LITERATURE 42

8

INTRODUCTION

Our mobile devices have become like an extension of our hands. The immense growth in

possibilities the device has given to users all around the globe ensured an almost

irreplaceable status. Texting to whomever we want to, wherever we want to in

whatever time suits us best. In the past centuries communication by means of letters

and words have been standard. However, many brands and (user interface) designers

all over the world have noticed the power of recognition by means of logos, shapes,

symbols and icons by the consumers. It is because of this notion that perhaps now, more

ever than before, we need to consider the universal language of symbols and icons and

allow them to take a greater part in our communication, especially from system to user

communication. Figure 1 – Iphone home screen

Nowadays there seems to be a habit in mobile phone usage.

Because of the growth in not only storage capacity, but as

well the processor capabilities, and overall performance, it

has nearly replaced many traditional media devices and

captured it in one, single device. In this time where screens

are getting smaller and information more important and

significant to communicate, symbols and icons have become

more valuable in communicating, thus making as much use of

the already decreasing screen space (figure 1). We can

already see that symbolic and iconic communication is

happening throughout human computer interaction,

especially when it comes to mobile screen communication. Subtle and carefully

considered design choices have to be made to ensure the best possible first interaction

with the user by means of these symbols and icons. But what elements make up for

these symbols and icons, what do they constitute, and what generates their

recognizability? Examining Hall’s theory on ending and decoding (1973) and looking at

Eco’s semiotic definition of signs and their meaning (1981) will prove to generate a

broader, more general understanding and definition of the elements that make up

symbols and icons.

In light of these events, the entirety of the complete design of applications cannot be

ignored. The process is enormous, where the users have become accustomed to a simple

intuitive design; the actual back ends have grown to be increasingly complicated

9

combining databases with each other to provide them with recommendations, news,

and all other information to their liking. This will be investigated when the importance

of a overarching-design is discussed. Moreover, the need for a preferred reading, as well

as retaining the sense of freedom of choice has become increasingly important to

application developers. Users have grown used to a very intuitive and natural design

and therefore usage of these devices. But how has this usage actually become natural

and intuitive? Is this claim of intuitive design due to common habit, or something

completely different? The flow of choice and usage seems to have some similarities with

the associative relations model by Saussure (2006)

When it comes to mobile applications, the symbol or icon is the first impression of the

application in the App Store or Google Play Store, standing out is enormously important.

However, with the same symbol or icon, the brand creates certain expectation that have

to be met in the design choices afterwards when the application is opened and used.

Many applications that design their small icons or symbols seem to look alike in design

choices such as shapes and the amount of intuitive thought left for the user. How these

choices are made, and on what basis, will be discussed. With the increasing complexity

of encoding symbols and icons and their exact features and attributes, there seems to be

no firm general understanding in what creates the exact distinction between all the

available symbols and icons. Peirce’s works and semiosis as described by Atkin (2006)

will make sure this will become completely comprehensible. Semiotics offers a broad

pallet of theories and structures that are able to strip off all excesses of aesthetic

abundance and grasp the essential elements that make up for the successes and failures.

Because of certain limitations, this thesis will focus on the actual encoding and decoding

processes of the symbols and icons to illustrate the influence in decision-making

processes. This is even though there are many more design decisions to be made when it

comes the application development and communication.

The human memory has a large influence in the perception and understanding of the

incoming information on signs, symbols and icons. Here, prior gained knowledge is

accessed and associations are made to gain a full comprehension of meaning. When

users are confronted with many options and information in one screen, these

associative meanings influence understanding and the decision-making process, some

models from communication studies like those by Lang (2000) and Chaiken and Eagly

(1989) have already pointed out the large influential effects of prior knowledge.

10

Therefore this thesis will be concerned with the following main question: How can we

instigate in-app intuitive choices during usage to ensure the highest desired preferred

reading? The usage of symbols and icons help illustrate the great influence caused by

them. In order to gain insight in the encoding process and establishment of symbols and

icons, this thesis aims to give suggestions to improve the design of these exact symbols

and icons.

One of the important assumptions made is that a macro design framework always

influences the output made by many designers on micro, or user, level. This thesis

intends to lay out a semiotic framework to enhance the understanding of the actual

realization of design choices, and the underlying meaning of each main symbol and icon

applicable to mobile screen communication.

In order to maintain the applicability of the gathered information and research, a media

innovation concept that concerns a mobile application with a focus on data collection

will be used a guideline. By means of polling the opinions of users, the application is not

only enriching existing profiles with actual real-time user-subjective information but it

also increases the opportunity to enhance the prediction of trends and the likability of

everyday items, events, people, politics or even the predicted trends. These days data is

king, and the access to it is getting simpler each day. The vast amount that is already

published by individuals themselves online is for the taking, but consumers have

become more aware of the dangers of publicly displaying their interests and so-called

likes. A great idea to gather this data with consent and interest by this same consumer

would be polling opinions on daily matters proposed by the user and companies,

ranging from general and regional polls. By analyzing all the information trends and

opinions, upcoming trends are monitored and predicted. Maintaining a threshold that is

as low as possible will ensure quick and easy access and above all, participation by this

consumer.

As mentioned before, symbols and icons make up an enormous part for the mobile

screen communication, as seen in figure 1, and the increasing complexity has made it

more difficult to get a general understanding on what exactly a symbol and icon are. In

order to understand how these screens are built, analyzing the symbols and icons used

and providing a semiotic argumentation will lead to a better understanding of what is

happening on the mobile screens.

11

CHAPTER ONE – DEFINING SYMBOL AND ICON

Even though it might seem fairly easy to define symbols and icons, when looking into

the semiotic definition of these, it is very likely to generate great confusion. All in all,

does semiotics describe a text as being a string of signs, either letters or symbols and

icons? Umberto Eco already stated that text can be used for multiple purposes, however

the interpretation of the text is being forced upon the readers to analyze and interpret.

These interpretations in return are multiple, if not infinite. Moreover, is the interpretive

operation of a text by no means indefinite but rather imposed by the semiotic strategies

used in the text? (1981: p36) Here the importance of a deepened semiotic analysis of

symbols and icons is already stressed because of the immense complexity that each sign

has been given throughout the years and now has come to a more definite definition.

Atkin (2006) describes Peirce’s process from his early to his final works on the theory of

signs. In this research Atkin describes Peirce’s three elements of sign and

understanding. First of all there are sign vehicles, second objects and finally

interpretants - even within these elements a distinction is made. In later works Peirce

came to the conclusion the parts making up the theory of signs could be defined even

more: all of this theory has been found in a correspondence of Peirce with mostly Lady

Welby, who seemed to have a same particular interest

in signs and their attributes. It is these final works that

give a strong all-round understanding of symbols and

icons in semiotic use. In order to fully comprehend the

creation of the final theory of signs we have to look

back from where the ideas stem, and this would be

Peirce’s earlier works. His triad (figure 2), which marks

the features of a sign, is essential in understanding the

Figure 2 – Peirce’s semiotic triad elements that make up a sign. This triad refutes three

important aspects when it comes to elements of a sign. The representamen refers to the

form that the sign takes, this could be anything ranging from a letter, word or even

image, whereas the object is the actual reference of the representamen - this part is also

known as the sign vehicle. Finally the interpretant is the final interpretation made by the

reader. However, this interpretant is largely influenced by the reader himself as Eco

described. All three elements are necessary for a sign to exist and be meaningful in its

12

communication. But where do all these elements come from and what creates this

interdependency between them?

The symbol and icons used in mobile screen communication are complex because of

their dynamic component; it is almost merely a one-way communication. The

complexity calls for a thorough research and examination of key elements in the theory.

In order to be able to do justice to the organization and consistency of Peirce’s work, it is

best to follow the line of a close reading and adapt parts of Atkin’s writing without

intervening with comments.

The three types of sign vehicles

The sign vehicle is defined as a particular signifying element of a qualified sign. In

Peirce’s earlier work he described the three elements to be a (1) qualisign, (2) sinsign

and (3) legising. (2006: p16-17)

1. A qualisign is any sign whose sign vehicle relies on simple abstracted qualities.

As an example, Atkin refers to the color cards used to decide on an exact color of

paint. Even though the color chip in particular sings multiple elements, like the

fabric, the shape and others there is only one sign of them that matters to the

signifying ability by the color card, which is the color itself.

2. The sinsign is defined as any sign whose sign vehicle relies on existential

connection with its object. This can be seen as, for example, smoke being a sign

for fire, temperature as a sign for fever or molehills as a sign for moles.

3. Finally, the legising concerns any sign whose crucial signifying element is

primarily due to convention, habit or law. Just as traffic lights are a sign of

priority, the sign vehicles here signify in virtue of the conventions surrounding

their use.

Besides these sign vehicles, Peirce notes three different kinds of objects, the (1) icon, (2)

index and (3) symbol. He proclaims that an object determines their sign,: this could also

be seen, as Atkin describes as the placing of constraints or conditions on successful

signification by the object. (2006: p17-19)

1. The icon reflects the qualitative features of the object, such as portraits,

paintings or diagrams.

2. The index is a sign that uses existential or the physical connection between it,

and its object. A pointing finger could be the best example, therefore the natural

and causal signs.

13

3. Finally, symbols are described as those signs that utilize some convention, habit,

social rule or law that connects it with its object like words.

Finally there are the three different types of interpretants being (1) rheme, (2) dicent

and (3) delome. The interpretants aid in the classification of the signs in term of their

relation with the interpretants. (2006: p19-20)

1. The rheme determines the interpretant by focusing on the understanding upon

qualitative features that it employs in signifying its object, here we could of

examples like ‘…is a dog’ or ‘…gives…to…’.

2. Dicent focusses on the actual understanding of the signs on existential features.

One could think of examples like ‘Tanga is a dog’ or ‘Nina gives food to Tanga’.

3. The delome focusses on the understanding of a sign on some conventional or

law-like features employed, like arguments used in discourse.

Finally a list of a total of ten possible combinations between these three main elements

of Peirce’s sign theory were developed, since some combinations yield constraints. If we

would define symbols and icons from these definitions we could state icons to be a

representation of actual, mostly, tangible assets or products. An icon showing the wheel

of a car would therefore be a proper icon if the signified message would be just that.

However, a symbol would then be limited to actual words. We can assume that the time

wherein Peirce was living and writing he did not take mobile screen communication into

account, and all of its communication by means of small, square blocks.

However, in a later period in Peirce’s life he further defined and interprets objects and

interpretants, dismissing the three types and extends the definitions into dynamic and

immediate. Furthermore, the objects now are not necessarily defined as icons or

symbols anymore, but more as semiotic processes that assist in the understanding of the

signage process.

First of all there are the dynamic objects; these generate a chain of signs. The aim of this

chain is to come to a full understanding of an object, attempting to assimilate that object

into the system of signs and therefore the memory. This concerns the object as is really

is or as it is known to be.

As an example Atkin portrays a tank of petroleum that is half full. There are many signs

that indicate the tank is half full, such as the sound of the tank when it is being stroked,

the indication of the fuel gauge, a light on the dashboard and so on. Even though there

14

are all these possible signs, they only add up to the indication. The object that is

underlying all of them is the actual half full petroleum tank, the dynamic object.

The immediate objects on the other hand are those objects what we suppose at any time

within the sign chain. The object is noticed since it is the first time it is used and

interpreted by the recipient, it is not an addition to the dynamic object, but rather some

informational, incomplete equal of the dynamic object generated at some stage in the

chain of signs. Here again, we can use Atkin’s example of the petroleum tank to get a

clear distinction between these two kinds of objects. When, for example, we would

strike the tank we could hear a tone (which then would be the sign vehicle for the

immediate object), which would represent to us that the tank is not full. It does not tell

us the exact amount left within the tank. Then, here, the immediate object is a less-than-

half-tank. (2006: p25-27)

Concluding, we can state that the dynamic object is the end-goal of understanding the

sign chain and semiotic process, where the immediate object is our grasp of that object

in any given time during the sign chain process itself, adding the then gained knowledge

to our general semiotic understanding.

Moreover, Peirce has redefined the interpretants’ roles and divisions. Now (1) the

immediate interpretant, (2) dynamic interpretant and (3) the final interpretant make up

for the interpretants as being a vital part in the theory of signs. All of these new divisions

seem to be strongly influenced by Peirce’s development if the levels of understanding,

where he divided these into three levels, being (i) having an unreflective grasp of some

concept, (ii) having, or being capable of, providing a general definition of the concept

and finally (iii) having conceived the object with one’s own conception. Then, our

conception of these effects is the whole of our conception of the object. (2006: p28-35)

1. The immediate interpretant (linked to ii) provides a general definitional

understanding of the relationship between the sign and dynamic object. When

we use ordinary sentences to explain this we can see that this interpretant will

recognize grammatical categories, syntactic structures and conventional rules of

use. For example: ‘We don’t want to hurt him, do we?’. The immediate

interpretant would find this to be a question, concerning doing harm to a male

person.

2. When it comes to the understanding we reach, or which the sign determines, at

any particular semiotic stage we speak of the dynamic interpretant (linked to i).

15

This refers to the interpretant being aware of the sender, receiver and the

knowledge the sender holds.

3. The final interpretant (linked to iii) can be described best as one that

incorporates a complete and true conception of the objects of the sign. In the

end, this would be the interpretant with which everyone should agree with on

the long run. Usually there is a series of dynamic interpretants’ questions

concerning the actual meaning of the signified.

Therefore, the final interpretant is important in our understanding of the dynamic

object. First of all, it marks the actual point where the grasp of this dynamic object

becomes understandable and complete. Second, it functions as an example of the

normative standard on which the judgment of interpretative responses can be made

concerning signs.

These definitions by Peirce do not yet provide conclusive arguments to completely

define symbols and icons; they remain vague in their understanding of the encoding

process prior to their reading by the user. Therefore, to deepen this insight in the

general definition of signs and therefore symbols and icons, taking a look at Stuart Hall’s

(1973) paper on encoding and decoding will provide conclusive arguments to define

symbols and icons for this thesis.

Connotative and denotative

Stuart Hall has written in his article ‘Encoding, Decoding’ on a theory known as the

reception theory in semiotics. Here an important distinction is made between

denotation, which is the actual representation of the literal meaning of a sign and

connotation, which implies references made towards and deducted from the denotation.

Here already we are able to note the important difference between an intended message

and the decoded message, being an important aspect in discovering how icons and

symbols are comprehended. (1973: p512-513) How these notions and those researched

above come together to one meaningful understanding will be elaborated on below.

Hall also quotes the famous semiotician Roland Barthes’ example to set a clear example

on how the difference between this denotation and connotation works. For example, the

sweater always signifies a ‘warm garment’ (denotation) and thus the activity or value of

‘keeping warm’. But it is also possible, at its more connotative levels, to signify ‘the coming

of winter’ or ‘a cold day’. (1973: p513)

16

The influences from community and culture can be decisive in the comprehension of

symbols and icons in user-system communication. Even though there are some simple

visual signs, as the peace sign, which have achieved maturity to such an extent they are

almost universal. Even though these still were influenced in their development stages by

culture and community.

Even though Hall’s theory is a more textual analysis, it does apply to visual

communication in form of symbols and icons. There is a focus on the scope for

negotiation and opposition on the part of the audience. This means that a "text"—be it a

book, movie, or other creative work—is not simply passively accepted by the audience,

but that the reader/viewer interprets the meanings of the text based on their individual

cultural background and life experiences. In essence, the meaning of a text is not

inherent within the text itself, but is created within the relationship between the text

and the reader.

We could propose two statements on defining symbols and icons: Symbols are those

visuals that are connotative, dynamic objects, whereas icons are denotative immediate

visual objects.

However, Huening (2013) refers to the works of Johansen who researched Peirce’s

works and they notify that even though there is this triad of components, being sign,

index and icon, they do not necessarily make up a sign individually. On the contrary, he

states

‘’Consider a photograph: it has properties in common with its object, and is

therefore an icon; it is directly and physically influenced by its object, and is

therefore an index; and lastly it requires a learned process of "reading" to

understand it, and is therefore a symbol’’

Figure 3 – Runkeeper image

If we would take the Runkeeper’s application image (figure 3), we

can already see there are signs including icon, since there is the

actual representation of someone. The symbolic reference made is

that concerning the probable fact the person is in a running position. The blue color then

again refers to the ignition of certain emotions towards this color. Again, a symbol or

icon does have certain important elements that define them, but they are, almost, never

to be found individually.

17

Looking back at figure one, we do see it is very important to take into account that in the

current mobile media landscape the usage of symbols and icons is virtually always a

combined one. Most used are icons that have a small symbolic addition in order to

create a larger familiarity and recognition. In this perspective the definition of symbol

and icon as described above are still applicable, however, Eco described that ‘each sign

possesses some way certain features, which prescribe its contextual fate.’ (1981: p37)

This context is vital for the mutual interaction between signs and the understanding of

the small symbol or icon used for applications, which is a way, is a subliminal

representation of the application itself. All signs interact with each other based on the

previous intertextual history they have been in contact with.

The determined strength of mobile screen communication via symbols and icons

therefore seems to lie in the storage and retrieval processes of the brain concerning

gained general knowledge. The associations made with every part is due to the fact

there is prior knowledge about them, this prior knowledge in its turn is influenced by

many different factors including culture, user experience, etc. At a certain point some

become a natural process or a heuristic. The heuristic-systematic model of information

processing by Chaiken and Eagly (1989) examines these natural, heuristic processes. In

order to gain a better understanding on how these symbols and icons are decoded by

users, this model will contribute largely to this apprehension.

18

CHAPTER TWO: SYMBOL AND ICON PROCESSING Already, Eco stressed the importance of contextual presence when it comes to the

understanding of signs, ‘be it a word or a visual item such as an directional arrow.’ (1981:

p37) Eco quotes Buyssens to be the first to stress this in the semiotic context; the full

meaning of signs only becomes apparent, more comprehendible and meaningful when it

is placed within a larger context. However, this context is influenced by the earlier

mentioned prior knowledge. Foucault (1966) has created a certain distinction between

different kinds of signs, one being natural and the other, conventional. Where the

natural sign is concerned, he states this sign has a completely individual interpretation,

‘an element selected from the world and constituted as a sign by our knowledge.’ (1966:

60). On the other hand, a conventional sign is developed and established by the human

brain and therefore easier to remember than the strict natural sign.

As all these elements make up for the understanding and the eventual interpretation the

user will have, it is therefore vitaly important to investigate what happens in the human

brain when it decodes such symbolic and iconic messages. In order to illustrate the

process between text and human, taking a step back from semiotics and into

communication literature, the strength of dynamic and immediate objects in sign

interpretation appears to be addressed and highlighted in the theories discussed below.

In a way, symbols and icons are a sublimation of multiple interpretations made by the

mind in memory. The possibilities are, as Eco states, endless, but however limited to the

extent in which the brain has knowledge of the contextual references made.

The heuristic-systematic model is one that proves the influential power of the brain

when it comes to prior knowledge and the judgment of information.

Heuristic-systematic model of information processing

This model initially focuses on how persuasive messages are processed and interpreted

by which exact manner. However, the understanding of symbols and icons do have a

great deal to do with heuristic and systematic evaluation as described by Chaiken and

Eagly (1989).

This model suggests two ways of processing, one of them being heuristic processing.

This uses multiple judgmental rules that are simply certain knowledge obtained in the

past by the user, which have been stored in the memory. The large advantage of

heuristic processing is the fact the user has only to put in a very minimal cognitive

effort, it is a process simply ruled by means of availability, applicability and of course

accessibility. All of these are earlier gathered information stored in the memory, as

19

mentioned before, but as well dependent on the environmental and cultural influences

gained by the user, and at the specific moment of use in time. Of course, availability is

concerned with the actual storage in the memory from previous experiences, while

accessibility applies to the current ability of the user to retrieve the memory and then

use this. Applicability in its turn refers to the relevance of all the recollected information

from the memory to the judgmental task the user has.

Of course there are some factors that could influence heuristic processing on the time of

reading or when the initial information is stored in the memory. The research showed

that messages or information delivered by experts or messages endorsed by others

were accepted as being true or interesting without actually fully processing the

semantic content of the message. As seen, this heuristic view shows a de-emphasis on

detailed information processing. It is this quick processing which is taken into account

by user interface designers when creating symbols and icons to developed those visuals

that take quick, heuristic processing into account to ensure a recognizable and fast

processed symbol or icon.

On the other hand, the polar/dichotomous opposite of heuristic processing is

systematic processing. This is less interesting when it comes to quick recognition and

processing since this focuses on comprehensive and analytic, cognitive processing of

judgment-relevant information. When this processing is chosen the users values source

reliability and message content more than normal, therefore the effect on persuasion,

when the message is appreciated, is greater than normal. The eventual judgment is

therefore relying heavily on in-depth treatment of the relevant information. However, as

mentioned earlier, this thesis is concerned with the symbols and icons on mobile

screens, these are not complete messages to be told, but need to be relevant and

recognizable in only a matter of milliseconds. Therefore this approach is not relevant

enough when actual usage is concerned, however the systematic processing might be

relevant when denotative and connotative signs are interpreted and stored in the

memory for the first time.

Moreover, Chaiken and Eagly (1989) have found that when, as they refer to it, economic

concerns are predominant, which means when users are constrained by time issues for

example and needs to act, read and interpret quickly, users are more likely to employ

heuristic processing when formulating their judgment on the decoding. However, when

it comes to judging the reliability of the signs, users will employ the systematic

20

processing approach since this researches a source into more detail and based on this,

even though quick, research by the user the source is validated as reliable or not.

All in all, in order to ensure the comprehension of symbols and icons, we have to

incorporate logic, and cultural denotative and connotative elements in the development,

or encoding, of these symbols and icons. By means of this approach there is no need for

extensive reading and judging by the user necessary, creating an efficient and logical

communication of meaning.

The reliability on prior knowledge and experiences is vital in decoding mobile screen

communication and interpretation. However, how is this knowledge gathered and how

do users cope with the, sometimes, overwhelming amount of it? The limited capacity

model (LCM) by Lang (2000) examines this process and proposes three main stages of

encoding, storage and retrieval. This theory demonstrates the importance and methods

of each stage. An understanding of each separate stage and the entirety of this process

will help creating a stronger argument for the use and encoding process of symbols and

icons.

Limited capacity model

The three steps that make up the LCM all play a very important part in gaining and

developing the prior knowledge as well as understanding the decoding process the user

experiences. Moreover, it contributes to our comprehension and understanding of how

to create symbols and icons relevant to users.

Firstly, there is the encoding within the model; what is key here is remembering to

create a sign that stimulates the receptors used when using mobile applications, and this

will enhance the associative memory when searching for connotative signs. When the

sign is perceived by the sensory receptors it is held in a sensory store, one that is quite

comparable with the linguistic and perceptual parser in the human associative memory

flowchart by Anderson and Bower (1980). After the selection processes, in this case

probably heuristic or systematic processes (Chaiken and Eagly 1989) stemming from

the eleven steps by Goodman (1967), some approved bits of the information are allowed

to continue to the short term memory.

After the encoding and judging of the signs and/or information we come to the storage

stage. If positive evaluated information is let through to the memory, there is not really a

21

strong division to be made between the short and long term memory, other than that

information stored in the short term memory is information that is much easier

accessed and used. Other terms for these are therefore the working (short term)

memory and the encyclopedic (long term) memory. Lang (2000) does confirm that the

more associations can be made within the entire memory system the more accessible

the memory trace becomes. This also means that not all parts of the message or sign are

stored in the same way.

Where the final stage of retrieval is concerned, Lang (2000) states it is the process of re-

activating a stored mental representation of some aspect of the message. Again here she

stresses the more that associative links that are made in the previous stage, the more

readily retrievable this becomes. Therefore, when the LCM is concerned we have to

build easy comprehensible symbols and icons and relations to these in the first place to

make sure encoding is quickly generated, storage is allowed and retrieval can be based

on many other associations made by the brain to accelerate the retrieval process. This

system of storage and retrieval, but especially retrieval, has been noted in the semiotic

epistemology as being semiosis. This is the process whereby signs are interpreted and

translated into other signs, as seen before in Peirce’s theory. These associations in their

turn provide a more comprehensive understanding of the given text.

With regard to these storage and retrieval processes, we can thus conclude that the

slight use of symbols in the visual expressions used is due to the fact that the recognition

and understanding of the total is made easier for the user. All elements that are involved

in the definition such as, immediate, dynamic, denotative, connotative, symbol, icon and

index make up whether a sign is more symbolic or iconic. However, in order to generate

a more clear and general understanding of reference from this point on, I would like to

propose the term symbolic icons as the reference to the slight symbol and large icon use

in the mobile screen communication.

22

CHAPTER THREE – ASSOCIATIVE RELATIONS

Having defined and examined the symbolic icon encoding and decoding processes for

users as well as designers, and the definition of such processes, one wonders how

exactly in-application choices made by users are influencing the system, and how these

choices can contribute to the most preferred reading. The underlying associative

relations that each symbolic icon and sign holds perform this major influence on user

and therefore the system. Semiotician Ferdinand de Saussure has examined these

relations and it is his model that will provide a more in-depth insight when elaborating

on the navigational aspects of mobile screen communication.

In Saussure’s model there are two key axes portrayed in his model, paradigms and

syntagms. Both show the importance of a coherent and balanced intuitive design. This

chapter examines both aspects and the correlation between them, in accordance with

mobile screen communication and application use.

One important aspect is the concept of paradigms: a paradigm is a set of associated

signifiers or signifieds which are all members of some defining category, but in which

each is significantly different (Chandler 2007). When wanting a full comprehension of

semiotic strings of signs and the mutual relations between the symbolic icon and the

application itself, Saussure’s associative relations cannot be left out. Moreover, his

model of these relations demonstrates the coherence between all separate elements. All

of this will be discussed in this chapter.

The other is the concept of syntagms; the Oxford dictionary defines this as ‘a linguistic

unit consisting of a set of linguistic forms (phonemes, words, or phrases) that are in a

sequential relationship to one another’. (2013) Therefore syntagms concern the position

of single signs formed into, for example, strings of text. When looking into this, perhaps,

we can dissect older forms of semiotics in order to amplify the theory and arrive at a

general scale.

The semiotician Saussure had a particular interest in the relation between the signifier

and the signified: signs versus all other elements of the system, and those between a

sign and the elements which surround it within a concrete signifying instance (Chandler

2007). He described the key differences as being these syntagmatic and paradigmatic

23

forms, and developed a model,that shows the exact implications of these relations and

models.

Figure 4 (Chandler 2007)

As seen in this model (figure 4) the syntagmatic axis

deals with the positioning of the signs, in this case

words, concerning their order. The paradigmatic

axis on the other hand deals with substitution of

signs, also described by Saussure as the associative

relations.

When breaking down these axes to a quite abstract level, we can see that each choice in

paradigmatic design has the ability to differ and change according to personal

interpretation and preference. Whereas the syntagmatic axis provides this earlier

described need for a preferred reading to ensure a logical mental process.

Paradigmatic relations

A paradigm is a term that has come from an elaboration on Saussure’s série associative

(‘associative series’), which is a set of signs linked by partial resemblances, either in

form or in meaning. Saussure described such sets as being established ‘in the memory’

and the item this associated as forming a ‘mnemonic series’ (Cobley, 2001: p233).

Paradigmatic relations are functional contrasts, since they involve differentiation.

This means that the paradigm is concerned with the selection of 'this---or---this---or---

this' (Chandler, 2007: p84). It is about the choice made which media will be consumed at

which particular point in time. One of the strong reasons for users to step out of the

syntagmatic flow, and step into the paradigmatic of mobile screen communication is to

discover more information on the application and its features.

When perceiving paradigm more as associative it seems the emphasis shifts to sets of

signs which relate to one another, again related to the possibilities of substitution in

particular positions. When breaking down the term and effect of paradigm it is not

merely a set of signs linked by resemblances but it offers the creator and user of the text

the ability to change signs and provide his own interpretation. This means a text is no

longer bound to a set of rules concerning narrative development. It provides a large

scale of freedom, as long as the entire text stays in canon.

24

When digging even deeper into paradigms we can see that beyond the point of creating

an individual interpretation, the text is coded in such a way that the user or creator of

the text offers the possibility to design his own significance within the text.

Within this perspective, each choice in paradigmatic design has the ability to differ and

change according to personal interpretation.

Syntagmatic relations

According to the Oxford dictionary, syntagms are defined as ‘a linguistic unit consisting

of a set of linguistic forms (phonemes, words, or phrases) that are in a sequential

relationship to one another’. (2013) This means a syntagm is always worth more than

merely the sum of its parts. It concerns the entire sequence chosen and the (possible)

chronology within this string of text. Syntagmatic relations are those into which a

linguistic unit enters in virtue of its linear concatenation in a speech chain (Cobley,

2001: p273).

Whereas within paradigmatic terms it concerns 'this---or---this---or---this', within

syntagmatic terms it is about the combination of 'this---and---this---and---this'.

(Chandler, 2007: p84) This order creates a clear and understandable linear path and

codes the communication in such a way that users experience certain emotions or

intuitive choices at carefully placed times. Setting this within the perspective of free

choice, it becomes a lot more complicated. Even though the user has the choice to

discover every available medium, the producer has to carefully place enough

motivational cues for them to use, or click exactly where the producer wants them to go.

When amplifying this concept of a string of signs to more than an example with words, it

is possible to apply this to application navigation. In a sense, all application features are

different texts, with each new text making distinctive contribution to the whole.

Moreover, all these text contribute to one large meta-text: the sum of all parts creates a

far larger importance than each individually, even though they are created to be

sufficient and satisfactory when used as only one text or feature.

Another important aspect of syntagms is the (chronological) order. This will provide

motivation for the users to travel across features to constantly discover a new (and

larger) part of the application. Therefore, the importance of this model lies within the

fact that all the elements that make up a sign come from the dynamic and

connected/linked illustration of the sign structures by the literature.

25

Since applications contain quite a number of features and some are more influential

than the others, this thesis will take the symbolic and iconic communication as the

guiding line, in order to understand how certain syntagmatic preferred readings may be

voluntarily forced by means of intuitive paradigmatic options and eventually, choices.

But this model has not taken the innovative steps and developments made by

technology and media currently. Even though this model does seem complete when it

comes to producing and understanding meaningful discourse where the signs are

words, it does not take the digital revolution into account yet. How do current

technological applications, such as databases, take this model into consideration? Could

they provide a more complete and modern associative relations model?

26

CHAPTER FOUR – MODEL CONTEMPLATION Saussure’s model does not seem to be fully complete in the context of current

technological development: in this time where mobile screen communication is

changing behavior and attitudes towards information processing and usage, we have to

take these new innovations into account and re-apply them to this semiotic model of

navigational possibilities. Nowadays when databases are packed with information and

are constantly expanding, there is the possibility of completely rearranging the way we

compose texts. There seems to be infinite possibilities for encoding messages, but the

importance is the actual encoding of those messages by the database, resulting from the

given user input, in order to create valuable discourse with the user.

A database of possibilities

The associative relations model has provided a clear understanding on the creation of

linear and understandable discourse. However, in the digital world, Aarseth (1997)

proposes the idea of cybertext as being a kind of ergodic literature to illustrate the

semiotic framework of a computerized system. The cybertext has a main focus on the

mechanical organization of the destined text: it centers the reader or user of the text as

an almost completely integrated figure. The thing that makes this research and

exploration so interesting is the fact that Aarseth describes a semiotic sequence of the

cybertext in which the user’s input affects this same sequence greatly.

Even though one might think the influence of the user is made up from the sum of

actions and decisions made on an intuitive as well as a conscious level, it is mostly due

to the immense capacity of the computer’s processing speed, and the unlimited access to

a enormous storage of information - databases.

Aarseth has never given much scholarly attention or appreciation to semiotics. Reviews

on his book sometimes even stated that ‘He obliterates semiotic and poststructuralistic

analyses (which are weak more because of their authors than their theory). One problem

with the approaches is that they do not account for emergent behavior (results not

predicted by the designer; i.e., a programmer being beaten by his own chess program).’

(2010) However, his idea concerning these digital texts and their link to database usage

is very interesting for this research. Moreover, when peeling down the definition to the

mere bones, we are able to understand the linkage to the use of signs within these

databases. The symbols in total, form the string of text, sign being digital.

27

Aarseth defines ergodic literature in his book ‘Cybertext’ as the following:

‘In ergodic literature, nontrivial effort is required to allow the reader to traverse

the text. If ergodic literature is to make sense as a concept, there must also be

nonergodic literature, where the effort to traverse the text is trivial, with

no extranoematic responsibilities placed on the reader except (for example) eye

movement and the periodic or arbitrary turning of pages.’ (1997: p1)

Essentially this would mean that ergodic literature, like the cybertext, is that text in

which the user has the mere function of being almost completely immersed, and

therefore true action is required in order to comprehend the text. Whereas the

nonergodic literature stands for the complete opposite, where the user does not need to

provide true input and actions in order to read. However, this would only count for

those consciously-made decisions on actions, and not the natural consequences in

actions when reading a text, such as eye movement.

The contrast created between the importance of this linearity that has to be created by

developers, and the ergodic cybertext that implies no linearity whatsoever, makes a

complete understanding and the designing of applications (options) extremely difficult.

Hjelmslev (1961) noted a very important issue when semiotics is concerned, stating that

for semiotics, texts are chains of signs and therefore are linear by definition. Aarseth

also quotes Jensen (1990: p44)) in respect of computer communication and sees ’the

symbols as strings of binary digits’ (1997: p27). To this he adds that since these digits,

(zeros and ones) are programmed, they could only mean whatever meaning has been

defined to them.

Aarseth, with his ergodic literature, has given a great example of how the computer, or

the system, has provided users with a unifying platform on which the range of

possibilities in different forms of sign expression have increased immensely. Due to the

ease in accessibility of the database via the system, paradigmatic choices are enlarged

immensely, and the associative signs therefore receive a more profound meaning. The

process of constructing texts is changed when set in a digital context: no longer is it a

fixed, static text like the thesis you are reading at this moment. Whereas printed texts

are subject to their definite meaning, digital texts are dynamic and subject to change

whenever the users requests it. All possible recourses have become accessible, but with

it the increase in complexity of encoding the text. It is because of this reason that design

can disclose the amount of paradigmatic options to lead the user through all the possible

28

paradigmatic choices available in the system and its database. The interaction between

user and system is vital for the composition of digital texts: to illustrate the idea of the

impact of these changes clearly, the following example is offered as an illustration:

.

The user can be offered a static text, such as a Word document, but at the same time the

user can replace every letter or sign with whatever input is given, thus changing a static

text to a dynamic one that is not fixed and set to a certain meaning. The software

program Word has been designed in such a way that replacement at the user’s will can

be done easily and with the most logical associative signs, these all being stored within

the program. However, if the user wants to, they have the possibility to access the

infinite database via the Internet and replace any sign with any other given sign.

Therefore, the program or application design does provide a paradigmatic goal of usage

but because of the dynamic and constant accessible database nothing is fixed, only

guided.

Taking this unlimited chain of possible signs, it can be concluded that providing

unlimited possibilities to the user will most probably not give developers any chance of

enforcing in-app usage in order to ensure the most beneficial syntagmatic course. It is

because of this that services and products have been tailor-made to provide for more

specific needs of the consumer, the signified message is becoming increasingly difficult

to encode because of these unlimited options. Therefore, context and a coherent design

will provide the user with a greater sense of intuitive design and choices.

Concluding, we see that the system is a dynamic machine that makes up the string of

signs (code) because of the generated input by the user. This constant importation of

information in return generates a complex and tailor-made code back to the user. But

the databases accessed by the dynamic machine are infinite and constantly expanding

because of the constant import of (new) information by users. It is then due to this that

the options become infinite and the order of signs are no longer fixed; they are

completely dependent on the input by the user. The order ceases to be prescribed by

writers, directors or other content creators. As an example, this thesis is a finite

document which is to be read in the order as prescribed, however when accessing a

system that has unlimited possibilities of which the output depends on the given input,

there is no longer a clear understanding of where text starts or ends cleanly. But the

importance of a linear, syntagmatic reading remains, and options and choices are then

influenced by the context revolving around them.

29

When considering our primary associative relations model, we see that all these

paradigmatic decisions and choices can be unlimited, but are limited by developers to

provide a clear and coherent service or product. The syntagmatic linearity on the other

hand is one that has to be used if developers want to ensure a natural and intuitive

design and choice process for their users.

Addition to the model

In order to understand and convey the exact message of signifier and signified, as

written by Saussure in his book Course in General Linguistics there has to be a coherent

design in choice of application and the symbolic icon used. From this ideology we would

find ourselves in a position wherein design choices become increasingly important to

signify the desired syntagmatic course exactly.

However, how is such a coherent design created? When breaking down to merely three

main components in design we could end up with the (1) symbolic icon design, (2) the

application design and finally the connecting factor, being the (3) overarching design.

(See figure 5)

Figure 5 – Complete Design overview

As can be seen in the figure above, the overall coherent design is dependent on those

choices made at an overarching design level. Examples of these choices are basic colors,

shapes etc. The importance of these choices is vital because designers need to ensure

that the signified message by means of this design is interpreted exactly as intended - if

not, this could lead to a break in the expectation pattern that the user has generated in

his mind. All of this could cause to misunderstanding and misinterpretation, which then

leads to the cessation of use.

At the second level, application design is where the unlimited cybertext and database

choices have to be limited to merely those most beneficial to the user and producer.

Moreover, ensuring the encoding of used signs will enhance the efficiency of usage since

30

the user will make more intuitive choices and therefore become more consistent with

the tenets of ergodic literature.

The final level is the key thing users will come in contact with first when

browsing/searching the App store or the Google Play Store. Here extreme caution in

design choices is required since this actually is a subliminal representation of the entire

design structure and the initial pointof the creation of expectations.

Even though all these choices are made and they are strongly influenced by the

Overarching design behind it, how do we actually make the choice for particular signs in

mobile screen communication? Whatever the sign, or symbolic icon represents is what

makes it attractive to users, or otherwise. The most important aspect is to ensure an

intuitive design in which the signifier is signified exactly as intended to. Therefore the

unity of the entire design is of great importance in generating a coherent message as

intended. One of the important aspects that create a logical and intuitive flow

throughout the application use is the attractiveness of its unity of design and use. All

chosen representational signs have to represent exactly what the Overarching design is,

and therefore indicates how the entire application works.

Where Saussure expresses different kinds of linguistic features, being words, this model

is closer to the one as created/proposed? by Hans Bouwknegt (2011) in his book,

Beyond the Simulacrum. He has already pointed out and researched a translation of the

initial model to one that is concerned with all possible signs, platforms or even

narratives. It is these findings made by Bouwknegt that create a larger understanding of

digital media usage and choice.

Here the database of unlimited signs comes to order (becomes particularly significant?)

once more. Bouwknegt states that the interaction between the user and database is one

of the vital aspects when it comes to the usage and understanding of the database.

Whenever the user provides the system with input, the system has to encode a specific

message. Here is becomes clear that the system is reliable on the input given by the user

and cannot provide meaningful discourse on its own.1 (2011)

11 Bouwknegt makes a specific reference to Baudrillard’s simulacrum. Since the simulacrum is not

applicable for this thesis, but important to understand Bouwknegt’s findings. The simulacrum that states

there is less and less truth, because we base our reality on something that we (mankind) have devised (in

film and media) (1998: p166). An example of a 'simulacrum' is that everyone knows how a crashing

airplane looks like. But we know this because we have seen this on television. The ‘knowledge’ of this image

is not based on our own truth, but the truth is created on TV.

31

This would mean that even though the media can display all kinds of signs that refer to

the state of reality, they could only imply this reality instead of being the reality. In order

to become the actual reality, media needs the unlimited resource to all possible signs

and codes.

By making the user information permanent and apparent, databases and the use of

these databases are improved to a higher level in which producers and developers are

completely aware of the consumption behavior of the users of media.

The model of Bouwknegt as mentioned (see figure 6) above is one that shows ‘whether

signs are coded logically by algorithms or coded culturally. All respective levels are related

and can influence each other’. (2011: p122)

Figure 6 – Bouwknegt’s Network of signs

Even though the axes of syntagms and paradigms are replaced with logic and cultural

references, we can still view this model as one that could have the same purpose. All

individual encoded signs relate to one another in a specific way. The logical, and

therefore syntagmatic process is due to the exact and intuitive choices made in

paradigmatic options provided to the user. Bouwknegt did see there is an additional

level to this already, being the Meta level of the signs and how they correlate (figure 7)

32

Figure 7 – Hierarchy model

It is these relations and correlations that each individual sign has that contribute partly

to the creation of understanding. One could interpret each sign differently, by involving

this hierarchy within every sign; the database can exclude those signs with a high risk of

misinterpretation by means of mis-encoding messages. The codes here cease to remain

superficial and create another deeper layer of meaning when generating strings of signs.

The discussion of Saussure’s and Bouwknegt’s model of associative relations mentioned

before and the paradigmatic and syntagmatic relations could very well use an addition.

As seen before, each symbolic icon refers to known, associative meanings. But there is

more to it; each symbolic icon does not only reference to that exact point, at the same

time it represents the actual application and its (flow of) design. This would

consequently mean an extra axis could be added to Saussure’s initial model. This would

be the profundity axis that stands for the representation of the actual application itself

and not only this reference to

associative meanings. The

model would then look more

like the following.

Figure 8 – Addition to model

33

After this additional feature to the innovated model of Saussure, how can we actually

use and interpret it, besides (alongside) the mere understanding of decision-making

processes and the need for linearity? We can use this model to determine static or

dynamic applications, and some short case studies on how to analyze them as such are

stated in the following section.

Static vs. dynamic

Every application has narrowed down all possible database options in order to generate

a coherent and comprehensible system. Because of the overwhelming choice in

specification no application is the same, neither do all of them have the same goal or

ideal linear syntagmatic path. It is possible to dissect these into static and dynamic

applications. When drawing new lines in the new associative relations model we can

infer three options in application genetic indication. (figure 9)

Figure 9 – Addition to model incl. genetic indication

Since syntagmatic design has to do with the linear comprehensibility and paradigmatic

design with the provision in clear and intuitive choices, conclusions can be made fairly

easily. When first looking at the bottom axis, we could assume we are speaking of a

more syntagmatic, static design, syntagm-genetic. This would imply straightforward,

almost purely informational applications in which the syntagmatic path is a strict one or

one that is pre-determined by the user himself.

34

Examples of these could be applications that offer travel instructions. These tend to offer

merely one primary service that users see as the sole reason of usage. They offer travel

information and no more, or less. The pre-determined syntagmatic path the user

develops here is one almost impossible to change, the understanding of the application

is the offering of travel information and routes, and therefore the usage will be limited

to only that use, which is thought of before opening the application. Making intuitive

choices are taken quickly because of clear set goals for usage.

The upper axis, however, is one focused more on the paradigmatic aspect, one that is

characterized by the large number of possibilities and associations made with these

preset, to be made, choices. This means that users have to choose from a large set of

options, that are correlated with each other by means of associative meanings. Even

though these meanings seem to imply the association with the signs themselves, they

could in fact also be related by means of the association with the application design as a

whole, and therefore the overarching-design.

An example of a paradigm-genetic designed application could be news applications, such

as Flipboard. They all offer a large variety of choices in news, and some general headline

news. Even though there still is the possibility to search more specifically, like internal

or foreign affairs, the news itself is continuously spread over subjects such as,

environment, weather, society, politics, economy and many others.

With the syntagm-genetic and paradigm-genetic applications or media there are of

course those in between. These stream-genetic designs provide users with not only

options to choose from, but they also fix this amount to ensure a logical, intuitive

process for the user. The application example as used in this thesis would end up being a

paradigm-genetic application. The number of choices given to the user is quite large

since the placed polls by users and clients are not divided into subjects or genres.

However, there is the possibility to arrange the polls on location. This would allow the

application design to still be clear and intuitive, without the information being spread all

over, and no preferred reading can be found.

35

CHAPTER FIVE – MODEL BROUGHT TO PRACTICE

Case study – narratives

How does this addition to Saussure’s and Bouwknegt’s models work in practice? This

question is asked in the context of the polling application as mentioned before, as well as

the issue of narratives. Thesexamples are presented because of the large influence these

additions have not only on digital signs, but for more reasons. When applying this model

in regard to narratives, we would quickly come across narratives that are across

multiple media platforms. To be more specific, this would result in the encounter with

transmedia storytelling. Transmedia seems to have developed because of the changing

needs of audiences. They have changed from an analogue to a digital group where the

need for a range of information seems to be the highest need of all. We have become

information hunters and gatherers.

Now extensive opinions are shared about whatever the audience sees in theatre; every

audience member has become a possible film critic with their own Twitter, Facebook,

YouTube and blogs. They can be very positive, but mostly tend to be critical since their

opinion can be given as anonymously as they want, I would like to refer to these quick

and imminent opinions as an audience siege. Moreover the audience has the possibility

to create their own versions of existing content for placement on the web as well as

distributing it via media channels like YouTube and Vimeo, better known as fan fiction.

Henry Jenkins researched this change in audiences, which he later called the

participatory culture.

A participatory culture is a culture with relatively low barriers to artistic expression and

civic engagement, strong support for creating and sharing one’s creations, and some type

of informal mentorship whereby what is known by the most experienced is passed along to

novices. A participatory culture is also one in which members believe their contributions

matter, and feel some degree of social connection with one another (at the least they care

what other people think about what they have created). (2006: p7)

Whenever talking about transmedia, there seem to be multiple definitions.

G. Long states in his master thesis: Transmedia narratives use a combination of

Barthesian hermeneutic codes, negative capability and migratory cues to guide audiences

across multiple media platforms. (2007: p3) This is a very interesting, but very extensive

approach towards the creation of transmedia storytelling.

36

Building a transmedia world

We have looked at G. Long’s definition of transmedia storytelling as a narrative that uses

Barthesian hermeneutic codes, negative capability and migratory cues. But what are

these terms and what contribution do they make to the transmedia concept? We will

discuss these terms here briefly.

For a transmedia concept to be a success it is of importance to make use of the negative

capability and migratory cues that Long uses in his definition. Long explains negative

capability as follows: Negative capability is the art of building strategic gaps into a

narrative to evoke a delicious sense of 'uncertainty, mystery, or doubt' in the audience.

(2007: p53)

It is not so much about creating tension or excitement, but more about the mystery,

which in its turn delivers a hunger for more information about what is next. It provides

the audience with the feeling of power they can fill in the gaps with their own

imagination, here it is of great importance to still leave them with enough curiosity to

find out more. Our growing need for information has made this a perfect storytelling

tool. Negative capability is also quite closely linked to the suspension of disbelief.

Janet Murray writes about the suspension of disbelief in her book Hamlet on the

Holodeck:

The pleasurable surrender of the mind to an imaginative world is often described, in

Coleridge’s phrase, as “the willing suspension of disbelief.” But this is too passive a

formulation even for traditional media. When we enter a fictional world, we do not merely

“suspend” a critical faculty; we also exercise a creative faculty. We do not suspend disbelief

so much as we actively create belief. Because of our desire to experience immersion, we

focus our attention on the enveloping world and we use our intelligence to reinforce rather

than question the reality of the experience. (1997: p110)

So Murray states that the notion of suspension of disbelief as it exists now is lacking in

its formulation. There is more; there is a psychological process, which takes place in the

mind of the , which creates belief. We are not willing to create belief, we are

unconsciously creating belief and are willing to create an even larger expansion of this.

37

Migratory cues are a way to create references to places or things that are not mentioned,

explained or shown in the story further. By placing these cues a content developer has

the opportunity to expand the story world to an enormous extent. Besides that, it gives

the option to the audience to further develop the story themselves, via blogs or forums,

which they set up. It creates hints in one media form to look for additional content in a

different extension.

Furthermore Long writes that Barthesian hermeneutic codes also are one of the key

elements in creating a transmedia world. Semiotician Barthes, together with other

structuralists, has created a classification system, which helps to further define things

like migratory cues and negative capability by developing five different codes:

hermeneutic, proairetic, semantic, symbolic and cultural. D. Felluga (2012) has given

clear explanations of these codes.

Hermeneutic codes can be compared to migratory cues. They refer to elements in a

story that are not explained, and therefore exist as an enigma for the reader. They raise

questions that demand explication.

Proairetic codes are responsible for the creation of interest and/or suspense, thus

having a major part in the structuring principle. Moreover the code applies to any action

that implies further narrative action.

It is obvious that these two codes are very much linked to the migratory cues and

negative capability; they all are small ‘narrative pleasures’, which give an incentive to

the mind to create and fill the gaps, which have been formulated, or as in transmedia

motivating the audience to go to different platforms.

Semantic codes refer to any element used in a narrative that suggests a particular,

often additional meaning, by means of connotation.

Symbolic codes are quite similar to semantic codes and Felluga finds it difficult to

distinguish the precise distinction between these two codes. However he does describe

them as a ‘’deeper’’ structural principle that organizes semantic meanings. This is

achieved mostly by means of antitheses, or because of mediations between antithetical

terms.

38

Cultural codes are those which point out our shared knowledge about the way the

world works, including properties that are physical, physiological, medical,

psychological, literary, historical, etc.

By using these different codes, it is possible to enhance the relationship the audience

has with the story, and because of the improvement of the intertextual connections

between the components in transmedia franchises, audiences are more motivated to go

from one media platform to another.

These three components, negative capability, migratory cues and hermeneutic codes,

are a very good help in creating a transmedia story. Building an intertextual franchise

can be difficult when it gets to narrative continuation, it is essential that all the media

platforms are linked to each other. Long gives a strong and precise proclamation about

building transmedia worlds:

A storyteller looking to craft a potential transmedia narrative should carefully craft the

world in which that story exists, and then make passing references to other cultures,

characters, events, places, sciences or philosophies of that world during the course of the

narrative to simultaneously spark audience imaginations through negative capability and

provide potential openings for future migratory cues. (2007: p68

When having your story told on different platforms you have to motivate the audience

to go from one platform to another. A transmedia story world needs different elements

to be successful in this, by creating the necessary ‘gaps’ in the story which can be filled

in by the audiences’ creativity, or by means of another platform it is possible to generate

a certain lust for more. It is essential to make sure audiences are willing to go from one

platform to another, even though if that platform is not their favourite.

It is the fact that all media platforms are combined in one way or another in the context

of transmedia storytelling, in order for the audience to comprehend the entire story

world they need to discover and experience each platform. The proposed narratology

options by Long show that by these gaps and references audiences are motivated to

discover more of the world. The model is applicable here as well. Even though there are

multiple media platforms, in which the audience has to make a paradigmatic choice in

what text to read first, there is certain linearity in the discovery. Producers are able to

steer this by releasing the platforms in the preferred order, but this would also imply

that only certain audiences are attracted when a media platform is released. The desired

39

linear, and thus syntagmatic, discovery is determined by the exact paradigmatic choices

made by audiences that are motivated to travel across platforms by means of negative

capability and migratory cues. Moreover, each medium already represents a deeper

understanding of the world; this would explain the addition of profundity to the model

and narrative.

40

Case study – application

If we look back at the application as proposed before, the polling application, we could

first of determine the genetic indication, since the user here has the enormous

possibility to choose the poll he or she would like to vote on. Moreover are there options

to subdivide polls into local polls or even search with certain keywords. This would

result in a paradigm-genetic application, since the user has a lot of possible

paradigmatic choices to make. Not only the choice in categorizing the polls, but the

variety in actual polls themselves all contribute to this paradigm-genetic characteristic.

However, when it comes to these kinds of applications it becomes increasingly more

difficult to steer, or voluntarily force, the user to a certain syntagmatic line. Because of

the infinite choice in paradigmatic options, how do we ensure the user gets the exact

encoded message, or poll in this example?

As already stated above, the enormous advantage of the database of signs is that the

system can interpret and hold all the information entered by the user. However, the

awareness of the user when actually entering the information is almost nil, since they

are not asked to give this information, simply because of the use and the voting behavior

of the user, the system can decode preferences and thus encode the most interesting and

intuitive polls for each specific user. When these kinds of polls, preferring polls, then

become more or less expected, and thus heuristic, the system can apply unexpected,

promoted polls that will give the user the opportunity to make minor changes in his or

her notion of preferences in the system.

Not only the change in preference is beneficial for the user, it generates an eye-catching

effect, when using different colors, and a different poll with different keywords, that are

in line with the paradigmatic associative intuitive choices the user has to make. This way

the developer can voluntarily force the user to make those exact intuitive paradigmatic

choices to follow the developer’s desired syntagmatic path. The development of such a

system that reads and understands the user’s behavior and can respond accurately with

use of the associative meanings, connected to another database that provides the system

with an unlimited access to all different kinds of signs that provides the system with

these meanings, is the key and semiotic understanding of building an application of this

kind.

41

CHAPTER SIX - CONCLUSION In conclusion, all these researched elements of the encoding – decoding processes, the

storage and retrieval, the associative signs model and its added feature - we can start to

see how we could, and should, actually design mobile screen communication in such a

way to ensure that users make logical and intuitive paradigmatic choices that allow for

the most preferred syntagmatic route throughout the application.

Most of the mobile screen communication currently has been using so-called symbolic

icons: they are a subliminal representation of the entire Overarching design an

application holds, and the first encounter with the user. The encoding of this symbolic

icon is strongly influenced by cultural and societal influences the users have

experienced. The storage and retrieval processes as described in the limited capacity

model show that even though we can signify one element, the retrieval makes the user

interpret signs and translate them into other signs, this in order to provide a more

comprehensive understanding of the given text. Moreover, by incorporating logical and

cultural denotative and connotative elements in the encoding of symbolic icons, there is

no need for the user to generate an extensive reading and judging. This way we can

create an efficient and logical communication of meaning and ensure the symbolic icon

is decoded exactly as meant to be, when encoded.

When the user enters the application a network of signs, known as Saussure’s ‘relations

associatives’ that when translated to digital media, by contributions of, among others,

Bouwknegt, we have concluded an additional feature has been missing, one that claimed

the profundity of each individual sign. Moreover we identified three different genetics

for applications to hold. All of these express the importance of the correct usage of the

database, which encodes all different options, the great importance of this database of

signs has become apparent.

We have seen that encoded user input is decoded by the system that has unlimited

resource capacity to all possible signs. This system in turn finds the appropriate

associative signs for encoding the appropriate message to the user to ensure intuitive

choices for the user. By understanding this exact system as a developer we can motivate

and voluntarily force users to specific linear syntagmatic processes.

42

LITERATURE Aarseth, E. J. (1997). Cybertext: perspectives on ergodic literature. Johns Hopkins

University Press.

Atkin, A. (2006). Peirce's Theory of Signs. Available:

http://plato.stanford.edu/entries/peirce-semiotics/. Last accessed 1st May 2013.

Baudrillard, J. (1998). Simulacra and Simulations. Jean Baudrillard. Stanford University

Press, pp.166-184. Available:

www.stanford.edu/dept/HPS/Baudrillard/Baudrillard_Simulacra.html

Berger, J. (2008). Ways of seeing. Penguin.

Bouwknegt, H (2011). Beyond the Simulacrum. Münster: Nodus Publikationen. p112-

138.

Chaiken, S., & Eagly, A. H. (1989). Heuristic and Systematic Information Processing

within and. Unintended thought, 212.

Chandler, D. (2007). Semiotics, the basics. New York: Routledge.

Cobley, P., et al. (2001). Semiotics and Linguistics. New York: Routledge, p233,

273.

Eco, U. (1981). The Theory of Signs and the Role of the Reader. The Bulletin of the

Midwest Modern Language Association. 14 (1), p35-45.

Felluga, D. "Modules on Barthes: On the Five Codes." Introductory Guide to Critical

Theory. Purdue U. Available:

http://www.purdue.edu/guidetotheory/narratology/modules/barthescodes.html Last

Accessed 27 Feb. 2012.

Foucault, Michel (1966). The order of things. New York: Vintage Books. Available:

http://www9.georgetown.edu/faculty/irvinem/theory/foucault-order_of_things-

text.html Last accessed 28 June 2013.

43

Hall, S (1973). Encoding and decoding in the television discourse. Birmingham: Centre for

Cultural Studies.

Hjelmslev, Louis (1961). Prolegomena to a Theory of Language. Madison: University of

Wisconsin Press.

Huening, D. (2013). symbol, index, icon. Available:

http://csmt.uchicago.edu/glossary2004/symbolindexicon.htm. Last accessed 17th May

2013.

Jenkins, H. (2006). Convergence culture. New York: NYU Press.

Jensen, F. 1990. "Formatering af forskningsfeltet: Computer-Kultur and Computer-

Semiotik." In Computer-Kultur- Computer-Medier- Computer-Semiotik, edited by Jens F.

Jensen, 10-50. Aalborg: Nordisk Sommeruniversitet.

Kress, G. (2010). Multimodality, A social semiotic approach to contemporary

communication. New York: Routledge.

Murray, J (1997). Hamlet on the Holodeck: The Future of Narrative in Cyberspace.

Cambridge: The MIT Press. 110.

Rachel. (2010). Rachel's review. Available:

http://www.goodreads.com/review/show/105315675. Last accessed 3rd June 2013.

Saussure, F (1972). Course in General Linguistics. Chicago: Open Court.

Sjöström, J. & Goldkuhl, G. (2005). The Semiotics of User Interfaces. Virtual, Distributed

and Flexible Organisations. 1 (2), pp 217-236.

Lang, A. (2000). The limited capacity model of mediated message processing. Journal of

Communication, 50(1), 46-70.

Long, G. (2007). Transmedia storytelling: business, aesthetics and production at the Jim

Henson Company. Massachusetts: Massachusetts Institute of Technology.