Semiotic approach to mobile screen communication: How we are instigated to voluntarily make descisions
Oscar Bastiaens. Master in Media Innovation, NHTV Breda University of Applied Sciences. July 5th 2013.
3
Semiotic approach to mobile screen communication: how we are instigated to voluntarily make decisions. Master thesis O.S.P. Bastiaens, student at NHTV Breda University of Applied Sciences, Master in Media Innovation. Supervisor, Associate Professor Hans Bouwknegt
July 2013
4
PREFACE After graduating from by bachelor I thought I’ve found the subject of my dreams;
transmedia storytelling. However, this master has provided me with so much more
inspiring theories, thoughts and lecturers, that this could only get better. Now, in my line
of interests, semiotics has managed to take a high position and helped me to deepen the
knowledge, insight and understanding of many theories, structures and narratives that
I’ve researched before. For this, I owe Hans Bouwknegt a great deal of thanks.
Not only has he introduced me in this line of theory, he has been supervising me
while writing this thesis and challenged me each and every time to think again
and go beyond the obvious and aesthetics. I couldn’t have wished for a better,
and fore mostly inspiring, supervisor for these last weeks.
Besides Hans I would also like to thank my very good friend and business
partner Joris Niessen for all his time, thoughts and ideas that all helped getting
my mind straight.
John van den Elst, a big thanks for helping me with all my other documents and
remaining realistic in projections and ideas.
My mother and father, Karen Rokx and Sjef Bastiaens who have listened to my
endless gibberish on semiotics and signs, thanks for all those hours willing to
listen to me.
My fellow classmates with whom I’ve had a fantastic year and all our
brainstorming sessions to get inspired once more and of course all our leisure
time well spent.
Of course Ling Teo for helping me out on marking up the document.
Finally many thanks to all of those who I’ve forgotten to mention here.
5
MANAGEMENT SUMMARY
Whereas users nowadays seem to be glued to their mobile devices and in particular
smartphones, the common communication has become more visual than before. This
thesis deals with the question of how in-application use can be steered by means of
symbolic icons and with that, indicate how a preferred reading comes about.
Furthermore it researches how these are best encoded and in return decoded by the
user. Semiotics offers a broad pallet of theories and structures that are able to strip off
all excesses of aesthetic abundance and grasp the essential elements that make up for
the successes and failures, therefore this line of research will be primary for this
research.
First of all the symbol and icon are defined in order to understand and create a larger
comprehension on their operation. Peirce (2006) here has already defined a large part
of these terms as he defined later on in his description of semiosis. However, by
combining this theory with the connotative and denotative features as described by
Stuart Hall (1973) and therefore incorporating the encoding and decoding in
formulating the correct definition, the conclusion can be made that symbols are those
visuals that are connotative, dynamic objects, whereas icons are denotative immediate
visual objects.
In order to understand how these symbols and icons are processed by the human brain
a thorough research is needed into the processing stages the user is experiencing,
consciously as unconsciously. This thesis finds that the slight use of symbols in the
visual expressions used is due to the fact that the recognition and understanding of the
complete message is made easier for the user. Since the combination of both symbol and
icon is common, a proposition to further refer to symbolic icons instead of symbols and
icons is presented.
When the entirety of symbolic icon definitions and processing has been reviewed, this
thesis examines the navigational aspects of mobile application usage. For this the
Saussurian model of associative relations is used to demonstrate how the linear,
syntagmatic reading comes about. However, this model is not concerned with the digital
implications yet, therefore Bouwknegt’s (2011) models of digital associations is used to
generate a larger understanding and ensure a valuable addition to the existing model of
Saussure.
6
The profundity axis is added to illustrate the fact that each sign, and thus symbolic icon,
in itself contains a subliminal reference to what lies beyond. Furthermore, does this new
model contain a so-called genetic indication for applications, which illustrates their
freedom of choice, and therefore influences the process of reaching the preferred
reading.
Finally, the findings are practically illustrated by means of two case studies,
demonstrating the effectiveness of the new model in current mobile screen
communication and narrative.
7
TABLE OF CONTENTS
PREFACE 4
MANAGEMENT SUMMARY 5
INTRODUCTION 8
CHAPTER ONE – DEFINING SYMBOL AND ICON 11 THE THREE TYPES OF SIGN VEHICLES 12 CONNOTATIVE AND DENOTATIVE 15
CHAPTER TWO: SYMBOL AND ICON PROCESSING 18 HEURISTIC-SYSTEMATIC MODEL OF INFORMATION PROCESSING 18 LIMITED CAPACITY MODEL 20
CHAPTER THREE – ASSOCIATIVE RELATIONS 22 PARADIGMATIC RELATIONS 23 SYNTAGMATIC RELATIONS 24
CHAPTER FOUR – MODEL CONTEMPLATION 26 A DATABASE OF POSSIBILITIES 26 ADDITION TO THE MODEL 29 STATIC VS. DYNAMIC 33
CHAPTER FIVE – MODEL BROUGHT TO PRACTICE 35 CASE STUDY – NARRATIVES 35 BUILDING A TRANSMEDIA WORLD 36 CASE STUDY – APPLICATION 40
CHAPTER SIX - CONCLUSION 41
LITERATURE 42
8
INTRODUCTION
Our mobile devices have become like an extension of our hands. The immense growth in
possibilities the device has given to users all around the globe ensured an almost
irreplaceable status. Texting to whomever we want to, wherever we want to in
whatever time suits us best. In the past centuries communication by means of letters
and words have been standard. However, many brands and (user interface) designers
all over the world have noticed the power of recognition by means of logos, shapes,
symbols and icons by the consumers. It is because of this notion that perhaps now, more
ever than before, we need to consider the universal language of symbols and icons and
allow them to take a greater part in our communication, especially from system to user
communication. Figure 1 – Iphone home screen
Nowadays there seems to be a habit in mobile phone usage.
Because of the growth in not only storage capacity, but as
well the processor capabilities, and overall performance, it
has nearly replaced many traditional media devices and
captured it in one, single device. In this time where screens
are getting smaller and information more important and
significant to communicate, symbols and icons have become
more valuable in communicating, thus making as much use of
the already decreasing screen space (figure 1). We can
already see that symbolic and iconic communication is
happening throughout human computer interaction,
especially when it comes to mobile screen communication. Subtle and carefully
considered design choices have to be made to ensure the best possible first interaction
with the user by means of these symbols and icons. But what elements make up for
these symbols and icons, what do they constitute, and what generates their
recognizability? Examining Hall’s theory on ending and decoding (1973) and looking at
Eco’s semiotic definition of signs and their meaning (1981) will prove to generate a
broader, more general understanding and definition of the elements that make up
symbols and icons.
In light of these events, the entirety of the complete design of applications cannot be
ignored. The process is enormous, where the users have become accustomed to a simple
intuitive design; the actual back ends have grown to be increasingly complicated
9
combining databases with each other to provide them with recommendations, news,
and all other information to their liking. This will be investigated when the importance
of a overarching-design is discussed. Moreover, the need for a preferred reading, as well
as retaining the sense of freedom of choice has become increasingly important to
application developers. Users have grown used to a very intuitive and natural design
and therefore usage of these devices. But how has this usage actually become natural
and intuitive? Is this claim of intuitive design due to common habit, or something
completely different? The flow of choice and usage seems to have some similarities with
the associative relations model by Saussure (2006)
When it comes to mobile applications, the symbol or icon is the first impression of the
application in the App Store or Google Play Store, standing out is enormously important.
However, with the same symbol or icon, the brand creates certain expectation that have
to be met in the design choices afterwards when the application is opened and used.
Many applications that design their small icons or symbols seem to look alike in design
choices such as shapes and the amount of intuitive thought left for the user. How these
choices are made, and on what basis, will be discussed. With the increasing complexity
of encoding symbols and icons and their exact features and attributes, there seems to be
no firm general understanding in what creates the exact distinction between all the
available symbols and icons. Peirce’s works and semiosis as described by Atkin (2006)
will make sure this will become completely comprehensible. Semiotics offers a broad
pallet of theories and structures that are able to strip off all excesses of aesthetic
abundance and grasp the essential elements that make up for the successes and failures.
Because of certain limitations, this thesis will focus on the actual encoding and decoding
processes of the symbols and icons to illustrate the influence in decision-making
processes. This is even though there are many more design decisions to be made when it
comes the application development and communication.
The human memory has a large influence in the perception and understanding of the
incoming information on signs, symbols and icons. Here, prior gained knowledge is
accessed and associations are made to gain a full comprehension of meaning. When
users are confronted with many options and information in one screen, these
associative meanings influence understanding and the decision-making process, some
models from communication studies like those by Lang (2000) and Chaiken and Eagly
(1989) have already pointed out the large influential effects of prior knowledge.
10
Therefore this thesis will be concerned with the following main question: How can we
instigate in-app intuitive choices during usage to ensure the highest desired preferred
reading? The usage of symbols and icons help illustrate the great influence caused by
them. In order to gain insight in the encoding process and establishment of symbols and
icons, this thesis aims to give suggestions to improve the design of these exact symbols
and icons.
One of the important assumptions made is that a macro design framework always
influences the output made by many designers on micro, or user, level. This thesis
intends to lay out a semiotic framework to enhance the understanding of the actual
realization of design choices, and the underlying meaning of each main symbol and icon
applicable to mobile screen communication.
In order to maintain the applicability of the gathered information and research, a media
innovation concept that concerns a mobile application with a focus on data collection
will be used a guideline. By means of polling the opinions of users, the application is not
only enriching existing profiles with actual real-time user-subjective information but it
also increases the opportunity to enhance the prediction of trends and the likability of
everyday items, events, people, politics or even the predicted trends. These days data is
king, and the access to it is getting simpler each day. The vast amount that is already
published by individuals themselves online is for the taking, but consumers have
become more aware of the dangers of publicly displaying their interests and so-called
likes. A great idea to gather this data with consent and interest by this same consumer
would be polling opinions on daily matters proposed by the user and companies,
ranging from general and regional polls. By analyzing all the information trends and
opinions, upcoming trends are monitored and predicted. Maintaining a threshold that is
as low as possible will ensure quick and easy access and above all, participation by this
consumer.
As mentioned before, symbols and icons make up an enormous part for the mobile
screen communication, as seen in figure 1, and the increasing complexity has made it
more difficult to get a general understanding on what exactly a symbol and icon are. In
order to understand how these screens are built, analyzing the symbols and icons used
and providing a semiotic argumentation will lead to a better understanding of what is
happening on the mobile screens.
11
CHAPTER ONE – DEFINING SYMBOL AND ICON
Even though it might seem fairly easy to define symbols and icons, when looking into
the semiotic definition of these, it is very likely to generate great confusion. All in all,
does semiotics describe a text as being a string of signs, either letters or symbols and
icons? Umberto Eco already stated that text can be used for multiple purposes, however
the interpretation of the text is being forced upon the readers to analyze and interpret.
These interpretations in return are multiple, if not infinite. Moreover, is the interpretive
operation of a text by no means indefinite but rather imposed by the semiotic strategies
used in the text? (1981: p36) Here the importance of a deepened semiotic analysis of
symbols and icons is already stressed because of the immense complexity that each sign
has been given throughout the years and now has come to a more definite definition.
Atkin (2006) describes Peirce’s process from his early to his final works on the theory of
signs. In this research Atkin describes Peirce’s three elements of sign and
understanding. First of all there are sign vehicles, second objects and finally
interpretants - even within these elements a distinction is made. In later works Peirce
came to the conclusion the parts making up the theory of signs could be defined even
more: all of this theory has been found in a correspondence of Peirce with mostly Lady
Welby, who seemed to have a same particular interest
in signs and their attributes. It is these final works that
give a strong all-round understanding of symbols and
icons in semiotic use. In order to fully comprehend the
creation of the final theory of signs we have to look
back from where the ideas stem, and this would be
Peirce’s earlier works. His triad (figure 2), which marks
the features of a sign, is essential in understanding the
Figure 2 – Peirce’s semiotic triad elements that make up a sign. This triad refutes three
important aspects when it comes to elements of a sign. The representamen refers to the
form that the sign takes, this could be anything ranging from a letter, word or even
image, whereas the object is the actual reference of the representamen - this part is also
known as the sign vehicle. Finally the interpretant is the final interpretation made by the
reader. However, this interpretant is largely influenced by the reader himself as Eco
described. All three elements are necessary for a sign to exist and be meaningful in its
12
communication. But where do all these elements come from and what creates this
interdependency between them?
The symbol and icons used in mobile screen communication are complex because of
their dynamic component; it is almost merely a one-way communication. The
complexity calls for a thorough research and examination of key elements in the theory.
In order to be able to do justice to the organization and consistency of Peirce’s work, it is
best to follow the line of a close reading and adapt parts of Atkin’s writing without
intervening with comments.
The three types of sign vehicles
The sign vehicle is defined as a particular signifying element of a qualified sign. In
Peirce’s earlier work he described the three elements to be a (1) qualisign, (2) sinsign
and (3) legising. (2006: p16-17)
1. A qualisign is any sign whose sign vehicle relies on simple abstracted qualities.
As an example, Atkin refers to the color cards used to decide on an exact color of
paint. Even though the color chip in particular sings multiple elements, like the
fabric, the shape and others there is only one sign of them that matters to the
signifying ability by the color card, which is the color itself.
2. The sinsign is defined as any sign whose sign vehicle relies on existential
connection with its object. This can be seen as, for example, smoke being a sign
for fire, temperature as a sign for fever or molehills as a sign for moles.
3. Finally, the legising concerns any sign whose crucial signifying element is
primarily due to convention, habit or law. Just as traffic lights are a sign of
priority, the sign vehicles here signify in virtue of the conventions surrounding
their use.
Besides these sign vehicles, Peirce notes three different kinds of objects, the (1) icon, (2)
index and (3) symbol. He proclaims that an object determines their sign,: this could also
be seen, as Atkin describes as the placing of constraints or conditions on successful
signification by the object. (2006: p17-19)
1. The icon reflects the qualitative features of the object, such as portraits,
paintings or diagrams.
2. The index is a sign that uses existential or the physical connection between it,
and its object. A pointing finger could be the best example, therefore the natural
and causal signs.
13
3. Finally, symbols are described as those signs that utilize some convention, habit,
social rule or law that connects it with its object like words.
Finally there are the three different types of interpretants being (1) rheme, (2) dicent
and (3) delome. The interpretants aid in the classification of the signs in term of their
relation with the interpretants. (2006: p19-20)
1. The rheme determines the interpretant by focusing on the understanding upon
qualitative features that it employs in signifying its object, here we could of
examples like ‘…is a dog’ or ‘…gives…to…’.
2. Dicent focusses on the actual understanding of the signs on existential features.
One could think of examples like ‘Tanga is a dog’ or ‘Nina gives food to Tanga’.
3. The delome focusses on the understanding of a sign on some conventional or
law-like features employed, like arguments used in discourse.
Finally a list of a total of ten possible combinations between these three main elements
of Peirce’s sign theory were developed, since some combinations yield constraints. If we
would define symbols and icons from these definitions we could state icons to be a
representation of actual, mostly, tangible assets or products. An icon showing the wheel
of a car would therefore be a proper icon if the signified message would be just that.
However, a symbol would then be limited to actual words. We can assume that the time
wherein Peirce was living and writing he did not take mobile screen communication into
account, and all of its communication by means of small, square blocks.
However, in a later period in Peirce’s life he further defined and interprets objects and
interpretants, dismissing the three types and extends the definitions into dynamic and
immediate. Furthermore, the objects now are not necessarily defined as icons or
symbols anymore, but more as semiotic processes that assist in the understanding of the
signage process.
First of all there are the dynamic objects; these generate a chain of signs. The aim of this
chain is to come to a full understanding of an object, attempting to assimilate that object
into the system of signs and therefore the memory. This concerns the object as is really
is or as it is known to be.
As an example Atkin portrays a tank of petroleum that is half full. There are many signs
that indicate the tank is half full, such as the sound of the tank when it is being stroked,
the indication of the fuel gauge, a light on the dashboard and so on. Even though there
14
are all these possible signs, they only add up to the indication. The object that is
underlying all of them is the actual half full petroleum tank, the dynamic object.
The immediate objects on the other hand are those objects what we suppose at any time
within the sign chain. The object is noticed since it is the first time it is used and
interpreted by the recipient, it is not an addition to the dynamic object, but rather some
informational, incomplete equal of the dynamic object generated at some stage in the
chain of signs. Here again, we can use Atkin’s example of the petroleum tank to get a
clear distinction between these two kinds of objects. When, for example, we would
strike the tank we could hear a tone (which then would be the sign vehicle for the
immediate object), which would represent to us that the tank is not full. It does not tell
us the exact amount left within the tank. Then, here, the immediate object is a less-than-
half-tank. (2006: p25-27)
Concluding, we can state that the dynamic object is the end-goal of understanding the
sign chain and semiotic process, where the immediate object is our grasp of that object
in any given time during the sign chain process itself, adding the then gained knowledge
to our general semiotic understanding.
Moreover, Peirce has redefined the interpretants’ roles and divisions. Now (1) the
immediate interpretant, (2) dynamic interpretant and (3) the final interpretant make up
for the interpretants as being a vital part in the theory of signs. All of these new divisions
seem to be strongly influenced by Peirce’s development if the levels of understanding,
where he divided these into three levels, being (i) having an unreflective grasp of some
concept, (ii) having, or being capable of, providing a general definition of the concept
and finally (iii) having conceived the object with one’s own conception. Then, our
conception of these effects is the whole of our conception of the object. (2006: p28-35)
1. The immediate interpretant (linked to ii) provides a general definitional
understanding of the relationship between the sign and dynamic object. When
we use ordinary sentences to explain this we can see that this interpretant will
recognize grammatical categories, syntactic structures and conventional rules of
use. For example: ‘We don’t want to hurt him, do we?’. The immediate
interpretant would find this to be a question, concerning doing harm to a male
person.
2. When it comes to the understanding we reach, or which the sign determines, at
any particular semiotic stage we speak of the dynamic interpretant (linked to i).
15
This refers to the interpretant being aware of the sender, receiver and the
knowledge the sender holds.
3. The final interpretant (linked to iii) can be described best as one that
incorporates a complete and true conception of the objects of the sign. In the
end, this would be the interpretant with which everyone should agree with on
the long run. Usually there is a series of dynamic interpretants’ questions
concerning the actual meaning of the signified.
Therefore, the final interpretant is important in our understanding of the dynamic
object. First of all, it marks the actual point where the grasp of this dynamic object
becomes understandable and complete. Second, it functions as an example of the
normative standard on which the judgment of interpretative responses can be made
concerning signs.
These definitions by Peirce do not yet provide conclusive arguments to completely
define symbols and icons; they remain vague in their understanding of the encoding
process prior to their reading by the user. Therefore, to deepen this insight in the
general definition of signs and therefore symbols and icons, taking a look at Stuart Hall’s
(1973) paper on encoding and decoding will provide conclusive arguments to define
symbols and icons for this thesis.
Connotative and denotative
Stuart Hall has written in his article ‘Encoding, Decoding’ on a theory known as the
reception theory in semiotics. Here an important distinction is made between
denotation, which is the actual representation of the literal meaning of a sign and
connotation, which implies references made towards and deducted from the denotation.
Here already we are able to note the important difference between an intended message
and the decoded message, being an important aspect in discovering how icons and
symbols are comprehended. (1973: p512-513) How these notions and those researched
above come together to one meaningful understanding will be elaborated on below.
Hall also quotes the famous semiotician Roland Barthes’ example to set a clear example
on how the difference between this denotation and connotation works. For example, the
sweater always signifies a ‘warm garment’ (denotation) and thus the activity or value of
‘keeping warm’. But it is also possible, at its more connotative levels, to signify ‘the coming
of winter’ or ‘a cold day’. (1973: p513)
16
The influences from community and culture can be decisive in the comprehension of
symbols and icons in user-system communication. Even though there are some simple
visual signs, as the peace sign, which have achieved maturity to such an extent they are
almost universal. Even though these still were influenced in their development stages by
culture and community.
Even though Hall’s theory is a more textual analysis, it does apply to visual
communication in form of symbols and icons. There is a focus on the scope for
negotiation and opposition on the part of the audience. This means that a "text"—be it a
book, movie, or other creative work—is not simply passively accepted by the audience,
but that the reader/viewer interprets the meanings of the text based on their individual
cultural background and life experiences. In essence, the meaning of a text is not
inherent within the text itself, but is created within the relationship between the text
and the reader.
We could propose two statements on defining symbols and icons: Symbols are those
visuals that are connotative, dynamic objects, whereas icons are denotative immediate
visual objects.
However, Huening (2013) refers to the works of Johansen who researched Peirce’s
works and they notify that even though there is this triad of components, being sign,
index and icon, they do not necessarily make up a sign individually. On the contrary, he
states
‘’Consider a photograph: it has properties in common with its object, and is
therefore an icon; it is directly and physically influenced by its object, and is
therefore an index; and lastly it requires a learned process of "reading" to
understand it, and is therefore a symbol’’
Figure 3 – Runkeeper image
If we would take the Runkeeper’s application image (figure 3), we
can already see there are signs including icon, since there is the
actual representation of someone. The symbolic reference made is
that concerning the probable fact the person is in a running position. The blue color then
again refers to the ignition of certain emotions towards this color. Again, a symbol or
icon does have certain important elements that define them, but they are, almost, never
to be found individually.
17
Looking back at figure one, we do see it is very important to take into account that in the
current mobile media landscape the usage of symbols and icons is virtually always a
combined one. Most used are icons that have a small symbolic addition in order to
create a larger familiarity and recognition. In this perspective the definition of symbol
and icon as described above are still applicable, however, Eco described that ‘each sign
possesses some way certain features, which prescribe its contextual fate.’ (1981: p37)
This context is vital for the mutual interaction between signs and the understanding of
the small symbol or icon used for applications, which is a way, is a subliminal
representation of the application itself. All signs interact with each other based on the
previous intertextual history they have been in contact with.
The determined strength of mobile screen communication via symbols and icons
therefore seems to lie in the storage and retrieval processes of the brain concerning
gained general knowledge. The associations made with every part is due to the fact
there is prior knowledge about them, this prior knowledge in its turn is influenced by
many different factors including culture, user experience, etc. At a certain point some
become a natural process or a heuristic. The heuristic-systematic model of information
processing by Chaiken and Eagly (1989) examines these natural, heuristic processes. In
order to gain a better understanding on how these symbols and icons are decoded by
users, this model will contribute largely to this apprehension.
18
CHAPTER TWO: SYMBOL AND ICON PROCESSING Already, Eco stressed the importance of contextual presence when it comes to the
understanding of signs, ‘be it a word or a visual item such as an directional arrow.’ (1981:
p37) Eco quotes Buyssens to be the first to stress this in the semiotic context; the full
meaning of signs only becomes apparent, more comprehendible and meaningful when it
is placed within a larger context. However, this context is influenced by the earlier
mentioned prior knowledge. Foucault (1966) has created a certain distinction between
different kinds of signs, one being natural and the other, conventional. Where the
natural sign is concerned, he states this sign has a completely individual interpretation,
‘an element selected from the world and constituted as a sign by our knowledge.’ (1966:
60). On the other hand, a conventional sign is developed and established by the human
brain and therefore easier to remember than the strict natural sign.
As all these elements make up for the understanding and the eventual interpretation the
user will have, it is therefore vitaly important to investigate what happens in the human
brain when it decodes such symbolic and iconic messages. In order to illustrate the
process between text and human, taking a step back from semiotics and into
communication literature, the strength of dynamic and immediate objects in sign
interpretation appears to be addressed and highlighted in the theories discussed below.
In a way, symbols and icons are a sublimation of multiple interpretations made by the
mind in memory. The possibilities are, as Eco states, endless, but however limited to the
extent in which the brain has knowledge of the contextual references made.
The heuristic-systematic model is one that proves the influential power of the brain
when it comes to prior knowledge and the judgment of information.
Heuristic-systematic model of information processing
This model initially focuses on how persuasive messages are processed and interpreted
by which exact manner. However, the understanding of symbols and icons do have a
great deal to do with heuristic and systematic evaluation as described by Chaiken and
Eagly (1989).
This model suggests two ways of processing, one of them being heuristic processing.
This uses multiple judgmental rules that are simply certain knowledge obtained in the
past by the user, which have been stored in the memory. The large advantage of
heuristic processing is the fact the user has only to put in a very minimal cognitive
effort, it is a process simply ruled by means of availability, applicability and of course
accessibility. All of these are earlier gathered information stored in the memory, as
19
mentioned before, but as well dependent on the environmental and cultural influences
gained by the user, and at the specific moment of use in time. Of course, availability is
concerned with the actual storage in the memory from previous experiences, while
accessibility applies to the current ability of the user to retrieve the memory and then
use this. Applicability in its turn refers to the relevance of all the recollected information
from the memory to the judgmental task the user has.
Of course there are some factors that could influence heuristic processing on the time of
reading or when the initial information is stored in the memory. The research showed
that messages or information delivered by experts or messages endorsed by others
were accepted as being true or interesting without actually fully processing the
semantic content of the message. As seen, this heuristic view shows a de-emphasis on
detailed information processing. It is this quick processing which is taken into account
by user interface designers when creating symbols and icons to developed those visuals
that take quick, heuristic processing into account to ensure a recognizable and fast
processed symbol or icon.
On the other hand, the polar/dichotomous opposite of heuristic processing is
systematic processing. This is less interesting when it comes to quick recognition and
processing since this focuses on comprehensive and analytic, cognitive processing of
judgment-relevant information. When this processing is chosen the users values source
reliability and message content more than normal, therefore the effect on persuasion,
when the message is appreciated, is greater than normal. The eventual judgment is
therefore relying heavily on in-depth treatment of the relevant information. However, as
mentioned earlier, this thesis is concerned with the symbols and icons on mobile
screens, these are not complete messages to be told, but need to be relevant and
recognizable in only a matter of milliseconds. Therefore this approach is not relevant
enough when actual usage is concerned, however the systematic processing might be
relevant when denotative and connotative signs are interpreted and stored in the
memory for the first time.
Moreover, Chaiken and Eagly (1989) have found that when, as they refer to it, economic
concerns are predominant, which means when users are constrained by time issues for
example and needs to act, read and interpret quickly, users are more likely to employ
heuristic processing when formulating their judgment on the decoding. However, when
it comes to judging the reliability of the signs, users will employ the systematic
20
processing approach since this researches a source into more detail and based on this,
even though quick, research by the user the source is validated as reliable or not.
All in all, in order to ensure the comprehension of symbols and icons, we have to
incorporate logic, and cultural denotative and connotative elements in the development,
or encoding, of these symbols and icons. By means of this approach there is no need for
extensive reading and judging by the user necessary, creating an efficient and logical
communication of meaning.
The reliability on prior knowledge and experiences is vital in decoding mobile screen
communication and interpretation. However, how is this knowledge gathered and how
do users cope with the, sometimes, overwhelming amount of it? The limited capacity
model (LCM) by Lang (2000) examines this process and proposes three main stages of
encoding, storage and retrieval. This theory demonstrates the importance and methods
of each stage. An understanding of each separate stage and the entirety of this process
will help creating a stronger argument for the use and encoding process of symbols and
icons.
Limited capacity model
The three steps that make up the LCM all play a very important part in gaining and
developing the prior knowledge as well as understanding the decoding process the user
experiences. Moreover, it contributes to our comprehension and understanding of how
to create symbols and icons relevant to users.
Firstly, there is the encoding within the model; what is key here is remembering to
create a sign that stimulates the receptors used when using mobile applications, and this
will enhance the associative memory when searching for connotative signs. When the
sign is perceived by the sensory receptors it is held in a sensory store, one that is quite
comparable with the linguistic and perceptual parser in the human associative memory
flowchart by Anderson and Bower (1980). After the selection processes, in this case
probably heuristic or systematic processes (Chaiken and Eagly 1989) stemming from
the eleven steps by Goodman (1967), some approved bits of the information are allowed
to continue to the short term memory.
After the encoding and judging of the signs and/or information we come to the storage
stage. If positive evaluated information is let through to the memory, there is not really a
21
strong division to be made between the short and long term memory, other than that
information stored in the short term memory is information that is much easier
accessed and used. Other terms for these are therefore the working (short term)
memory and the encyclopedic (long term) memory. Lang (2000) does confirm that the
more associations can be made within the entire memory system the more accessible
the memory trace becomes. This also means that not all parts of the message or sign are
stored in the same way.
Where the final stage of retrieval is concerned, Lang (2000) states it is the process of re-
activating a stored mental representation of some aspect of the message. Again here she
stresses the more that associative links that are made in the previous stage, the more
readily retrievable this becomes. Therefore, when the LCM is concerned we have to
build easy comprehensible symbols and icons and relations to these in the first place to
make sure encoding is quickly generated, storage is allowed and retrieval can be based
on many other associations made by the brain to accelerate the retrieval process. This
system of storage and retrieval, but especially retrieval, has been noted in the semiotic
epistemology as being semiosis. This is the process whereby signs are interpreted and
translated into other signs, as seen before in Peirce’s theory. These associations in their
turn provide a more comprehensive understanding of the given text.
With regard to these storage and retrieval processes, we can thus conclude that the
slight use of symbols in the visual expressions used is due to the fact that the recognition
and understanding of the total is made easier for the user. All elements that are involved
in the definition such as, immediate, dynamic, denotative, connotative, symbol, icon and
index make up whether a sign is more symbolic or iconic. However, in order to generate
a more clear and general understanding of reference from this point on, I would like to
propose the term symbolic icons as the reference to the slight symbol and large icon use
in the mobile screen communication.
22
CHAPTER THREE – ASSOCIATIVE RELATIONS
Having defined and examined the symbolic icon encoding and decoding processes for
users as well as designers, and the definition of such processes, one wonders how
exactly in-application choices made by users are influencing the system, and how these
choices can contribute to the most preferred reading. The underlying associative
relations that each symbolic icon and sign holds perform this major influence on user
and therefore the system. Semiotician Ferdinand de Saussure has examined these
relations and it is his model that will provide a more in-depth insight when elaborating
on the navigational aspects of mobile screen communication.
In Saussure’s model there are two key axes portrayed in his model, paradigms and
syntagms. Both show the importance of a coherent and balanced intuitive design. This
chapter examines both aspects and the correlation between them, in accordance with
mobile screen communication and application use.
One important aspect is the concept of paradigms: a paradigm is a set of associated
signifiers or signifieds which are all members of some defining category, but in which
each is significantly different (Chandler 2007). When wanting a full comprehension of
semiotic strings of signs and the mutual relations between the symbolic icon and the
application itself, Saussure’s associative relations cannot be left out. Moreover, his
model of these relations demonstrates the coherence between all separate elements. All
of this will be discussed in this chapter.
The other is the concept of syntagms; the Oxford dictionary defines this as ‘a linguistic
unit consisting of a set of linguistic forms (phonemes, words, or phrases) that are in a
sequential relationship to one another’. (2013) Therefore syntagms concern the position
of single signs formed into, for example, strings of text. When looking into this, perhaps,
we can dissect older forms of semiotics in order to amplify the theory and arrive at a
general scale.
The semiotician Saussure had a particular interest in the relation between the signifier
and the signified: signs versus all other elements of the system, and those between a
sign and the elements which surround it within a concrete signifying instance (Chandler
2007). He described the key differences as being these syntagmatic and paradigmatic
23
forms, and developed a model,that shows the exact implications of these relations and
models.
Figure 4 (Chandler 2007)
As seen in this model (figure 4) the syntagmatic axis
deals with the positioning of the signs, in this case
words, concerning their order. The paradigmatic
axis on the other hand deals with substitution of
signs, also described by Saussure as the associative
relations.
When breaking down these axes to a quite abstract level, we can see that each choice in
paradigmatic design has the ability to differ and change according to personal
interpretation and preference. Whereas the syntagmatic axis provides this earlier
described need for a preferred reading to ensure a logical mental process.
Paradigmatic relations
A paradigm is a term that has come from an elaboration on Saussure’s série associative
(‘associative series’), which is a set of signs linked by partial resemblances, either in
form or in meaning. Saussure described such sets as being established ‘in the memory’
and the item this associated as forming a ‘mnemonic series’ (Cobley, 2001: p233).
Paradigmatic relations are functional contrasts, since they involve differentiation.
This means that the paradigm is concerned with the selection of 'this---or---this---or---
this' (Chandler, 2007: p84). It is about the choice made which media will be consumed at
which particular point in time. One of the strong reasons for users to step out of the
syntagmatic flow, and step into the paradigmatic of mobile screen communication is to
discover more information on the application and its features.
When perceiving paradigm more as associative it seems the emphasis shifts to sets of
signs which relate to one another, again related to the possibilities of substitution in
particular positions. When breaking down the term and effect of paradigm it is not
merely a set of signs linked by resemblances but it offers the creator and user of the text
the ability to change signs and provide his own interpretation. This means a text is no
longer bound to a set of rules concerning narrative development. It provides a large
scale of freedom, as long as the entire text stays in canon.
24
When digging even deeper into paradigms we can see that beyond the point of creating
an individual interpretation, the text is coded in such a way that the user or creator of
the text offers the possibility to design his own significance within the text.
Within this perspective, each choice in paradigmatic design has the ability to differ and
change according to personal interpretation.
Syntagmatic relations
According to the Oxford dictionary, syntagms are defined as ‘a linguistic unit consisting
of a set of linguistic forms (phonemes, words, or phrases) that are in a sequential
relationship to one another’. (2013) This means a syntagm is always worth more than
merely the sum of its parts. It concerns the entire sequence chosen and the (possible)
chronology within this string of text. Syntagmatic relations are those into which a
linguistic unit enters in virtue of its linear concatenation in a speech chain (Cobley,
2001: p273).
Whereas within paradigmatic terms it concerns 'this---or---this---or---this', within
syntagmatic terms it is about the combination of 'this---and---this---and---this'.
(Chandler, 2007: p84) This order creates a clear and understandable linear path and
codes the communication in such a way that users experience certain emotions or
intuitive choices at carefully placed times. Setting this within the perspective of free
choice, it becomes a lot more complicated. Even though the user has the choice to
discover every available medium, the producer has to carefully place enough
motivational cues for them to use, or click exactly where the producer wants them to go.
When amplifying this concept of a string of signs to more than an example with words, it
is possible to apply this to application navigation. In a sense, all application features are
different texts, with each new text making distinctive contribution to the whole.
Moreover, all these text contribute to one large meta-text: the sum of all parts creates a
far larger importance than each individually, even though they are created to be
sufficient and satisfactory when used as only one text or feature.
Another important aspect of syntagms is the (chronological) order. This will provide
motivation for the users to travel across features to constantly discover a new (and
larger) part of the application. Therefore, the importance of this model lies within the
fact that all the elements that make up a sign come from the dynamic and
connected/linked illustration of the sign structures by the literature.
25
Since applications contain quite a number of features and some are more influential
than the others, this thesis will take the symbolic and iconic communication as the
guiding line, in order to understand how certain syntagmatic preferred readings may be
voluntarily forced by means of intuitive paradigmatic options and eventually, choices.
But this model has not taken the innovative steps and developments made by
technology and media currently. Even though this model does seem complete when it
comes to producing and understanding meaningful discourse where the signs are
words, it does not take the digital revolution into account yet. How do current
technological applications, such as databases, take this model into consideration? Could
they provide a more complete and modern associative relations model?
26
CHAPTER FOUR – MODEL CONTEMPLATION Saussure’s model does not seem to be fully complete in the context of current
technological development: in this time where mobile screen communication is
changing behavior and attitudes towards information processing and usage, we have to
take these new innovations into account and re-apply them to this semiotic model of
navigational possibilities. Nowadays when databases are packed with information and
are constantly expanding, there is the possibility of completely rearranging the way we
compose texts. There seems to be infinite possibilities for encoding messages, but the
importance is the actual encoding of those messages by the database, resulting from the
given user input, in order to create valuable discourse with the user.
A database of possibilities
The associative relations model has provided a clear understanding on the creation of
linear and understandable discourse. However, in the digital world, Aarseth (1997)
proposes the idea of cybertext as being a kind of ergodic literature to illustrate the
semiotic framework of a computerized system. The cybertext has a main focus on the
mechanical organization of the destined text: it centers the reader or user of the text as
an almost completely integrated figure. The thing that makes this research and
exploration so interesting is the fact that Aarseth describes a semiotic sequence of the
cybertext in which the user’s input affects this same sequence greatly.
Even though one might think the influence of the user is made up from the sum of
actions and decisions made on an intuitive as well as a conscious level, it is mostly due
to the immense capacity of the computer’s processing speed, and the unlimited access to
a enormous storage of information - databases.
Aarseth has never given much scholarly attention or appreciation to semiotics. Reviews
on his book sometimes even stated that ‘He obliterates semiotic and poststructuralistic
analyses (which are weak more because of their authors than their theory). One problem
with the approaches is that they do not account for emergent behavior (results not
predicted by the designer; i.e., a programmer being beaten by his own chess program).’
(2010) However, his idea concerning these digital texts and their link to database usage
is very interesting for this research. Moreover, when peeling down the definition to the
mere bones, we are able to understand the linkage to the use of signs within these
databases. The symbols in total, form the string of text, sign being digital.
27
Aarseth defines ergodic literature in his book ‘Cybertext’ as the following:
‘In ergodic literature, nontrivial effort is required to allow the reader to traverse
the text. If ergodic literature is to make sense as a concept, there must also be
nonergodic literature, where the effort to traverse the text is trivial, with
no extranoematic responsibilities placed on the reader except (for example) eye
movement and the periodic or arbitrary turning of pages.’ (1997: p1)
Essentially this would mean that ergodic literature, like the cybertext, is that text in
which the user has the mere function of being almost completely immersed, and
therefore true action is required in order to comprehend the text. Whereas the
nonergodic literature stands for the complete opposite, where the user does not need to
provide true input and actions in order to read. However, this would only count for
those consciously-made decisions on actions, and not the natural consequences in
actions when reading a text, such as eye movement.
The contrast created between the importance of this linearity that has to be created by
developers, and the ergodic cybertext that implies no linearity whatsoever, makes a
complete understanding and the designing of applications (options) extremely difficult.
Hjelmslev (1961) noted a very important issue when semiotics is concerned, stating that
for semiotics, texts are chains of signs and therefore are linear by definition. Aarseth
also quotes Jensen (1990: p44)) in respect of computer communication and sees ’the
symbols as strings of binary digits’ (1997: p27). To this he adds that since these digits,
(zeros and ones) are programmed, they could only mean whatever meaning has been
defined to them.
Aarseth, with his ergodic literature, has given a great example of how the computer, or
the system, has provided users with a unifying platform on which the range of
possibilities in different forms of sign expression have increased immensely. Due to the
ease in accessibility of the database via the system, paradigmatic choices are enlarged
immensely, and the associative signs therefore receive a more profound meaning. The
process of constructing texts is changed when set in a digital context: no longer is it a
fixed, static text like the thesis you are reading at this moment. Whereas printed texts
are subject to their definite meaning, digital texts are dynamic and subject to change
whenever the users requests it. All possible recourses have become accessible, but with
it the increase in complexity of encoding the text. It is because of this reason that design
can disclose the amount of paradigmatic options to lead the user through all the possible
28
paradigmatic choices available in the system and its database. The interaction between
user and system is vital for the composition of digital texts: to illustrate the idea of the
impact of these changes clearly, the following example is offered as an illustration:
.
The user can be offered a static text, such as a Word document, but at the same time the
user can replace every letter or sign with whatever input is given, thus changing a static
text to a dynamic one that is not fixed and set to a certain meaning. The software
program Word has been designed in such a way that replacement at the user’s will can
be done easily and with the most logical associative signs, these all being stored within
the program. However, if the user wants to, they have the possibility to access the
infinite database via the Internet and replace any sign with any other given sign.
Therefore, the program or application design does provide a paradigmatic goal of usage
but because of the dynamic and constant accessible database nothing is fixed, only
guided.
Taking this unlimited chain of possible signs, it can be concluded that providing
unlimited possibilities to the user will most probably not give developers any chance of
enforcing in-app usage in order to ensure the most beneficial syntagmatic course. It is
because of this that services and products have been tailor-made to provide for more
specific needs of the consumer, the signified message is becoming increasingly difficult
to encode because of these unlimited options. Therefore, context and a coherent design
will provide the user with a greater sense of intuitive design and choices.
Concluding, we see that the system is a dynamic machine that makes up the string of
signs (code) because of the generated input by the user. This constant importation of
information in return generates a complex and tailor-made code back to the user. But
the databases accessed by the dynamic machine are infinite and constantly expanding
because of the constant import of (new) information by users. It is then due to this that
the options become infinite and the order of signs are no longer fixed; they are
completely dependent on the input by the user. The order ceases to be prescribed by
writers, directors or other content creators. As an example, this thesis is a finite
document which is to be read in the order as prescribed, however when accessing a
system that has unlimited possibilities of which the output depends on the given input,
there is no longer a clear understanding of where text starts or ends cleanly. But the
importance of a linear, syntagmatic reading remains, and options and choices are then
influenced by the context revolving around them.
29
When considering our primary associative relations model, we see that all these
paradigmatic decisions and choices can be unlimited, but are limited by developers to
provide a clear and coherent service or product. The syntagmatic linearity on the other
hand is one that has to be used if developers want to ensure a natural and intuitive
design and choice process for their users.
Addition to the model
In order to understand and convey the exact message of signifier and signified, as
written by Saussure in his book Course in General Linguistics there has to be a coherent
design in choice of application and the symbolic icon used. From this ideology we would
find ourselves in a position wherein design choices become increasingly important to
signify the desired syntagmatic course exactly.
However, how is such a coherent design created? When breaking down to merely three
main components in design we could end up with the (1) symbolic icon design, (2) the
application design and finally the connecting factor, being the (3) overarching design.
(See figure 5)
Figure 5 – Complete Design overview
As can be seen in the figure above, the overall coherent design is dependent on those
choices made at an overarching design level. Examples of these choices are basic colors,
shapes etc. The importance of these choices is vital because designers need to ensure
that the signified message by means of this design is interpreted exactly as intended - if
not, this could lead to a break in the expectation pattern that the user has generated in
his mind. All of this could cause to misunderstanding and misinterpretation, which then
leads to the cessation of use.
At the second level, application design is where the unlimited cybertext and database
choices have to be limited to merely those most beneficial to the user and producer.
Moreover, ensuring the encoding of used signs will enhance the efficiency of usage since
30
the user will make more intuitive choices and therefore become more consistent with
the tenets of ergodic literature.
The final level is the key thing users will come in contact with first when
browsing/searching the App store or the Google Play Store. Here extreme caution in
design choices is required since this actually is a subliminal representation of the entire
design structure and the initial pointof the creation of expectations.
Even though all these choices are made and they are strongly influenced by the
Overarching design behind it, how do we actually make the choice for particular signs in
mobile screen communication? Whatever the sign, or symbolic icon represents is what
makes it attractive to users, or otherwise. The most important aspect is to ensure an
intuitive design in which the signifier is signified exactly as intended to. Therefore the
unity of the entire design is of great importance in generating a coherent message as
intended. One of the important aspects that create a logical and intuitive flow
throughout the application use is the attractiveness of its unity of design and use. All
chosen representational signs have to represent exactly what the Overarching design is,
and therefore indicates how the entire application works.
Where Saussure expresses different kinds of linguistic features, being words, this model
is closer to the one as created/proposed? by Hans Bouwknegt (2011) in his book,
Beyond the Simulacrum. He has already pointed out and researched a translation of the
initial model to one that is concerned with all possible signs, platforms or even
narratives. It is these findings made by Bouwknegt that create a larger understanding of
digital media usage and choice.
Here the database of unlimited signs comes to order (becomes particularly significant?)
once more. Bouwknegt states that the interaction between the user and database is one
of the vital aspects when it comes to the usage and understanding of the database.
Whenever the user provides the system with input, the system has to encode a specific
message. Here is becomes clear that the system is reliable on the input given by the user
and cannot provide meaningful discourse on its own.1 (2011)
11 Bouwknegt makes a specific reference to Baudrillard’s simulacrum. Since the simulacrum is not
applicable for this thesis, but important to understand Bouwknegt’s findings. The simulacrum that states
there is less and less truth, because we base our reality on something that we (mankind) have devised (in
film and media) (1998: p166). An example of a 'simulacrum' is that everyone knows how a crashing
airplane looks like. But we know this because we have seen this on television. The ‘knowledge’ of this image
is not based on our own truth, but the truth is created on TV.
31
This would mean that even though the media can display all kinds of signs that refer to
the state of reality, they could only imply this reality instead of being the reality. In order
to become the actual reality, media needs the unlimited resource to all possible signs
and codes.
By making the user information permanent and apparent, databases and the use of
these databases are improved to a higher level in which producers and developers are
completely aware of the consumption behavior of the users of media.
The model of Bouwknegt as mentioned (see figure 6) above is one that shows ‘whether
signs are coded logically by algorithms or coded culturally. All respective levels are related
and can influence each other’. (2011: p122)
Figure 6 – Bouwknegt’s Network of signs
Even though the axes of syntagms and paradigms are replaced with logic and cultural
references, we can still view this model as one that could have the same purpose. All
individual encoded signs relate to one another in a specific way. The logical, and
therefore syntagmatic process is due to the exact and intuitive choices made in
paradigmatic options provided to the user. Bouwknegt did see there is an additional
level to this already, being the Meta level of the signs and how they correlate (figure 7)
32
Figure 7 – Hierarchy model
It is these relations and correlations that each individual sign has that contribute partly
to the creation of understanding. One could interpret each sign differently, by involving
this hierarchy within every sign; the database can exclude those signs with a high risk of
misinterpretation by means of mis-encoding messages. The codes here cease to remain
superficial and create another deeper layer of meaning when generating strings of signs.
The discussion of Saussure’s and Bouwknegt’s model of associative relations mentioned
before and the paradigmatic and syntagmatic relations could very well use an addition.
As seen before, each symbolic icon refers to known, associative meanings. But there is
more to it; each symbolic icon does not only reference to that exact point, at the same
time it represents the actual application and its (flow of) design. This would
consequently mean an extra axis could be added to Saussure’s initial model. This would
be the profundity axis that stands for the representation of the actual application itself
and not only this reference to
associative meanings. The
model would then look more
like the following.
Figure 8 – Addition to model
33
After this additional feature to the innovated model of Saussure, how can we actually
use and interpret it, besides (alongside) the mere understanding of decision-making
processes and the need for linearity? We can use this model to determine static or
dynamic applications, and some short case studies on how to analyze them as such are
stated in the following section.
Static vs. dynamic
Every application has narrowed down all possible database options in order to generate
a coherent and comprehensible system. Because of the overwhelming choice in
specification no application is the same, neither do all of them have the same goal or
ideal linear syntagmatic path. It is possible to dissect these into static and dynamic
applications. When drawing new lines in the new associative relations model we can
infer three options in application genetic indication. (figure 9)
Figure 9 – Addition to model incl. genetic indication
Since syntagmatic design has to do with the linear comprehensibility and paradigmatic
design with the provision in clear and intuitive choices, conclusions can be made fairly
easily. When first looking at the bottom axis, we could assume we are speaking of a
more syntagmatic, static design, syntagm-genetic. This would imply straightforward,
almost purely informational applications in which the syntagmatic path is a strict one or
one that is pre-determined by the user himself.
34
Examples of these could be applications that offer travel instructions. These tend to offer
merely one primary service that users see as the sole reason of usage. They offer travel
information and no more, or less. The pre-determined syntagmatic path the user
develops here is one almost impossible to change, the understanding of the application
is the offering of travel information and routes, and therefore the usage will be limited
to only that use, which is thought of before opening the application. Making intuitive
choices are taken quickly because of clear set goals for usage.
The upper axis, however, is one focused more on the paradigmatic aspect, one that is
characterized by the large number of possibilities and associations made with these
preset, to be made, choices. This means that users have to choose from a large set of
options, that are correlated with each other by means of associative meanings. Even
though these meanings seem to imply the association with the signs themselves, they
could in fact also be related by means of the association with the application design as a
whole, and therefore the overarching-design.
An example of a paradigm-genetic designed application could be news applications, such
as Flipboard. They all offer a large variety of choices in news, and some general headline
news. Even though there still is the possibility to search more specifically, like internal
or foreign affairs, the news itself is continuously spread over subjects such as,
environment, weather, society, politics, economy and many others.
With the syntagm-genetic and paradigm-genetic applications or media there are of
course those in between. These stream-genetic designs provide users with not only
options to choose from, but they also fix this amount to ensure a logical, intuitive
process for the user. The application example as used in this thesis would end up being a
paradigm-genetic application. The number of choices given to the user is quite large
since the placed polls by users and clients are not divided into subjects or genres.
However, there is the possibility to arrange the polls on location. This would allow the
application design to still be clear and intuitive, without the information being spread all
over, and no preferred reading can be found.
35
CHAPTER FIVE – MODEL BROUGHT TO PRACTICE
Case study – narratives
How does this addition to Saussure’s and Bouwknegt’s models work in practice? This
question is asked in the context of the polling application as mentioned before, as well as
the issue of narratives. Thesexamples are presented because of the large influence these
additions have not only on digital signs, but for more reasons. When applying this model
in regard to narratives, we would quickly come across narratives that are across
multiple media platforms. To be more specific, this would result in the encounter with
transmedia storytelling. Transmedia seems to have developed because of the changing
needs of audiences. They have changed from an analogue to a digital group where the
need for a range of information seems to be the highest need of all. We have become
information hunters and gatherers.
Now extensive opinions are shared about whatever the audience sees in theatre; every
audience member has become a possible film critic with their own Twitter, Facebook,
YouTube and blogs. They can be very positive, but mostly tend to be critical since their
opinion can be given as anonymously as they want, I would like to refer to these quick
and imminent opinions as an audience siege. Moreover the audience has the possibility
to create their own versions of existing content for placement on the web as well as
distributing it via media channels like YouTube and Vimeo, better known as fan fiction.
Henry Jenkins researched this change in audiences, which he later called the
participatory culture.
A participatory culture is a culture with relatively low barriers to artistic expression and
civic engagement, strong support for creating and sharing one’s creations, and some type
of informal mentorship whereby what is known by the most experienced is passed along to
novices. A participatory culture is also one in which members believe their contributions
matter, and feel some degree of social connection with one another (at the least they care
what other people think about what they have created). (2006: p7)
Whenever talking about transmedia, there seem to be multiple definitions.
G. Long states in his master thesis: Transmedia narratives use a combination of
Barthesian hermeneutic codes, negative capability and migratory cues to guide audiences
across multiple media platforms. (2007: p3) This is a very interesting, but very extensive
approach towards the creation of transmedia storytelling.
36
Building a transmedia world
We have looked at G. Long’s definition of transmedia storytelling as a narrative that uses
Barthesian hermeneutic codes, negative capability and migratory cues. But what are
these terms and what contribution do they make to the transmedia concept? We will
discuss these terms here briefly.
For a transmedia concept to be a success it is of importance to make use of the negative
capability and migratory cues that Long uses in his definition. Long explains negative
capability as follows: Negative capability is the art of building strategic gaps into a
narrative to evoke a delicious sense of 'uncertainty, mystery, or doubt' in the audience.
(2007: p53)
It is not so much about creating tension or excitement, but more about the mystery,
which in its turn delivers a hunger for more information about what is next. It provides
the audience with the feeling of power they can fill in the gaps with their own
imagination, here it is of great importance to still leave them with enough curiosity to
find out more. Our growing need for information has made this a perfect storytelling
tool. Negative capability is also quite closely linked to the suspension of disbelief.
Janet Murray writes about the suspension of disbelief in her book Hamlet on the
Holodeck:
The pleasurable surrender of the mind to an imaginative world is often described, in
Coleridge’s phrase, as “the willing suspension of disbelief.” But this is too passive a
formulation even for traditional media. When we enter a fictional world, we do not merely
“suspend” a critical faculty; we also exercise a creative faculty. We do not suspend disbelief
so much as we actively create belief. Because of our desire to experience immersion, we
focus our attention on the enveloping world and we use our intelligence to reinforce rather
than question the reality of the experience. (1997: p110)
So Murray states that the notion of suspension of disbelief as it exists now is lacking in
its formulation. There is more; there is a psychological process, which takes place in the
mind of the , which creates belief. We are not willing to create belief, we are
unconsciously creating belief and are willing to create an even larger expansion of this.
37
Migratory cues are a way to create references to places or things that are not mentioned,
explained or shown in the story further. By placing these cues a content developer has
the opportunity to expand the story world to an enormous extent. Besides that, it gives
the option to the audience to further develop the story themselves, via blogs or forums,
which they set up. It creates hints in one media form to look for additional content in a
different extension.
Furthermore Long writes that Barthesian hermeneutic codes also are one of the key
elements in creating a transmedia world. Semiotician Barthes, together with other
structuralists, has created a classification system, which helps to further define things
like migratory cues and negative capability by developing five different codes:
hermeneutic, proairetic, semantic, symbolic and cultural. D. Felluga (2012) has given
clear explanations of these codes.
Hermeneutic codes can be compared to migratory cues. They refer to elements in a
story that are not explained, and therefore exist as an enigma for the reader. They raise
questions that demand explication.
Proairetic codes are responsible for the creation of interest and/or suspense, thus
having a major part in the structuring principle. Moreover the code applies to any action
that implies further narrative action.
It is obvious that these two codes are very much linked to the migratory cues and
negative capability; they all are small ‘narrative pleasures’, which give an incentive to
the mind to create and fill the gaps, which have been formulated, or as in transmedia
motivating the audience to go to different platforms.
Semantic codes refer to any element used in a narrative that suggests a particular,
often additional meaning, by means of connotation.
Symbolic codes are quite similar to semantic codes and Felluga finds it difficult to
distinguish the precise distinction between these two codes. However he does describe
them as a ‘’deeper’’ structural principle that organizes semantic meanings. This is
achieved mostly by means of antitheses, or because of mediations between antithetical
terms.
38
Cultural codes are those which point out our shared knowledge about the way the
world works, including properties that are physical, physiological, medical,
psychological, literary, historical, etc.
By using these different codes, it is possible to enhance the relationship the audience
has with the story, and because of the improvement of the intertextual connections
between the components in transmedia franchises, audiences are more motivated to go
from one media platform to another.
These three components, negative capability, migratory cues and hermeneutic codes,
are a very good help in creating a transmedia story. Building an intertextual franchise
can be difficult when it gets to narrative continuation, it is essential that all the media
platforms are linked to each other. Long gives a strong and precise proclamation about
building transmedia worlds:
A storyteller looking to craft a potential transmedia narrative should carefully craft the
world in which that story exists, and then make passing references to other cultures,
characters, events, places, sciences or philosophies of that world during the course of the
narrative to simultaneously spark audience imaginations through negative capability and
provide potential openings for future migratory cues. (2007: p68
When having your story told on different platforms you have to motivate the audience
to go from one platform to another. A transmedia story world needs different elements
to be successful in this, by creating the necessary ‘gaps’ in the story which can be filled
in by the audiences’ creativity, or by means of another platform it is possible to generate
a certain lust for more. It is essential to make sure audiences are willing to go from one
platform to another, even though if that platform is not their favourite.
It is the fact that all media platforms are combined in one way or another in the context
of transmedia storytelling, in order for the audience to comprehend the entire story
world they need to discover and experience each platform. The proposed narratology
options by Long show that by these gaps and references audiences are motivated to
discover more of the world. The model is applicable here as well. Even though there are
multiple media platforms, in which the audience has to make a paradigmatic choice in
what text to read first, there is certain linearity in the discovery. Producers are able to
steer this by releasing the platforms in the preferred order, but this would also imply
that only certain audiences are attracted when a media platform is released. The desired
39
linear, and thus syntagmatic, discovery is determined by the exact paradigmatic choices
made by audiences that are motivated to travel across platforms by means of negative
capability and migratory cues. Moreover, each medium already represents a deeper
understanding of the world; this would explain the addition of profundity to the model
and narrative.
40
Case study – application
If we look back at the application as proposed before, the polling application, we could
first of determine the genetic indication, since the user here has the enormous
possibility to choose the poll he or she would like to vote on. Moreover are there options
to subdivide polls into local polls or even search with certain keywords. This would
result in a paradigm-genetic application, since the user has a lot of possible
paradigmatic choices to make. Not only the choice in categorizing the polls, but the
variety in actual polls themselves all contribute to this paradigm-genetic characteristic.
However, when it comes to these kinds of applications it becomes increasingly more
difficult to steer, or voluntarily force, the user to a certain syntagmatic line. Because of
the infinite choice in paradigmatic options, how do we ensure the user gets the exact
encoded message, or poll in this example?
As already stated above, the enormous advantage of the database of signs is that the
system can interpret and hold all the information entered by the user. However, the
awareness of the user when actually entering the information is almost nil, since they
are not asked to give this information, simply because of the use and the voting behavior
of the user, the system can decode preferences and thus encode the most interesting and
intuitive polls for each specific user. When these kinds of polls, preferring polls, then
become more or less expected, and thus heuristic, the system can apply unexpected,
promoted polls that will give the user the opportunity to make minor changes in his or
her notion of preferences in the system.
Not only the change in preference is beneficial for the user, it generates an eye-catching
effect, when using different colors, and a different poll with different keywords, that are
in line with the paradigmatic associative intuitive choices the user has to make. This way
the developer can voluntarily force the user to make those exact intuitive paradigmatic
choices to follow the developer’s desired syntagmatic path. The development of such a
system that reads and understands the user’s behavior and can respond accurately with
use of the associative meanings, connected to another database that provides the system
with an unlimited access to all different kinds of signs that provides the system with
these meanings, is the key and semiotic understanding of building an application of this
kind.
41
CHAPTER SIX - CONCLUSION In conclusion, all these researched elements of the encoding – decoding processes, the
storage and retrieval, the associative signs model and its added feature - we can start to
see how we could, and should, actually design mobile screen communication in such a
way to ensure that users make logical and intuitive paradigmatic choices that allow for
the most preferred syntagmatic route throughout the application.
Most of the mobile screen communication currently has been using so-called symbolic
icons: they are a subliminal representation of the entire Overarching design an
application holds, and the first encounter with the user. The encoding of this symbolic
icon is strongly influenced by cultural and societal influences the users have
experienced. The storage and retrieval processes as described in the limited capacity
model show that even though we can signify one element, the retrieval makes the user
interpret signs and translate them into other signs, this in order to provide a more
comprehensive understanding of the given text. Moreover, by incorporating logical and
cultural denotative and connotative elements in the encoding of symbolic icons, there is
no need for the user to generate an extensive reading and judging. This way we can
create an efficient and logical communication of meaning and ensure the symbolic icon
is decoded exactly as meant to be, when encoded.
When the user enters the application a network of signs, known as Saussure’s ‘relations
associatives’ that when translated to digital media, by contributions of, among others,
Bouwknegt, we have concluded an additional feature has been missing, one that claimed
the profundity of each individual sign. Moreover we identified three different genetics
for applications to hold. All of these express the importance of the correct usage of the
database, which encodes all different options, the great importance of this database of
signs has become apparent.
We have seen that encoded user input is decoded by the system that has unlimited
resource capacity to all possible signs. This system in turn finds the appropriate
associative signs for encoding the appropriate message to the user to ensure intuitive
choices for the user. By understanding this exact system as a developer we can motivate
and voluntarily force users to specific linear syntagmatic processes.
42
LITERATURE Aarseth, E. J. (1997). Cybertext: perspectives on ergodic literature. Johns Hopkins
University Press.
Atkin, A. (2006). Peirce's Theory of Signs. Available:
http://plato.stanford.edu/entries/peirce-semiotics/. Last accessed 1st May 2013.
Baudrillard, J. (1998). Simulacra and Simulations. Jean Baudrillard. Stanford University
Press, pp.166-184. Available:
www.stanford.edu/dept/HPS/Baudrillard/Baudrillard_Simulacra.html
Berger, J. (2008). Ways of seeing. Penguin.
Bouwknegt, H (2011). Beyond the Simulacrum. Münster: Nodus Publikationen. p112-
138.
Chaiken, S., & Eagly, A. H. (1989). Heuristic and Systematic Information Processing
within and. Unintended thought, 212.
Chandler, D. (2007). Semiotics, the basics. New York: Routledge.
Cobley, P., et al. (2001). Semiotics and Linguistics. New York: Routledge, p233,
273.
Eco, U. (1981). The Theory of Signs and the Role of the Reader. The Bulletin of the
Midwest Modern Language Association. 14 (1), p35-45.
Felluga, D. "Modules on Barthes: On the Five Codes." Introductory Guide to Critical
Theory. Purdue U. Available:
http://www.purdue.edu/guidetotheory/narratology/modules/barthescodes.html Last
Accessed 27 Feb. 2012.
Foucault, Michel (1966). The order of things. New York: Vintage Books. Available:
http://www9.georgetown.edu/faculty/irvinem/theory/foucault-order_of_things-
text.html Last accessed 28 June 2013.
43
Hall, S (1973). Encoding and decoding in the television discourse. Birmingham: Centre for
Cultural Studies.
Hjelmslev, Louis (1961). Prolegomena to a Theory of Language. Madison: University of
Wisconsin Press.
Huening, D. (2013). symbol, index, icon. Available:
http://csmt.uchicago.edu/glossary2004/symbolindexicon.htm. Last accessed 17th May
2013.
Jenkins, H. (2006). Convergence culture. New York: NYU Press.
Jensen, F. 1990. "Formatering af forskningsfeltet: Computer-Kultur and Computer-
Semiotik." In Computer-Kultur- Computer-Medier- Computer-Semiotik, edited by Jens F.
Jensen, 10-50. Aalborg: Nordisk Sommeruniversitet.
Kress, G. (2010). Multimodality, A social semiotic approach to contemporary
communication. New York: Routledge.
Murray, J (1997). Hamlet on the Holodeck: The Future of Narrative in Cyberspace.
Cambridge: The MIT Press. 110.
Rachel. (2010). Rachel's review. Available:
http://www.goodreads.com/review/show/105315675. Last accessed 3rd June 2013.
Saussure, F (1972). Course in General Linguistics. Chicago: Open Court.
Sjöström, J. & Goldkuhl, G. (2005). The Semiotics of User Interfaces. Virtual, Distributed
and Flexible Organisations. 1 (2), pp 217-236.
Lang, A. (2000). The limited capacity model of mediated message processing. Journal of
Communication, 50(1), 46-70.
Long, G. (2007). Transmedia storytelling: business, aesthetics and production at the Jim
Henson Company. Massachusetts: Massachusetts Institute of Technology.
Top Related