Affect Informatization

By Gentaro Makinoda

Nov. 1, 2015

Abstract

This project transforms "Affect” into an item of information that can be visualized.

While many philosophers and researchers have been discussing the Affect in connection to art,

architecture, or other forms of visual practices (in some cases, it is discussed as a political issue),

there seems to be no clear definition of what it is, and how it looks. While one may argue that the

value of Affect lies exactly in its non-representable qualities, this project speculates on the

possibilities of extracting the Affect itself. To accomplish this, the project first looks into the

theoretical works: starting with the discussions of the Affect by thinkers from varying fields,

including emerging Object Oriented Ontology, and ending with cognitive theories of

applied-sciences. Informed by these theories, a hypothesis for a visualization methodology is

presented, followed by an experimentation with a video footage to test its validity. The work

documented here is situated between a theoretical approach and a visualization method with

which researches of the kind could test and collectively advance the identification of the Affect.

Keyword: Affect; Information visualization; Object Oriented Ontology (OOO), Applied-science

Introduction: what recoding devices do and don’t do

(A photograph is) the illusion of a literal description of what the camera saw.

–Garry Winogrand

It should be appropriate to assume that almost everyone must have felt at some point in

his/her life that photographs do not do the justice of recreating what it "feels" like to be there.

2

People may concede this particular ineptness, as they are beholden to the photographs’ ability to

“record” events. But what exactly do cameras record? Photographs are “the illusion of a literal

description of what the camera saw” (Bogost, p.72). In Alien Phenomenology, Ian Bogost (2012) uses

Garry Winogrand’s above quote to argue that qualities of photographs are only relative to the

sensors of the cameras and “not the human eyes” (p.72). In fact, the sensors sense the light

reflected off of the objects in color data (Red, Green, Blue), and the display mechanisms recreate

them with their own logics. It has nothing to do with how human eyes sense the light through

retinae and process it in the brain. When people look at photographs, they are looking at the

recreations of what the camera sensors saw, and not what the human eyes saw.

Nonetheless, this does not prove the absolute incapacity of the cameras in capturing

affective qualities. As a matter of fact, they are capable of make people sympathize with the events

or the objects. Where is this capability coming from? According to some cognitive theories, it

might be coming from the basic pattern recognitions. Counter-intuitively, the fastidiousness in

recreating scenes may be disruptive and may even obscure the legibility of the affective qualities.

This should not be a huge surprise as “good” photographs are often being praised for their

simplicities. The purpose of this project, however, is not to distinguish “good” recordings from the

bad ones. Rather than seeking for techniques of scene recreations germane to human eyes, it will

focus on a particular quality in the recordings and identify an element of the Affect to emancipate

it from fitful discussions.

What is Affect?

One of the discussions around the Affect has been in the realm of art. Deleuze and

Guattari (1994) in What is Philosophy tie the Affect with the sensations one experiences with art.

“What is preserved – the thing or the work of art” argue Delezue and Guattari, “–is a block of

sensations, that is to say, a compound of percepts and affects” (p.164). On the same token, Farshid

Moussavi (2006) writes in Ornament of Forms that the Affect in architecture is people’s instinctive

response to either the form or the texture of buildings. For these critics, the interests are in the

Affect itself, yet for the thinkers like Fredric Jameson, the Affect is also valuable in the context of

politics where the distinctions create the oppositions of ideologies:

Affect may anchor claims about the materiality of bodies and physiological processes that are not contained

or representable by language or cognition alone. …much productive critical work has been invested in how

3

concepts like affect, emotion, and feelings aid in comprehending subject-formation and political

oppositionality for an age when neoliberal capital has reduced possibilities for collective political praxis”

(Pellegrini, p.37).

The detailed analysis of Jameson’s work is not necessary for this project, yet the very fact that

Jameson posits the Affect with emotion and feelings in the discussion of politics suggests that he is

in agreement with other theorists that the Affect (whether it is real, or just a tool for the politics) is

something that is elusive and perhaps independent from human interpretations. Deleuze and

Guattari write, “Sensations, percepts, and affects are being (bold font usage by the author of this

text) whose validity lies in themselves...” (Deleuze, p.164). In other words, Affect is not just an

effect caused by human sensations. It could be treated as something that exists as a being. In the

light of Kant’s “things in themselves” and Graham Harman’s Object Oriented Ontology, any

semantic paraphrasing will not help identify what an Affect is unless we first find a way to interact

with at least one quality of it. (Graham Harman (2011) in The Quadruple Object explicates ways in

which humans – or any object in the world – are only capable of interacting with only few qualities

of other objects) Yet, as all of the above thinkers are in agreement, the Affect seems to be slipping

away from our cognitive process. The Affect is “sensation,” “reaction,” or “not contained or

representable by language or cognition.” When it is understood, the Affect is lost.

Outplaying the cognition

If the Affect is such that it could not be comprehended by our cognitive system, how is it

even possible to extract it (let alone visualizing)? In Information Visualization, Colin Ware (2013)

offers solutions to this puzzle. He introduces the idea of “sensory” symbols, which seems to fit the

definition of the Affect, and concludes that they are processed “without learning” and that “they

are well matched to the first stages of neural processing” (p.9). In the early stage of the cognitive

process, people abstract the features of the objects they see, where “visual image is analyzed in

terms of primitive elements of form, motion, color and stereoscopic depth” (Ware, p.179). During

this stage, no connection has yet been made between what one sees and what one knows about

them. It is merely a scanning of an entire image to find patterns. It is only after this early stage, the

system tries to match the extracted patterns with what one knows. Under this mechanism, “when

we look at something that is well known to us,” explains Ware, “what we perceive is largely a

product of information scored in our brain” (p.236). For this reason, one could speculate that the

4

Affect only appears during the early stage of the cognition. This might explain why seeing familiar

objects creates less sensation. For such objects, the cycle of pattern extractions to knowledge

assignments has already been practiced repeatedly that the system zap through the process, hence

the Affect is left unnoticed.

Theoretically, if there is a shape from which our brains cannot extract any known

patterns, it may remain in the early stage of the cognition process. The shape will be free from the

knowledge application, and will not be comprehended as something that was “made sense.” By

definition of the Affect, this shape could be considered as an object that remains in the affective

(or sensational)” state. This hypothesis creates the ground for enabling the visualization of the

Affect. On one hand, it is most likely not possible to “represent” a specific Affect because it would

require paraphrasing, yet it might be possible to extract an essence of the Affect itself.

A technique of visualizing Affect

Ware argues that the “pure instances of sensory or arbitrary coding may not exist” (p.17),

but this does not mean that the share of sensory coding instances cannot be increased, to the point

where the pattern-recognition is delayed or paralyzed, and this is exactly what this project is hoping

to accomplish. Learning from Ware, the key to delay the pattern-recognition is to free the

visualization from the knowledge application or the meaning creation. For this, reconsidering the

Gestalt principles may become crucial. The Gestalt principles place emphasis on lines, particularly

in “Continuity” and “Closure and Common Region” principles. “We are more likely to construct

visual entities out of visual elements that are smooth and continuous” (p.183) under the principle

of Continuity, and “there is a very strong perceptual tendency to divide regions of space into

‘inside’ or ‘outside’ of the contour” under that of Closure (Ware, p.186). Ware follows-up on the

importance of the contours by stressing that “the brain is exquisitely sensitive to their (contour

lines’) presence” (p.191). Moreover, he also establishes that “one of the most important products

of early visual processing (of the human cognition) is the extraction of linear features in the visual

array… either the visual boundaries of objects or the lines in a line drawing” (p.9). In the primitive

stage of the cognition, objects are abstracted into lines, where the Continuity and the Closure of

the lines help the cognition system to quickly (approximately < 100 milliseconds according to N.

Katherine Hayle, a James B. Duke Professor of Literature at Duke University) extract patterns,

through which the system tries to assign meanings.

This project will turn this mechanism around and will keep the objects in the affective

5

state. Without completely losing the qualities of the original, a video material will be transformed

into a collection of lines. Each frame will be redrawn with minimum number of lines so that

neither the Continuity nor the Closure will function, delaying the pattern recognition.

The Affect Visualized

The material for the experiment is a video footage of a view from Metro Expo Line in

Los Angeles. The recording starts approximately 30 seconds before arriving at the Western station

on the eastbound train, and the camera faces south (image-1). The original footage is slightly

shorter than 30 seconds. This particular stretch was chosen for its varieties – varying building

taxonomy, heights, and depths, so that when the visualization is completed, “types” of Affect (if

any) could be identified and compared from one section to another. The original footage can be

viewed here: https://youtu.be/UbRw9BonIA0

Image 1: a scene from the original video footage

With this footage, the first step is to abstract the objects into outlines, from which the

cognitive system will try to extract patterns. In doing so, a gray tone was chosen as the background

because Edward Tufte’s (1990) recommends gray as being a non-disruptive background color

(mentioned five times in “Color and Information” chapter of his Envisioning Information). Also, the

line-weights were kept consistent to prevent them from having depth or hierarchical meanings.

6

Image-2 shows the outcome of an earlier attempt. Needless to say, this particular version

has some issues. A house and a car are easily identifiable from their formal characteristics, i.e. the

gabled-roof lines and the headlight to front wheel positional relation. This was rather ironic

because if one tries to identify distinctive outlines, he/she is bound to choose “patterns” that are

most easily identifiable, which is exactly what the project intended not to do. Distinctiveness is

most often a synonym for recognizable-ness, yet if it is not recognizable, it bears the risk of being

arbitrary. For instance, there is a car to the right, and its windshield has been outlined but other

windows are left untouched. If all windows were depicted, it may become even more recognizable,

but which window(s) is the correct window to draw? For a successful visualization of the Affect,

more rigorous and neutral procedure is needed.

Image 2: Identification of “distinctive” boundaries of the objects. A failure

The technique of Affect visualization should lead to identical results regardless of who

visualized them. The Affect is a being; therefore it should be independent from interpretations.

Perhaps, the procedure of the visualization should be informed by Deleuze and Guattari’s this

statement: “Sometimes this (Affect to ‘stand up on its own’) requires…. great geometrical

improbability, physical impression, and organic abnormality” (Deleuze, 1994, p.164). Also, it must

not be forgotten that the brains are “exquisitely sensitive” (Ware, p.191) to the presence of

“Continuity” and “Closure.”

7

Subsequently, the technique of Affect visualization could be to extract ONLY the

vertical (=parallel to the direction of the gravity) lines of the objects’ boundaries. Image-3 and 4

show an example of this technique applied to the same video footage. Admittedly, this technique

also encapsulates a risk of arbitrariness as what seems to be “vertical” may not perfectly be vertical,

or vice versa. Nonetheless, it works on two accounts. First, since all extracted lines are conceptually

parallel, they do not fuel the tendency to see Continuity and/or Closure. Second, as the train is

moving in the horizontal direction, new lines enter the frame rapidly, obstructing the cognitive

process from catching up. A video constructed with this technique can be viewed here:

http://youtu.be/cG9DchXg6Tc. One weakness of this video is that the depth can still be detected

from the gradually-shortening-as-you-go-farther-away pattern; the “perspective.”

Image - 3: Extracting the “vertical” lines

8

Image 4: Extracted “vertical” lines

The initial intuition was that the ambient sound would help to amplify the Affect,

however the result proved otherwise. The sound - especially for people who are familiar with the

particular sound - carried another set of meanings, to which one’s cognitive system reacts. The

viewers of this version will literally “picture” a scene where the sound may fit; hence the Affect is

weakened or lost. (The version with the sound can be viewed here:

http://youtu.be/xxbkYTqVp9Q) Consequently, the version without the sound should be

evaluated as more successful visualization for the purpose of this project.

Conclusion

The project proposed a way to convert the Affect into a set of information, through a

technique of visualization informed by theories from both science and art. The extraction of

vertical lines disabled the cognitive process from applying the pattern recognition, extending the

first stage of the process. The outcome, or the visualization itself may feel differently across

different viewers, yet it does not disprove the hypothesis. The Affect is what lies in itself, as

Deleuze and Guattari argues; therefore, how each viewer consumes it should be irrelevant from

how the Affect presents itself. If the “vertical-lines” successfully disrupted the cognition process,

arguably, they are the Affect. Nevertheless some rules and procedures should be revisited to

minimize the perspective pattern recognition. Perhaps, extracting only the paralleling horizontal

9

lines may delay this pattern recognition. Upon iteration, the technique should be tested in

different cultures. Will there be any startling similarities or differences across different cultures? Is

there any relationship between the formations of the Affect and an ethos? These are the critical

questions that could be addressed moving forward. Although the hypothesis is still in its immature

stage, addressing such questions will potentially make the technique a valuable tool to discover the

hidden relationships of affective qualities within human enterprises.

Image 5: Lines in residential area

10

References

Bogost, I. (2012). Alien phenomenology, or what it's like to be a thing. Minneapolis, MN: University of

Minnesota Press.

Deleuze, G., & Guattari, F. (1994). What is philosophy?. New York: Columbia University Press.

Drucker, J. (2011). “Humanities approaches to graphical display.” Digital Humanities Quarterly, 5.

Harman, G. (2011). The quadruple object. Winchester, UK: Zero Books.

Hayles, N. K. (2015). Sci-arc lecture. Rethinking the Mind of Architecture, Sci-arc.

Moussavi, F. (2006). The function of ornament. Actar.

Pellegrini, A., & Puar, J. (2009). “Affect.” Social Text, 27(No.3-Fall 2009), 35--38.

Ware, C. (2013). Information visualization (pp.1). Massachusetts, USA: Elsevier.

Tufte, E. (1990). “Color and information.” Envisioning Information (1990) Graphics Press.