The Semiotics of Embodied Interaction… Connection Through Pragmatism

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

 

“we live in a world that has been vastly altered by our cognitive abilities such that we inhabit not only the empirical world of physical entities but also the world of sign systems, which are a direct result of our cumulative interactions with the world (and each other) over time. Thus, the relationship between the subject and the object is dealt in a pragmatic way, where external phenomena are experienced as signs that are meaningful to the organism and there is no separation of the two.” (p144)

Annotation

The experience of an empirical authentic relationship within an environment between the person and the objects in that environment is mediated by the inauthentic semiotics of what those same objects afford to the person as to how they can be used. O’Neill argues that there is no separation between the modes of a meaningful existence, “Because we perceive before we conceive, we find that the body is at the root of our conceptual apparatus as well as being able to engage with the world without having to think about it” (p158). As the human is an essential organism in the environment and not distinct from the environment. Seen in a pragmatic way, the semiotic signs from the objects in the environment communicates meanings that lead to action that impacts on the nature of the same environment.

The Semiotics of Embodied Interaction… Behaviour Change

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“The key to solving this fundamental problem [_{compatibility of Heidegger’s, Merleau-Ponty’s and Peirce’s theories}] is in understanding the relationship between perception and conception. In other words, in developing a theory that takes into account how veridical perceptual experiences of the ‘real’ world that are derived through direct perception become, stored, represented or re-perceived in our minds as knowledge. Furthermore this theory also has to take into account how this knowledge is fed back into the world as mediated representations that signify that same knowledge in our heads, allowing us to communicate and socially construct the everyday world of our reality.” (p133)

Annotation

Phenomenologically it is difficult to know for certain what one person sees is the same for others, but through a socio-cultural consensus meaning is agreed, attributed, and mediated through a visual grammar of signs. These signify meanings that are socially constructed and that can be successfully interpreted leading to a change in behaviour that the designer is trying to facilitate. How this behaviour change unfolds can be phenomenologically revealed, and also visually communicated.

The Semiotics of Embodied Interaction… Visual Grammar

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“the ability of users to recognize the grammar of interactive media sign systems is what allows them to interact.” (p116)

Annotation

How the visual communication works is through an agreed visual grammar of semiotic signs within the design that the user will be able to understand and successfully interpret in order to interact.

The Semiotics of Embodied Interaction… Perceiving and Interpretation of the Relevant Calls to Action

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“screen-based interactive media are extremely semiotic in character. The symbols, graphics, and pictograms, as well as the layout and structure of its emerging forms, are all related to the remediation of older convergent media (…). The key difference is that it not only has to be perceived and interpreted to understand what it is but it also has to be used and manipulated to reveal what it does. The logic of immediacy is strong here in that much of the way in which interactive media is presented is often a simulation of previous, physical real-world media forms, e.g., drawing packages and word processing. However, the way in which those representations are conceptually structured often has to be very different from the older version, in order to take advantage of the computational aspects of remediation.” (p105)

Annotation
The visual communication of affordances through the graphical user interface naturally involves perceiving and interpretation of the relevant calls to action at any one time. But it also involves communicating through user manipulation, what it does. To enact the desired behaviour in the user the visual communication has to attract attention, retain attention, communicate action, suggest how the affordance is to be manipulated and suggest the outcome. This is more than simply designing a “button.” The selection of the semiotic signs involved in this communication must appeal and make sense to the target audience, and that congruently make sense in the design. As Eco suggests there is much more to consider in the design of an interface’s visual communication, “The Principle that form follows function might be restated: the form of the object must, besides making the function possible, denote that function clearly enough to make it practicable as well as desirable, clearly enough to dispose one to the actions through which it would be fulfilled” (Eco, 1986, p63 cited in O’Neill, 2008, p113). 

The Semiotics of Embodied Interaction… The Phenomenology of Signification

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“From the Peircean perspective, the focus is on the phenomenology of signification, i.e. signs as they are experienced. This is very useful as a starting point for understanding how users might experience interactive media signs.” (p81)

Annotation

Drawing on work by Peirce, O’Neill conjures up a term that connects my research areas together. The phenomenology of signification looks at how semiotic signs are experienced. Although no more than a term at present, like aesthetics of interaction it presents a way of synthesising pragmatics, semiotics and phenomenology together to explore how visual communication can influence interaction design.

The Semiotics of Embodied Interaction… Socio-cultural Codes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Metaphor then, is important to understanding interactive media. Connotation on the other hand has not really been considered in relation to interactive media. The focus in HCI has largely been to increase efficiency and usability by reducing ambiguity, thus removing the possibility of connotation. Thinking about the connotative aspects might provide new avenues for signification and further layers of interactive media. It might give us an extra tool to help us understand how people interpret interactive media within the larger social context in which they appear. (…) If interpretation is important to understanding interactive media, then the semiotic concept of codes is particularly relevant to developing a semiotics of interactive media, because it identifies interactive media objects as texts that can be decoded or even recoded culturally by a user at the interface level. It is important here, not to confuse cultural codes with binary code or programming, even though there is also something inherently semiotic about them.” (p78)

Annotation
O’Neill develops his case away from the denotative binary HCI perspective into a more nuanced world that visual communication occupies through connotation and metaphor. These two areas of nuanced communication have been a staple in visual communication. The selection and shaping of semiotic signs has been a valuable tool in the designer’s skill set, but how these tools work cannot be quantified. Visual communication draws from the socio-cultural codes of the relevant audience to encode meaning in subtle and interesting ways, to draw the audience into reading the design. Metaphor helps put the audience into a familiar place to understand how to interpret the meaning encoded in the design. Denotation, or this means this may communicate quickly but can be cold and, well, binary. Connotation on the other hand draws on an abstract richness of human communication that can, if interpreted correctly by the target audience, impart far more meaning than it’s parts. This is the difference between automated and anaesthetic experiences, and a phenomenology of enjoyment that an aesthetic experience can afford.

Guerilla Metaphysics… Perception

HARMAN, G. (2007) Guerilla Metaphysics: Phenomenology and the Carpentry of Things. Open Court Publishing Company.

“Perception can only assist our acts of meaning without being directly responsible for them, since these acts by definition go beyond the materials with which they work.” (p24)

Annotation:
Harman, when discussing perception, believes it is not directly responsible for intentional acts. The attribution of meaning to that act is contextual to the object associated with the intentional act, as the material the object is composed of affords information to the call to action that is communicated. Perception of this affordance, through the semiotic signs, is enabled but it is in interpreting and understanding the semiotic signs that leads to the interaction.