The Semiotics of Embodied Interaction… Connection Through Pragmatism

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

 

“we live in a world that has been vastly altered by our cognitive abilities such that we inhabit not only the empirical world of physical entities but also the world of sign systems, which are a direct result of our cumulative interactions with the world (and each other) over time. Thus, the relationship between the subject and the object is dealt in a pragmatic way, where external phenomena are experienced as signs that are meaningful to the organism and there is no separation of the two.” (p144)

Annotation

The experience of an empirical authentic relationship within an environment between the person and the objects in that environment is mediated by the inauthentic semiotics of what those same objects afford to the person as to how they can be used. O’Neill argues that there is no separation between the modes of a meaningful existence, “Because we perceive before we conceive, we find that the body is at the root of our conceptual apparatus as well as being able to engage with the world without having to think about it” (p158). As the human is an essential organism in the environment and not distinct from the environment. Seen in a pragmatic way, the semiotic signs from the objects in the environment communicates meanings that lead to action that impacts on the nature of the same environment.

The Semiotics of Embodied Interaction… Ready-to-hand and Present-at-hand Modes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Interacting with the media of all kinds then falls between these two modes of being. On one hand we perceive and encounter our media-rich environment directly, we manipulate it and transform it through our-ready-to-hand mode of being. On the other hand, we are constantly viewing, reading and interpreting mediated information through the reflective mode of encountering it as present-at-hand. At the same time we move between these two modes of being as we inscribe, interpret, transcribe and transform our mediated environment. In a very clear sense we are deeply entwined physically with the media in our environment and in another we are constantly making, and making sense of, the inscriptions that the media environment affords us.” (p140)

Annotation

An experience fluctuates between a ready-to-hand mode of being and a present-at-hand mode. The former involves the perception, interpretation, action and physically embodied manipulation using what is at hand in the experience of interacting within the present environment the person finds themselves in. The latter a more reflexive interpreting state of scanning what is available, making sense of the semiotic messages inscribed within the present environment the person finds themselves in.

The Semiotics of Embodied Interaction… Mediated Inauthentic Experience

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“when we are engaged in making physical representations of our thoughts, i.e., placing them out in the world either through writing, speaking or drawing, we are making further inauthentic media elements to be thought about. Therefore we cannot help but inhabit an inauthentic mediated environment.” (p138)

Annotation

The act of creating, especially of visual communication outcomes, and placing these in-the-world is a step away from a directly sensed and experienced reality, into a mediated inauthentic experience prescribed by the designer. The act of interpreting these outcomes to understand the meaning in order to decide upon an action in the interaction, leads to further inauthentic outcomes of clicking, moving, selecting within a digital realm. As our being-in-the-world is dependent on interplay between firstly an authentic direct natural empirical experience within an environment, and secondly an inauthentic mediated experience within that environment of stimuli that is not naturally available. Visual communication outcomes populate an authentic reality.

The Semiotics of Embodied Interaction… Hermeneutic Circle

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Related to the concepts of ‘Ready-to-hand’ and ‘Present-to-hand’ are the concepts of ‘authentic’ and ‘inauthentic’ being. For Heidegger, ‘authentic’ being comes about through experiencing the world as ready-to-hand in its firstness, its primary authentically disclosed state in a direct one to one relationship with a natural environment without any mediation. ‘Inauthentic’ being then, for Heidegger, is the experience of being-in-the-world that is predominantly based on being thrown into a prescribed world; i.e., it is an experience of living in a media-saturated world where most of our experiences are second hand. Both ‘authentic’ and ‘inauthentic’ experiences can occur in relation to both ready-to-hand and present-to-hand modes of being. That is, we can experience the natural world and the mediated world from the perspective of doing things with it or thinking about it.” (p136)

Annotation

An individual absorbs new knowledge from a variety of authentic (direct) and inauthentic (mediated) sources throughout their life. This impacts on an individual’s interpretation at any time (Bohman, 1991, p140), and explain aspects of deviation of interpretation when compared to the collective in a hermeneutic circle.

The Semiotics of Embodied Interaction… Behaviour Change

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“The key to solving this fundamental problem [_{compatibility of Heidegger’s, Merleau-Ponty’s and Peirce’s theories}] is in understanding the relationship between perception and conception. In other words, in developing a theory that takes into account how veridical perceptual experiences of the ‘real’ world that are derived through direct perception become, stored, represented or re-perceived in our minds as knowledge. Furthermore this theory also has to take into account how this knowledge is fed back into the world as mediated representations that signify that same knowledge in our heads, allowing us to communicate and socially construct the everyday world of our reality.” (p133)

Annotation

Phenomenologically it is difficult to know for certain what one person sees is the same for others, but through a socio-cultural consensus meaning is agreed, attributed, and mediated through a visual grammar of signs. These signify meanings that are socially constructed and that can be successfully interpreted leading to a change in behaviour that the designer is trying to facilitate. How this behaviour change unfolds can be phenomenologically revealed, and also visually communicated.

The Semiotics of Embodied Interaction… Visual Grammar

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“the ability of users to recognize the grammar of interactive media sign systems is what allows them to interact.” (p116)

Annotation

How the visual communication works is through an agreed visual grammar of semiotic signs within the design that the user will be able to understand and successfully interpret in order to interact.

The Semiotics of Embodied Interaction… Perceiving and Interpretation of the Relevant Calls to Action

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“screen-based interactive media are extremely semiotic in character. The symbols, graphics, and pictograms, as well as the layout and structure of its emerging forms, are all related to the remediation of older convergent media (…). The key difference is that it not only has to be perceived and interpreted to understand what it is but it also has to be used and manipulated to reveal what it does. The logic of immediacy is strong here in that much of the way in which interactive media is presented is often a simulation of previous, physical real-world media forms, e.g., drawing packages and word processing. However, the way in which those representations are conceptually structured often has to be very different from the older version, in order to take advantage of the computational aspects of remediation.” (p105)

Annotation
The visual communication of affordances through the graphical user interface naturally involves perceiving and interpretation of the relevant calls to action at any one time. But it also involves communicating through user manipulation, what it does. To enact the desired behaviour in the user the visual communication has to attract attention, retain attention, communicate action, suggest how the affordance is to be manipulated and suggest the outcome. This is more than simply designing a “button.” The selection of the semiotic signs involved in this communication must appeal and make sense to the target audience, and that congruently make sense in the design. As Eco suggests there is much more to consider in the design of an interface’s visual communication, “The Principle that form follows function might be restated: the form of the object must, besides making the function possible, denote that function clearly enough to make it practicable as well as desirable, clearly enough to dispose one to the actions through which it would be fulfilled” (Eco, 1986, p63 cited in O’Neill, 2008, p113). 

The Semiotics of Embodied Interaction… Graphic Language Displayed

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“the functionality of graphics often works in a metaphorical way that attempts to imbue the virtual world with a similarity to the real world (e.g., the desktop metaphor), which does not always display the same logic or affordances that would be apparent in the real world. (…) affordances occur only when we interact without having to think consciously about what we are doing.” (p85)

Annotation
The graphic language displayed through the visual communication in a digital interface manifests itself in ways that do not translate the real world directly. How something physically works in the real world may not translate to a representation in the digital world. The subtleties of how an idea of, or a suggestion to, a call to action is visually communicated are nuanced and seldom pedestrian. 

The Semiotics of Embodied Interaction… The Phenomenology of Signification

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“From the Peircean perspective, the focus is on the phenomenology of signification, i.e. signs as they are experienced. This is very useful as a starting point for understanding how users might experience interactive media signs.” (p81)

Annotation

Drawing on work by Peirce, O’Neill conjures up a term that connects my research areas together. The phenomenology of signification looks at how semiotic signs are experienced. Although no more than a term at present, like aesthetics of interaction it presents a way of synthesising pragmatics, semiotics and phenomenology together to explore how visual communication can influence interaction design.

The Semiotics of Embodied Interaction… Socio-cultural Codes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Metaphor then, is important to understanding interactive media. Connotation on the other hand has not really been considered in relation to interactive media. The focus in HCI has largely been to increase efficiency and usability by reducing ambiguity, thus removing the possibility of connotation. Thinking about the connotative aspects might provide new avenues for signification and further layers of interactive media. It might give us an extra tool to help us understand how people interpret interactive media within the larger social context in which they appear. (…) If interpretation is important to understanding interactive media, then the semiotic concept of codes is particularly relevant to developing a semiotics of interactive media, because it identifies interactive media objects as texts that can be decoded or even recoded culturally by a user at the interface level. It is important here, not to confuse cultural codes with binary code or programming, even though there is also something inherently semiotic about them.” (p78)

Annotation
O’Neill develops his case away from the denotative binary HCI perspective into a more nuanced world that visual communication occupies through connotation and metaphor. These two areas of nuanced communication have been a staple in visual communication. The selection and shaping of semiotic signs has been a valuable tool in the designer’s skill set, but how these tools work cannot be quantified. Visual communication draws from the socio-cultural codes of the relevant audience to encode meaning in subtle and interesting ways, to draw the audience into reading the design. Metaphor helps put the audience into a familiar place to understand how to interpret the meaning encoded in the design. Denotation, or this means this may communicate quickly but can be cold and, well, binary. Connotation on the other hand draws on an abstract richness of human communication that can, if interpreted correctly by the target audience, impart far more meaning than it’s parts. This is the difference between automated and anaesthetic experiences, and a phenomenology of enjoyment that an aesthetic experience can afford.

The Semiotics of Embodied Interaction… Horizon for Understanding

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“For Umberto Eco, for example, meaning is no longer an individual construct, as in the arbitrary semiotics of Saussure. It is now seen as the result of a process in which an individual takes part in society through the coding and decoding of his/her relationship with the cultural values and societal norms of the time.” (p72)

Annotation

Umberto Eco’s semiotics is far more embodied with an active relationship for a person, in their background culture and society, in interpreting the coding and de-coding of the norms for understanding. This makes any meaning arrived at not as an individual construct but one that is constructed from both the individual’s internal pre-understanding and external socio-cultural factors that form a horizon for the understanding. For Palmer (1969) the pre-understanding of the individual and the horizonality of the context in which the individual is interpreting the meaning dictate the terms in which any meaning is constructed.

The Semiotics of Embodied Interaction… Semiotically Communicated Affordances

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“If affordance essentially arises from the direct perception of the environment, as embodied knowledge, then mediated knowledge, which relies on indirect second-hand signification, must inherently be semiotic in character. (…) [a semiotic sign] is what is experienced when someone comes into contact with a set of stimuli that can be equated to a mental concept.” (p67)

Annotation
O’Neill argues that the communication of an affordances call to action is naturally mediated, and if mediated then it is semiotically communicated. This is important as it does make a valuable contribution in how visual communication can be far more influential in the design of an interaction than it has been previously thought. Through mediation in a visual medium of an interface, visual communication has always been present but under appreciated (denigrated in importance as ‘visual design’). With O’Neill’s argument comes opportunity to not only raise the importance of visual communication of the interface, but also to use visual communication to reveal the themes of an interactive experience that can be used to help explain the interactive experience.

The Semiotics of Embodied Interaction… Variance and Invariance

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“In Gibson’s view of perception, an active perceiver picks up and distinguishes between variant and invariant information. Either of these may be information about the perceiver or the environment but the distinction between the variant and invariant builds to create an awareness of the perceiver’s place within that environment.” (p61)

Annotation
The relationship between ourselves in the world, and the environment that surrounds us at any one time, is an important perceptual one. The environment is always perceptively relative to the person in that environment. They are aware that if they move, spatially and temporally that environment persistently envelope them. The perceptual changes within that environment are what guide and suggest options. These perceptual changes can be described as variant and invariant. In other words, some information within the environment stays constant and some will vary allowing behaviour change to be enacted. To give a simple example of variance and invariance it may be useful to look at an interactive experience on a computer. During any interaction the keyboard, screen and device will always be there (invariant) despite what is happening on the screen or whatever input device is being used (variant). The changes perceived between what is in the environment (keyboard, screen, device, table, room etc.) and what is changing within that environment is directly attributed to the user’s sense of being in the interaction. Whether this experience can be described as an aesthetic or an anaesthetic experience is another matter. What is important is the distinction between the two states and how this helps the conscious and subconscious directly as embodied knowledge perceive what is happening and can happen next.

The Semiotics of Embodied Interaction… False Affordances

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Gaver goes on to clarify his version of affordance by distinguishing between affordances themselves (i.e. the properties of things) and information about them (i.e. what we perceive). Gaver does this in order to show how we can be mistaken about affordances, claiming that if there is no information available for an affordance then it is hidden and that if information suggests a non-existent affordance, a false affordance exists.” (p52)

Annotation
Gaver adds another perspective to affordance of false affordances. Without going to far down this argument what Gaver does provide here are clues to how the semiotics of visually communicating the affordance can be problematic. 

The Semiotics of Embodied Interaction… Affordances

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Norman separates ‘real’ affordances (the physical properties of the world) from ‘perceived’ affordances (subjective representations in the mind), and is more concerned with the perceptual properties of affordances rather than the actual properties of real objects themselves. (…) The strength of Norman’s version of how perception and affordance work is that it attempts to solve the problem of how to explain the role of knowledge in understanding the world around us. However, the problem with Norman’s version of affordance is that it abandons the unique contribution of Gibson’s ideas in bridging the gap between the object and the subject.” (p51)

Annotation

Prof. Don Norman classifies different forms of affordances into those that are properties of the physical world (real), and those that are in the mind and are representative and subjective (perceived). It is the perceived affordances that Norman is more interested in and in doing so he distances himself away from Gibson’s original affordance concept. Lately Norman has refined his ideas on affordances now, unfortunately, referring to them as signifiers. In doing so he has moved affordances closer to semiotics but also now muddied the clear separations between the two. Affordances are about communicating potential action while semiotics defines how the visual communication will communicate the call to action. An example could be a water tap. An affordance would communicate how to get water from the tap. Whether the water is hot or cold is communicated through the semiotic signs of red or blue. The outcome of the semiotics isn’t the pouring water, but the communication of the expected temperature. The outcome of the affordance is how the tap is operated to access the pouring water. Both help in the ultimate outcome of accessing water but perform different tasks.

The Semiotics of Embodied Interaction… Perceptual Changes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“[Gibson’s] Theory of Ecological Perception begins by considering the relationship between living entities that perceive things in the world and the environment that surrounds them. (…) Invariant information (…) specifies the persistence of the environment and of oneself. (…) The perceiver is then aware of her own existence in a persistent environment relative to her own and others’ movements within the environment. This awareness provides us with information about the possibilities for action that the world around us directly affords. (…) Affordances then (…) are an emergent property of the physical relationship between environment and the direct perceptual acts of embodied beings.” (p50)

Annotation

The relationship between ourselves in the world, and the environment that surrounds us at any one time, is an important perceptual one. The environment is always perceptively relative to the person in that environment. They are aware that if they move, spatially and temporally that environment persistently envelope them. The perceptual changes within that environment are what guide and suggest options. How these guides and options within an environment communicate to the person is described as affordances, “an emergent property of the physical relationship between environment and the direct perceptual acts of embodied beings.” (O’Neill, 2008, p50). The calls to action that are afforded are defined within the semiotics of the visual communication.