The Semiotics of Embodied Interaction… Connection Through Pragmatism

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

 

“we live in a world that has been vastly altered by our cognitive abilities such that we inhabit not only the empirical world of physical entities but also the world of sign systems, which are a direct result of our cumulative interactions with the world (and each other) over time. Thus, the relationship between the subject and the object is dealt in a pragmatic way, where external phenomena are experienced as signs that are meaningful to the organism and there is no separation of the two.” (p144)

Annotation

The experience of an empirical authentic relationship within an environment between the person and the objects in that environment is mediated by the inauthentic semiotics of what those same objects afford to the person as to how they can be used. O’Neill argues that there is no separation between the modes of a meaningful existence, “Because we perceive before we conceive, we find that the body is at the root of our conceptual apparatus as well as being able to engage with the world without having to think about it” (p158). As the human is an essential organism in the environment and not distinct from the environment. Seen in a pragmatic way, the semiotic signs from the objects in the environment communicates meanings that lead to action that impacts on the nature of the same environment.

The Semiotics of Embodied Interaction… Mediated Inauthentic Experience

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“when we are engaged in making physical representations of our thoughts, i.e., placing them out in the world either through writing, speaking or drawing, we are making further inauthentic media elements to be thought about. Therefore we cannot help but inhabit an inauthentic mediated environment.” (p138)

Annotation

The act of creating, especially of visual communication outcomes, and placing these in-the-world is a step away from a directly sensed and experienced reality, into a mediated inauthentic experience prescribed by the designer. The act of interpreting these outcomes to understand the meaning in order to decide upon an action in the interaction, leads to further inauthentic outcomes of clicking, moving, selecting within a digital realm. As our being-in-the-world is dependent on interplay between firstly an authentic direct natural empirical experience within an environment, and secondly an inauthentic mediated experience within that environment of stimuli that is not naturally available. Visual communication outcomes populate an authentic reality.

The Semiotics of Embodied Interaction… Behaviour Change

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“The key to solving this fundamental problem [_{compatibility of Heidegger’s, Merleau-Ponty’s and Peirce’s theories}] is in understanding the relationship between perception and conception. In other words, in developing a theory that takes into account how veridical perceptual experiences of the ‘real’ world that are derived through direct perception become, stored, represented or re-perceived in our minds as knowledge. Furthermore this theory also has to take into account how this knowledge is fed back into the world as mediated representations that signify that same knowledge in our heads, allowing us to communicate and socially construct the everyday world of our reality.” (p133)

Annotation

Phenomenologically it is difficult to know for certain what one person sees is the same for others, but through a socio-cultural consensus meaning is agreed, attributed, and mediated through a visual grammar of signs. These signify meanings that are socially constructed and that can be successfully interpreted leading to a change in behaviour that the designer is trying to facilitate. How this behaviour change unfolds can be phenomenologically revealed, and also visually communicated.

The Semiotics of Embodied Interaction… Socio-cultural Codes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Metaphor then, is important to understanding interactive media. Connotation on the other hand has not really been considered in relation to interactive media. The focus in HCI has largely been to increase efficiency and usability by reducing ambiguity, thus removing the possibility of connotation. Thinking about the connotative aspects might provide new avenues for signification and further layers of interactive media. It might give us an extra tool to help us understand how people interpret interactive media within the larger social context in which they appear. (…) If interpretation is important to understanding interactive media, then the semiotic concept of codes is particularly relevant to developing a semiotics of interactive media, because it identifies interactive media objects as texts that can be decoded or even recoded culturally by a user at the interface level. It is important here, not to confuse cultural codes with binary code or programming, even though there is also something inherently semiotic about them.” (p78)

Annotation
O’Neill develops his case away from the denotative binary HCI perspective into a more nuanced world that visual communication occupies through connotation and metaphor. These two areas of nuanced communication have been a staple in visual communication. The selection and shaping of semiotic signs has been a valuable tool in the designer’s skill set, but how these tools work cannot be quantified. Visual communication draws from the socio-cultural codes of the relevant audience to encode meaning in subtle and interesting ways, to draw the audience into reading the design. Metaphor helps put the audience into a familiar place to understand how to interpret the meaning encoded in the design. Denotation, or this means this may communicate quickly but can be cold and, well, binary. Connotation on the other hand draws on an abstract richness of human communication that can, if interpreted correctly by the target audience, impart far more meaning than it’s parts. This is the difference between automated and anaesthetic experiences, and a phenomenology of enjoyment that an aesthetic experience can afford.

The Semiotics of Embodied Interaction… Horizon for Understanding

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“For Umberto Eco, for example, meaning is no longer an individual construct, as in the arbitrary semiotics of Saussure. It is now seen as the result of a process in which an individual takes part in society through the coding and decoding of his/her relationship with the cultural values and societal norms of the time.” (p72)

Annotation

Umberto Eco’s semiotics is far more embodied with an active relationship for a person, in their background culture and society, in interpreting the coding and de-coding of the norms for understanding. This makes any meaning arrived at not as an individual construct but one that is constructed from both the individual’s internal pre-understanding and external socio-cultural factors that form a horizon for the understanding. For Palmer (1969) the pre-understanding of the individual and the horizonality of the context in which the individual is interpreting the meaning dictate the terms in which any meaning is constructed.

The Semiotics of Embodied Interaction… Intentionality

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“The problem of inter-subjectivity is that, while we can each experience the world in an ontological sense, how can we know that each of us is experiencing the same things, given that we do not have direct access to each other’s thoughts and experiences. (…) Dourish posits that intentionality sets up the relationship between embodied action and meaning (Dourish, 2001, p138).” (p38)

Annotation:

Harman (2007) posits that phenomenology can re-establish itself “by expanding the concept of intentionality to the point where it covers the entirety of the things themselves, thereby freeing us from the growing staleness of the philosophy of human access” (p123). Dourish (2001) sees in the Husserlian distinction between act and matter that the manifested intentionality of acting over the matter, of interacting, holds a relationship between the meaning of the interaction and the embodied action of the interaction. O’Neill (2008) concisely reminds that phenomenology offers a philosophy and methodology to reveal what others see as much as possible considering problem of a metaphysical state of being only being existentially experienced.  

Guerilla Metaphysics… Intentionality

HARMAN, G. (2007) Guerilla Metaphysics: Phenomenology and the Carpentry of Things. Open Court Publishing Company.

“The way to revive phenomenology is not through external rituals of compliance with Husserl’s vocabulary, but by expanding the concept of intentionality to the point where it covers the entirety of the things themselves, thereby freeing us from the growing staleness of the philosophy of human access. (…) Instead of maintaining the usual focus on categorical intuition, so favoured by disciples of Heidegger, I propose that we examine the simple Husserlian distinction between act and matter.” (p23)

Annotation:
Harman (2007) posits that phenomenology can re-establish itself “by expanding the concept of intentionality to the point where it covers the entirety of the things themselves, thereby freeing us from the growing staleness of the philosophy of human access” (p123). Dourish (2001) sees in the Husserlian distinction between act and matter that the manifested intentionality of acting over the matter, of interacting, holds a relationship between the meaning of the interaction and the embodied action of the interaction. O’Neill (2008) concisely reminds that phenomenology offers a philosophy and methodology to reveal what others see as much as possible considering problem of a metaphysical state of being only being existentially experienced. 

The Semiotics of Embodied Interaction… From Concealed Potential to Visual Targets of our Conscious Use

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“As Michael Wheeler points out, most expertly, there are not just simply two ways of interacting with the world that surrounds us. These two poles provided by Heidegger, represent the extreme ends of a spectrum of interaction opportunities that blend into one another (…) where we are often shifting quickly from one mode to the other in a very inexpert ‘unready to hand’ way. What is perhaps most important about this approach to understanding our relationship to things in the world and interactive technologies in particular is that, while it does not deny that we might have some form of representational knowledge about the world, it focuses on our connection to the world through our bodies in the first instance, promoting the idea that we are so connected to our surroundings that we need not build mental models of the world around us in order to act, but that we might simply act through a ‘direct’ relationship to them. This is very different from the original cognitive approach to HCI” (p36)

Annotation
Ready-to-hand and present–at-hand interactive opportunities are opposite ends of ways of interacting within the world. They present an interesting directness to interaction, as they suggest that our relationship to the objects we use changes from concealed potential to visual targets of our conscious use. How this is approached can bypass tacit knowledge leading to an embodied response. As understanding is not always articulated through language it can be an embodiment leading to direct action. This direct action can be articulated and expressed through an interpreted meaning being derived from the calls to action that prompted the interaction.

Beneath Interpretation… Distinction Between Understanding and Interpretation

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“we can only test our prior understanding by subsequent interpretation (…). Considerations of this sort have led Gadamer and other hermeneutic universalists to the radical claim that ‘all understanding is interpretation.’ But this claim, I have argued, is not only uncompelling but misleading in suggesting that we can never understand anything without interpreting it. For in many cases we are simply satisfied with our initial understanding and do not go on to interpret; there are always other and usually better things to do. Moreover, if we could never understand anything without interpreting it, how could we ever understand the interpretation itself? It, too, would have to be interpreted, and so would its interpretation, and so on ad infinitum. As Wittgenstein notes, ‘Any interpretation still hangs in the air along with what it interprets.’ Interpretation must ultimately depend on some prior understanding, some ‘way of grasping…which is not an interpretation.’ This is just a point of philosophical grammar about how these notions are related: understanding grounds and guides interpretation, while interpretation enlarges, validates, or corrects understanding. We must remember that the distinction is functional or relational, not ontological. The prior and grounding understanding ‘which is not an interpretation’ may have been the product of prior interpretations, though now it is immediately grasped. Moreover, it need be an explicitly formulated or conscious understanding, and the ground it provides in not an incorrigible ground.” (p122)

Annotation
Shusterman sees the distinction between understanding and interpretation not as ‘all understanding is interpretation,’ nor a metaphysical state of being (ontological), but as relational or functional. The relationship can be described as, “understanding grounds and guides interpretation, while interpretation enlarges, validates, or corrects understanding” (p122). As he points out if understanding is interpretation and we could never understand anything without an interpretation, then “how could we ever understand the interpretation itself?” (ibid.).  Hirsh (1976) agrees and urges that when the person interprets they first are trying to match what they sense with what they already know in order to first understand before and interpretation and explanation can begin.  Validation is a factor in interpretation and can be placed within systematic process of understanding > construction of meaning > interpretation > validation. When validation is achieved it is “only with respect to known hypotheses and known facts” (Hirsch, 1967, p170). Palmer (1969) points out that in itself understanding is a preliminary act of interpretation from which future interpretations are built. 

Beneath Interpretation… Understanding as a Base and a Guide

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“understanding provides interpretation not only with a meaning-giving contrast, but with a meaning-giving ground. It supplies something on which to base and guide our interpretations, and represents something by which we can distinguish between different levels or sequential acts of interpretation.” (p120)

Annotation

Understanding what is experienced gives a base and guidance to interpreting the meaning. This base and guide can provide interpretation with contrasts by which different levels or sequential acts of interpretation can be distinguished. Hermeneutically this is structured within the circle of interpretation by understanding the parts and the whole of the experience.

Beneath Interpretation… Implying a Conscious Processing

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“In short, I am arguing that though all understanding is selective, not all selective understanding is interpretive. If understanding’s selection is neither conscious nor deliberate but prereflective and immediate, we have no reason to regard that selection or the resultant understanding as interpretation, since interpretation standardly implies some deliberate or at least conscious thinking, whereas understanding does not. We can understand something without thinking about it at all; but to interpret something, we need to think about it.” (p114

Annotation
Shusterman is arguing, “all understanding is selective, not all selective understanding is interpretive” (p114) as since interpretation implies a conscious processing of information to arrive at a meaning, any prereflective and/or immediate understanding does not result in an interpretation. He concludes that, “We can understand something without thinking about it at all; but to interpret something, we need to think about it.” (p114) 

Beneath Interpretation… Pragmatism Meets Phenomenology

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“Interpretation is also practiced and theorized in terms of formal structure with the aim not so much of exposing hidden meanings but of connecting unconcealed features and surfaces so as to see and present the work as a well-related whole.” (p108)

Annotation
The formal structure of interpretation from a pragmatist perspective seeks to connect the parts with the whole experience being interpreted. It is not so much focused on simply revealing the hidden, it is more interested in the connections between the internal and external features of an experience to arrive at a full understanding of its meaning. This separates this from a phenomenological approach that is concerned with revealing the hidden. So pragmatism supports interpretation and synthesised with phenomenology can provide a structure to not only examine an experience through the aesthetic, but also to strengthen the validity of a phenomenological methodology against accusations of being unscientific. Although a full Ihdean postphenomenology is not what is being advocated, there are lessons in the synthesising the best of both philosophies into a practical visual communication methodology. 

Beneath Interpretation… Pragmatists and Interpretation

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“Pragmatists, like Nietzcheans, insist on rejecting the very idea of any foundational, mind-independent, and permanently fixed reality that could be grasped or even sensibly thought of without the mediation of human structuring. Such structuring or shaping of perception is today typically considered to be interpretation, and so we find contemporary pragmatists like Stanley Fish repeatedly insisting that interpretation comprises all of our meaningful and intelligent human activity, that ‘interpretation is the only game in town.’ All perception and understanding must be interpretation, since ‘information only comes in an interpreted form.’ Thus, even in our most primitive and initial seeing of an object, ‘interpretation has already done its work.’” (pp103-104)

Annotation
Through a pragmatist perspective all perception and understanding arises out of a mediated human structuring into meaningful interpretation. Pragmatists reject a reality that is independent of the mind, foundational and permanently ‘fixed.’ Interpretation is the conduit through which we construct our reality.

Hermeneutics: Interpretation Theory… Visually Revealing the Whole and Parts of an Experience

PALMER, R.E. (1969) Hermeneutics: Interpretation Theory in Schleiermacher, Dilthey, Heidegger, and Gadamer. Evanston: Northwestern University Press.

“For the theory of interpretation, then, it makes a great difference whether thinking is conceived strictly in ideational terms, for then interpretation itself is dealing not with an unknown matter which has to be brought to light but with the clarification and evaluation of already known data. Then its task is not the primary ‘showing’ of the thing but that of achieving correctness among several possible interpretations. Such presuppositions tend to keep one always in clear light of what is already known instead of bridging the gap between light and darkness.” (p146)

Annotation:
In this context interpretation is framed within a phenomenological inquiry into visually revealing the whole and parts of an experience, bringing to light the meaning of actions within. It is not about interpreting what is already revealed in a new way. As Palmer puts it, “its task is not the primary ‘showing’ of the thing but that of achieving correctness among several possible interpretations” (p146).