The Semiotics of Embodied Interaction… Ready-to-hand and Present-at-hand Modes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Interacting with the media of all kinds then falls between these two modes of being. On one hand we perceive and encounter our media-rich environment directly, we manipulate it and transform it through our-ready-to-hand mode of being. On the other hand, we are constantly viewing, reading and interpreting mediated information through the reflective mode of encountering it as present-at-hand. At the same time we move between these two modes of being as we inscribe, interpret, transcribe and transform our mediated environment. In a very clear sense we are deeply entwined physically with the media in our environment and in another we are constantly making, and making sense of, the inscriptions that the media environment affords us.” (p140)

Annotation

An experience fluctuates between a ready-to-hand mode of being and a present-at-hand mode. The former involves the perception, interpretation, action and physically embodied manipulation using what is at hand in the experience of interacting within the present environment the person finds themselves in. The latter a more reflexive interpreting state of scanning what is available, making sense of the semiotic messages inscribed within the present environment the person finds themselves in.

The Semiotics of Embodied Interaction… Perceiving and Interpretation of the Relevant Calls to Action

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“screen-based interactive media are extremely semiotic in character. The symbols, graphics, and pictograms, as well as the layout and structure of its emerging forms, are all related to the remediation of older convergent media (…). The key difference is that it not only has to be perceived and interpreted to understand what it is but it also has to be used and manipulated to reveal what it does. The logic of immediacy is strong here in that much of the way in which interactive media is presented is often a simulation of previous, physical real-world media forms, e.g., drawing packages and word processing. However, the way in which those representations are conceptually structured often has to be very different from the older version, in order to take advantage of the computational aspects of remediation.” (p105)

Annotation
The visual communication of affordances through the graphical user interface naturally involves perceiving and interpretation of the relevant calls to action at any one time. But it also involves communicating through user manipulation, what it does. To enact the desired behaviour in the user the visual communication has to attract attention, retain attention, communicate action, suggest how the affordance is to be manipulated and suggest the outcome. This is more than simply designing a “button.” The selection of the semiotic signs involved in this communication must appeal and make sense to the target audience, and that congruently make sense in the design. As Eco suggests there is much more to consider in the design of an interface’s visual communication, “The Principle that form follows function might be restated: the form of the object must, besides making the function possible, denote that function clearly enough to make it practicable as well as desirable, clearly enough to dispose one to the actions through which it would be fulfilled” (Eco, 1986, p63 cited in O’Neill, 2008, p113). 

The Semiotics of Embodied Interaction… Variance and Invariance

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“In Gibson’s view of perception, an active perceiver picks up and distinguishes between variant and invariant information. Either of these may be information about the perceiver or the environment but the distinction between the variant and invariant builds to create an awareness of the perceiver’s place within that environment.” (p61)

Annotation
The relationship between ourselves in the world, and the environment that surrounds us at any one time, is an important perceptual one. The environment is always perceptively relative to the person in that environment. They are aware that if they move, spatially and temporally that environment persistently envelope them. The perceptual changes within that environment are what guide and suggest options. These perceptual changes can be described as variant and invariant. In other words, some information within the environment stays constant and some will vary allowing behaviour change to be enacted. To give a simple example of variance and invariance it may be useful to look at an interactive experience on a computer. During any interaction the keyboard, screen and device will always be there (invariant) despite what is happening on the screen or whatever input device is being used (variant). The changes perceived between what is in the environment (keyboard, screen, device, table, room etc.) and what is changing within that environment is directly attributed to the user’s sense of being in the interaction. Whether this experience can be described as an aesthetic or an anaesthetic experience is another matter. What is important is the distinction between the two states and how this helps the conscious and subconscious directly as embodied knowledge perceive what is happening and can happen next.

The Semiotics of Embodied Interaction… Affordances

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Norman separates ‘real’ affordances (the physical properties of the world) from ‘perceived’ affordances (subjective representations in the mind), and is more concerned with the perceptual properties of affordances rather than the actual properties of real objects themselves. (…) The strength of Norman’s version of how perception and affordance work is that it attempts to solve the problem of how to explain the role of knowledge in understanding the world around us. However, the problem with Norman’s version of affordance is that it abandons the unique contribution of Gibson’s ideas in bridging the gap between the object and the subject.” (p51)

Annotation

Prof. Don Norman classifies different forms of affordances into those that are properties of the physical world (real), and those that are in the mind and are representative and subjective (perceived). It is the perceived affordances that Norman is more interested in and in doing so he distances himself away from Gibson’s original affordance concept. Lately Norman has refined his ideas on affordances now, unfortunately, referring to them as signifiers. In doing so he has moved affordances closer to semiotics but also now muddied the clear separations between the two. Affordances are about communicating potential action while semiotics defines how the visual communication will communicate the call to action. An example could be a water tap. An affordance would communicate how to get water from the tap. Whether the water is hot or cold is communicated through the semiotic signs of red or blue. The outcome of the semiotics isn’t the pouring water, but the communication of the expected temperature. The outcome of the affordance is how the tap is operated to access the pouring water. Both help in the ultimate outcome of accessing water but perform different tasks.

The Semiotics of Embodied Interaction… Perceptual Changes

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“[Gibson’s] Theory of Ecological Perception begins by considering the relationship between living entities that perceive things in the world and the environment that surrounds them. (…) Invariant information (…) specifies the persistence of the environment and of oneself. (…) The perceiver is then aware of her own existence in a persistent environment relative to her own and others’ movements within the environment. This awareness provides us with information about the possibilities for action that the world around us directly affords. (…) Affordances then (…) are an emergent property of the physical relationship between environment and the direct perceptual acts of embodied beings.” (p50)

Annotation

The relationship between ourselves in the world, and the environment that surrounds us at any one time, is an important perceptual one. The environment is always perceptively relative to the person in that environment. They are aware that if they move, spatially and temporally that environment persistently envelope them. The perceptual changes within that environment are what guide and suggest options. How these guides and options within an environment communicate to the person is described as affordances, “an emergent property of the physical relationship between environment and the direct perceptual acts of embodied beings.” (O’Neill, 2008, p50). The calls to action that are afforded are defined within the semiotics of the visual communication.

The Semiotics of Embodied Interaction… Phenomenological Relationships

O’Neill, S. (2008) Interactive Media: The Semiotics of Embodied Interaction. London: Springer-Verlag.

“Heidegger makes explicit the idea that the nature of our being-in-the-world is not only related to the world we inhabit, but to the fact that we are aware that we exist in that world. In short, a fundamental aspect of our being-in-the-world is that we are concerned about (aware of) ourselves being-in-the-world. Furthermore, another aspect of Heidegger’s conception of ‘Being’ is that our being-in-the-world is a being-with-other-beings (being as people, things or entities that exist in the world around us). Those other beings may or may not have a concern for being with us, but we most definitely have a concern for being-with them as they have an effect on how we exist. Our consciousness and our knowledge of the world are deeply entwined with our phenomenological relationship to it.” (pp34-35)

Annotation
O’Neil in building his case for the semiotics of embodied interaction naturally takes a Heideggerian perspective seeing “our consciousness and our knowledge of the world [as being] deeply entwined with our phenomenological relationship to it” (p35). Our interconnectivity with others shapes how we see and interact with the world we perceive as our reality. 
 

Beneath Interpretation… No Understanding is Foundational

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“Since our current hermeneutic turn derives in large part from the rejection of foundationalism, it is not surprising that the central arguments for hermeneutic universalism turn on rejecting foundationalist ideas of transparent fact, absolute and univocal truth, and mind-independent objectivity. For such ideas underwrite the possibility of attaining some perfect God’s-eye grasp of things as they really are, independent of how we differently perceive them, a seeing or understanding that is free from corrigibility and perspectival pluralities and prejudices that we willingly recognize as intrinsic to all interpretation. I think the universalists are right to reject such foundational understanding, but wrong to conclude from this that all understanding is interpretation. Their mistake, a grave but simple one, is to equate the nonfoundational with the interpretive. In other words, what the universalists are successfully arguing is that all understanding is nonfoundational; that is always corrigible, perspectival, and somehow prejudiced or prestructured; that no meaningful experience is passively neutral and disinterestedly nonselective. But since, in the traditional foundationalist framework, interpretation is contrasted and designated as the form of nonfoundational understanding, the inferior foster home of all corrigible, perspectival perception, it is easy to confuse the view that no understanding is foundational with the view that all understanding is interpretive. Yet this confusion of hermeneutic universalism betrays an unseemly residual bond to the foundationalist framework, in the assumption that what is not foundational must be interpretive.” (pp108-109)

Annotation

There are intrinsic prejudices within interpretation as we all perceive within our own reality, collectively mediated within the socio-cultural conditions we share. Hermeneutic interpretation rejects any foundationalist ideas that there is an absolute truth, for a pluralism of perspectival truth that can be corrected, rectified, or reformed with fresh data. But there is a danger in assuming that all understanding that does not have a foundational base has to be interpretive. Prof. Shusterman (1991) cautions that a pragmatic universalist perspective on understanding as nonfoundational should not confuse “the view that no understanding is foundational with the view that all understanding is interpretive” (pp108-109). Equating the interpretative with the nonfoundational has a danger of seeing all understanding as interpretation. Palmer (1969) does have the distance to see understanding as a preliminary act of interpretation, and Bohman (1991) from a holistic perspective acknowledges that a ‘correct’ functionalist or deterministic interpretation cannot explain how humans process and experience the world.

Beneath Interpretation… Pragmatists and Interpretation

SHUSTERMAN, R. (1991) Beneath Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp102-128

“Pragmatists, like Nietzcheans, insist on rejecting the very idea of any foundational, mind-independent, and permanently fixed reality that could be grasped or even sensibly thought of without the mediation of human structuring. Such structuring or shaping of perception is today typically considered to be interpretation, and so we find contemporary pragmatists like Stanley Fish repeatedly insisting that interpretation comprises all of our meaningful and intelligent human activity, that ‘interpretation is the only game in town.’ All perception and understanding must be interpretation, since ‘information only comes in an interpreted form.’ Thus, even in our most primitive and initial seeing of an object, ‘interpretation has already done its work.’” (pp103-104)

Annotation
Through a pragmatist perspective all perception and understanding arises out of a mediated human structuring into meaningful interpretation. Pragmatists reject a reality that is independent of the mind, foundational and permanently ‘fixed.’ Interpretation is the conduit through which we construct our reality.

Postphenomenology and Technoscience… Hermeneutic Relationship

IHDE, D. (2009) Postphenomenology and Technoscience: The Peking University Lectures. New York: State University of New York Press.

“Hermeneutic relations. I have always held that human technology (experiential) relations form a continuum. As one moves along a continuum, one finds technologies that engage one’s more linguistic, meaning-orientated capacities. Here, while the engagement remains active, the process is more analogous to our reading or interpreting actions than to our bodily action. There are hints of this in Heidegger’s example of the old style turn signal on old European cars, a pointerlike device that pops up and points as a signifying artefact. Writing, of course, is itself a technology, and it is one of the rare examples partially analysed by Husserl as a technology that changes one’s sense of meaning. But my own earlier examples were drawn from instrument readings. Instrument panels remain ‘referential,’ but perceptually they display dials, gauges, or other ‘readable technologies’ into the human-world relationship. And while, referentially, one ‘reads through’ the artefact, bodily-perceptually, it is what is read.” (p43)

Annotation:
Ihde discusses an experiential continuum in human technology, where the process of active engagement in a technology is more meaning-orientated. It is more about interpreting actions than conscious bodily action. Perceptively the information in the visual affordances and calls to action are referentially ‘read through,’ what is read is ‘bodily-perceived’ for the action to be performed. There is a hermeneutic relationship here that pragmatically is practical but can be revealed phenomenologically. 

Postphenomenology and Technoscience… Hybrid Philosophy

IHDE, D. (2009) Postphenomenology and Technoscience: The Peking University Lectures. New York: State University of New York Press.

“Postphenomenology is a modified, hybrid phenomenology. On the one side, it recognizes the role of pragmatism in the overcoming of early modern epistemology and metaphysics. It sees in classical pragmatism a way to avoid the problems and misunderstandings of phenomenology as a subjectivist philosophy, sometimes taken as antiscientific, locked into idealism or solipsism. Pragmatism has never been thought of this way, and I regard this as a positive feature. On the other side, it sees in the history of phenomenology a development of a rigorous style of analysis through the use of variational theory, the deeper phenomenological understanding of embodiment and human active bodily perception, and a dynamic understanding of a lifeworld as a fruitful enrichment of pragmatism. And, finally, with the emergence of the philosophy of technology, it finds a way to probe and analyse the role of technologies in social, personal, and cultural life that it undertakes by concrete – empirical – studies of technologies in the plural. This, then, is a minimal outline of what constitutes postphenomenology.” (p23)

Annotation:
How postphenomenology is useful is that it is a hybrid philosophy, and as such it has solved some of the criticisms of phenomenology as being antiscientific and “locked into idealism or solipsism” (p23) by turning to the strengths of pragmatism. Phenomenology offers a “rigorous style of analysis through the use of variational theory” (ibid.) that coupled with pragmatic structures enriches how embodiment and the perception of the human body are understood. 

The Aims of Interpretation… Kant & Lebenswelt

HIRSCH, E.D. (1976) The Aims of Interpretation. Chicago: The University of Chicago Press.

“Kant’s procedure shows very clearly that it was he and not the twentieth-century phenomenologists who first conceived the value-laden objects of cultural experience as epistemological ultimates. It was he, not ourselves, who first discovered the irreducible connection of ‘fact’ and ‘value’ in the Lebenswelt [_{world of lived experience}]. For Kant argues that the objects of cultural experience, no less than those of primary sensation, are constituted by the mental organization of the perceiver.” (p101)

 

Annotation:
It is not important here to enter an argument of who came up with what, but Kant’s argument that the objects of our socio-cultural experience, on which our interpretations are based, are attributed meaning by the perceiver. The objects in themselves do not hold meaning until they are perceived. 

Holism without Skepticism… Holism Framed Interpretation

BOHMAN, J.F. (1991) Holism without Skepticism: Contexualism and the Limits of Interpretation. In: D.R. HILEY, J.F. BOHMAN, and R. SHUSTERMAN (Eds.) The Interpretive Turn. Cornell University Press. pp129-154

“the perspectival and contextual character of interpretation leads to the most common form of scepticism about interpretation: since we can interpret thing only from ‘our’ point of view, our interpretations are inevitably ethnocentric. It is impossible to understand others as they understand themselves: we understand them only according to ‘our own lights.’ My aim here is to dispute the common philosophical basis for all such views that assert that there are such inherent, contextual limits on how we interpret others.” (pp130-131)

Annotation:
Phenomenology attempts to reveal and understand the individual’s perception of an experience. Bohman is using a transcendental Holism that is useful to a phenomenologist as it frames an understanding on the nature of interpretation. Coming from a nursing perspective Holism treats the whole person rather than just the problem. The theory is that there is an interconnection between the parts that constitute the whole. With such an interconnection the parts cannot be wholly understood without understanding the whole. This is circular, reiterative and integrated. Therefore this is helpful to a phenomenological hermeneutic circle being applied to interpret and understand how people perceive and experience an interaction in their lives. 

Representational Theory… Inauthentic Mediated World

KENNEY, K. (2005) Representational Theory. In: K. SMITH, S. MORIARTY, G. BARBATSIS and K. KENNEY (Eds) Handbook of Visual Communication: Theory, Methods, and Media. New Jersey: Lawrence Erlbaum Associates. pp99-116.

“Behaviourists attempt to observe and measure the real world directly. Phenomenologists are exclusively interested in a person’s introspective experience. Semioticians and rhetoricians try to understand the linkages between our internal world and the external world, and that linkage is necessary, they believe, because the external world is always mediated by our senses and our mind. Whereas rhetoricians have investigated how humans create and manipulate symbols in order to persuade other humans, semioticians have been more interested in how humans (and other animals) interpret all kinds of signs, including symbols, that were created by other people, as well as natural signs, including symbols, that were created by other people, as well as natural signs that may have resulted from plants, animals, or inorganic matter. Both rhetoricians and semioticians, therefore, are concerned with how ‘signs’ mediate’ between the external world and our internal ‘world,’ or how a sign ‘stands for’ or ‘takes the place of’ something from the real world in the mind of a person.” (p99)

Annotation:

We exist in a mediated world that in the digital age is sometimes a re-mediated world. This existence can be framed as inauthentic (a Heideggerian concept). How someone navigates and orientates their way through this mediation, making sense of and interacting in their world is important to understand for interaction design. There are a variety of qualitative methods to observe and reveal this processing. Kenney discusses what aspects Behaviourists, Phenomenologists, Semioticians, and Rhetoricians are interested in studying, and where they overlap or conflict with each other.

The main grouping between semiotics and rhetoric focus on the relationship within the internal and external world of the processing of semiotic signs, how the message is (re)mediated, perceived and understood. Behaviourists obviously are interested in the exhibited behaviour within the world, of the manifestation of how this process affects them. Whereas the phenomenologist is interested in the internal processes of how they experience the perception, understanding and reaction to the message that changes their behaviour.

Visual communication in this context is concerned with both the internal and external processing of the semiotic-facilitated meaning. It is concerned with the shaping of the semiotics externally in the design, but can also be used to reveal and visualise the phenomenological internal processes of the experience, to “convert the sheer impact of the world into pictures of simulacra” (Harman, 2007, p#)

Aesthetics Theory… Visual logic

DAKE, D. (2005a) Aesthetics Theory. In: K. SMITH, S. MORIARTY, G. BARBATSIS and K. KENNEY (Eds) Handbook of Visual Communication: Theory, Methods, and Media. New Jersey: Lawrence Erlbaum Associates. pp3-22.

“Visual logic is defined here as a system of visual relationships that encourages a developed internal sense of image cohesiveness, integrity, and elegance necessary to attract attention and guide the viewer to a sense of aesthetic completion and comprehension. Six individual, interlocking principles guide the development of this sense of visual logic. 

These six principles are 
(a) Ambiguity and Meaning, (b) Control of Direction, (c) Ecological Relationships, (d) Tensional, (e) Unity, and (f) Realism.

(a) Ambiguity and Meaning

(…) The ambiguity and meaning principle suggests that, if aesthetic form and content are effectively related, the visually literate viewer is able to extract relevant information by concentrating on visible relationships and the nature of meaning-making in the human brain. Multiple meanings emerge later from the layered relationships with the other five principles.

(b) Control of Direction

(…) Individual viewers will arrive at some degree of individual connection and relationship to the image based on their individual wants, needs, and expectations. The individual image-maker can control many aspects of ‘how’ the viewer’s perceptual system interacts with visible relationships but not the final interpretation. The maker can direct the viewer’s eye throughout the image’s significant areas through directional clues, groupings of elements, and tensions. The maker can also choose to emphasize certain aspects of the image and to subordinate others in order to direct the viewer’s mind to significant matters. However, the sheer number of visible relationships will quickly overwhelm the parsing and decision-making functions of the brain. (…)

(c) Ecological Relationships

When interpreting the inherent visual logic imbedded in an image, the maker (during the act of ideation) and the viewer (in a process of perception) both become involved in an intimate reciprocal and ecological [The study of living things, their environment, and the relation between the two.] relationship with the image. The visual structure offers parameters for perception of possible multiple meaning levels, as well as the potential for fixation on one particular meaning or focused aesthetic response. The maker, through trial and error, shapes this image, and the viewer must empathize with this relationship while at the same time understanding his or her ecological interactions with the structure. (…)

(d) Tensional

(…) the visual elements on the picture plane’s flat surface have tensional relationships with each other, with sides of image, and with its center of the image. The sum total of all tensions, both explicitly perceived and implicitly apprehended, creates an aesthetic impression of mood and a visible foundation for the communicated message. (…)

(e) Unity

(…) Without unity, the experience for the viewer lacks cohesion, making communicative interpretation less sure. With unity, the viewer’s own natural perceptual abilities can interpret visual clues to determine the nature and type of visual message and its relationship to reality.

(f) Realism

Application of the realism principle provides a ground of information that is either believable and true or false and manipulative. Informed visual communication must consider the medium of realism selected by the message designer and its aesthetic and affective dimensions created within the viewer.

” (pp16-18)

Annotation:

The six principles of a Visual Logic mirror a semiotic structure built on perception, interpretation and meaning. The six principles of (a) ambiguity and meaning, (b) control of direction, (c) ecological relationships, (d) tensional, (e) unity, and (f) realism are complimentary to the semiotic process.

A semiotic meaning-journey follows this path Sender > Intention > Message > Transmission > Noise > Receiver > Destination. Principle (b) control of direction sits comfortably within the area that the Sender occupies. The designer is the sender, and as creator they can “direct the viewer’s eye throughout the [design’s] significant areas through directional clues [visual affordances], groupings of elements [visual hierarchy], and tensions [white space, layout, typography, image and colour].”

The Intention, the Message itself and the Transmission of it affect and are affected by the principles of (a) ambiguity and meaning and (c) ecological relationships. The designer’s intention reflected in the design is to facilitate the transmission of the message to the receiver (user). The “visible relationships” that a visual hierarchy communicates helps the user to “extract relevant information” from interpreting the meaning of these relationships. This process of interpretation is a “intimate reciprocal and ecological relationship” with the semiotics in the design visually communicating the intended message(s) at that particular moment in the interaction with the design.

The principles of (d) tensional and (e) unity clearly affect the Transmission of the Message and contribute to the Noise that can prevent the understanding and interpretation of the message. Principle (a) ambiguity and meaning also can become noise if the aesthetic form is conflicted. If the message(s) being communicated semiotically through the visual communication of the interaction are not cohesive and understandable to the user; or if the “natural perceptual abilities [of the user cannot interpret the] visual clues to determine the nature and type” of the visual message then miscommunication will occur. The noise that prevents the communication is a lack of unity between the message, the chosen semiotic sign, and the transmission of it. This noise could be attributed to the design choices in the layout and visual mood of the design. These elements are what are referred to as tensional in the six principles of Visual Logic.

Finally the Destination and Receiver are affected by the principles (e) unity and (f) realism. The level of or lack of unity in the semiotics of the design obviously has a direct impact on the user (receiver) once the design outcome has reached its destination (being used). The realism in the visual communication rests with the perception of the user as to whether they trust that the design will let them achieve whatever outcome they require (contextual to the genre the design operates in), such as does a corporate design reflect the business? does the design of an eCommerce website communicate trust?