Philosophy for Multisensory Communication and Media

ACKNOWLEDGMENTS

Several people have contributed to my thinking about multisensory media theory, but I want to particularly thank three individuals who have greatly helped me to write this book.

Tara Mortensen, a faculty member in the School of Journalism and Mass Communications at the University of South Carolina (USC), was my primary reader concerning the what of the book’s contents. She is a colleague in the Visual Communication sequence at the School of Journalism and Mass Communications. Even though my initial drafts were unclear—due to unclear thinking as well as writing—Tara persevered and offered invaluable comments. All authors need someone to read early stages of their manuscripts and challenge ideas, note gaps, and, if deserved, offer encouragement. Tara Mortensen’s research focuses on amateur visual media and visual framing in the era of citizen photojournalism.

Jane O’Boyle not only offered lots of encouragement, when it was most needed, but she also was my primary reader concerning the how of the book’s writing. She is a doctoral student in USC’s School of Journalism and Mass Communications whose research focuses on digital media and international communications. More importantly, for me, she is a former publishing executive at Random House and also author or editor of more than 30 books. If ← vii | viii → you find this book flows well and is relatively easy to understand, then much of the credit goes to Jane.

Augie Grant, a professor and endowed chair in the same school, agreed to read the manuscript as it was in the process of being written. He not only suggested including ideas that I would otherwise have overlooked, but he also challenged several of my ideas, which spurred me to rethink and/or rewrite sections.

I also sincerely thank the contributors to Philosophy for Multisensory Communication and Media. In addition to Jane O’Boyle and Tara Mortensen, writers of sidebars include Katherine Beth LaPrad, Nicholas David Bowman, and Jaime Banks.

Katherine LaPrad is a doctoral candidate in USC’s School of Journalism and Mass Communications. She analyzes news content and practices through cultural, critical, and historical lenses, focusing her research on the role of journalism in social movements and social change. She uses a range of analytical methods, including qualitative and quantitative content analysis, to study the past, collective memory, and the present.

Nicholas David Bowman (Ph.D., Michigan State University) is an assistant professor of communication studies at West Virginia University, where he works as a research associate in the Media and Interaction Lab (#WVUCOMMIL). His work focuses on the psychology of entertainment media and how our interactions with and through technology impact our thoughts, actions, and feelings.

Jaime Banks (Ph.D., Colorado State University) is an assistant professor of communication studies at West Virginia University and a research associate in WVU’s Media and Interaction Lab (#WVUCOMMIL). Her work focuses on the nature of identity and embodiment in relation to interactive digital media, and how users have relationships with their communication technologies.

And special thanks to Ted Wachter, a philosophy enthusiast who made some useful comments. ← viii | ix →

FOREWORD

Fortunately, I discovered quite a long time ago that what I enjoy most in life is learning. You and I learn by doing, by listening, by watching (foreign) films, and by reading. As a professor, I also have opportunities to learn by teaching. Not only do I learn when preparing for class, but I learn from my students during seminars. I also learn from my service to the university, profession, and academy. I seek service opportunities such as conducting workshops for groups outside the United States. I learn the most, however, when fulfilling my responsibilities for scholarship. And this book has been the highlight of my life in terms of learning (and, perhaps, of scholarship).

When I started writing this book, I considered myself a visual communications scholar. I read the journals and books pertaining to visual communication and visual culture. I also read some of the articles and books pertaining to visual literacy, visual anthropology, and visual sociology. And based upon the references in these articles/books, it seems as if I had read many of the scholarly works that my peers had written. I felt as if I had kept up to date with my field.

But it seemed to me that the theories in “my” field of visual communication were dated. Either my academic reading efforts were not as thorough as I had thought, or not many theories were being created, especially theories that ← ix | x → would explain the changes brought about by the Internet and smartphones. I wanted to learn about the newer theories, so I searched for “visual communication” and “theory or theories.”

I already knew that the term “visual communication” is misleading. When communication relies upon the sense of sight, it is called nonverbal communication, not visual communication. Nonverbal communication means that people use eye movements, facial expressions, gestures, and body language to express and to perceive meanings. The term “visual communication” really means “visual media,” which include drawings, paintings, and photographs. The term “visual media” also refers to multimedia, such as newspapers, magazines, and websites, and hybrid media such as television, film, and animation. Hybrid media, such as television and film, are auditory as well as visual media, so instead of defining myself as a visual communication scholar, I should have called myself an audio-visual scholar. But then I discovered haptic media and I began to learn about olfactory media. I wondered if gustatory media also existed. Eventually I came to understand that what I’m really interested in is multisensory media.

A Note on Names

I draw upon the work of dozens of philosophers, some of whom readers may know, but other philosophers may be unfamiliar to readers. To distinguish philosophers from non-philosophers, I provide the following cues. Names initially followed by dates of birth and death—for example, “John Dewey (1859–1952)”—indicate that the name belongs to a deceased philosopher. For contemporary philosophers, I write “philosopher” and the country in which the philosopher works—for example, “Jean-Luc Nancy (French philosopher).” For philosophers working in the United States, I add the university that employs them—for example, “Raymond D. Boisvert (U.S. philosopher at Siena College).” When I refer to non-philosophers, I identify their profession and do not give dates of birth and death—for example, “Sharon Oviatt, a highly respected scholar who specializes in mobile and ubiquitous interfaces and multimedia.” ← x | 1 →

· 1 ·

INTRODUCTION

Although some media are in decline, especially media that are printed and delivered via the postal system or by someone driving around different neighborhoods and throwing newspapers out of car windows, the use of digital media is very robust. Computers connected to the Internet and phones connected to cell phone towers seem to be in constant use. But I am less concerned with the amount of time we are using media and more concerned with the type of media being used. I argue that instead of multimedia, we are using hybrid media, and in the near future we’ll be using multisensory media.

In multimedia, the content of auditory media and visual media appear next to each other. A web page, for example, may have text next to photographs, which are next to videos. Similarly, a PowerPoint file may display text on a page and have additional types of media inserted into the page. On both web and PowerPoint pages, each medium retains its own file format and its own way of accessing and editing its data. Even though we may look at a photograph while listening to an audio story, the photo and audio are separate.

Hybrid media are different because disparate media are merged together. With films, for example, we always hear the sounds synchronized with the pictures. An example of a newer and more complex hybrid medium is Google Earth, which combines aerial photography, satellite imagery, 3D computer ← 1 | 2 → graphics, still photography, and other media to create what Google engineers call a “3D interface to the planet” (Manovich, 2013: 163). Another example is the type of motion graphics sequence seen at the beginning of James Bond movies. These animated visuals may include content and techniques from live-action video, 3D computer animation, 2D animation, and painting.

Before hybrid media are merged together, each medium can be edited separately. Today, when I edit video, the software program I use has a timeline that shows the duration of each shot. A shot includes synchronized audio and video, and during editing, the audio can be separated from the video. In fact, the timeline includes at least one track for the moving pictures and multiple tracks for the audio. For example, there may be separate tracks for natural sound, dialogue, music, sound effects, and so on. When I export my work, however, everything is merged into one file with one file format.

Multisensory media are hybrid media that engage more than the auditory and visual senses. Eventually it will be possible to develop applications that integrate and synchronize all of the senses. Instead of having separate tracks for audio and visual, there will be separate tracks for audio, visual, haptic, olfactory, and gustatory information. We will be able to edit each track separately, keep them in sync or not, and then merge the tracks into a single file with its own format.

Multisensory media are used for the purposes of communicating at a distance, learning, and entertaining. When two people are apart, for example, they can use video calls to share news and maintain their relationship. With the Apple Watch, people can also send each other different types of vibrations or forces. Soon people will also be able to use their smartphones to share scents. In order to help people learn, haptic and olfactory artworks are beginning to appear in museums. And simulators for military training include all of the senses except the gustatory sense, but that may be incorporated in the future. In the entertainment area, video game players are using their auditory, visual, and haptic senses. Some game developers have experimented with adding scents to gamers’ experiences.

As multisensory media improve, I believe we will reconsider the purpose of communicating. When we send text and email messages, we are using media to exchange information in the form of words and numbers. When we call each other, we also exchange information, but our tones of voice indicate how we are feeling. With video calls we do all of the above and we see emotions in others’ facial expressions, gestures, and body language. But if we could share scents, which are strongly associated with emotional memories, ← 2 | 3 → then we would have an additional way of sharing personal experiences and maintaining emotional ties. And if we could touch one another, then we would feel connected both literally and metaphorically. Finally, if we could share the flavors of the foods we are about to eat as well as the drinks we are making, then we could trigger nostalgic feelings, share our culture, and arouse emotions. As we add senses to our “conversation,” in other words, the exchange of information continues, but we gain more and more ways of sharing our emotions. We also gain an increased sense of social presence, which means we are not only aware of each other physically, but we are also psychologically involved. Social presence is valuable for maintaining relationships, especially when people are seldom able to meet each other face-to-face. Communication, therefore, may shift from primarily exchanging words for the purpose of informing and persuading, to sharing experiences for the purpose of bonding.

At this moment, however, challenges remain. Hardware for capturing and emitting different types of sensory data are still being developed. So is software for mixing and synchronizing the sensory data. In addition, researchers in industry and academia are trying to create research methods in order to study multisensory experiences. Theories of multisensory media experiences are also needed.

This book attempts to lay a foundation for building theory concerning multisensory media. To create this foundation, I draw upon the ideas of philosophers who write about sensory perception as well as each of the senses. I also draw upon philosophers who write about media such as painting, photography, film, music, dance, and interactive art. Of the 50 or so philosophers mentioned in this book, almost 30 are alive and are either still writing or have had work published recently. I synthesize their ideas in order to make a series of points applicable to multisensory media.

Several audiences may be interested in this book. One group of potential readers is people who are interested in new philosophical ideas about the senses and media. Another is people interested in multimedia, hybrid media, and multisensory media who either (a) want ideas about how to take full advantage of sensory perception when they create user experiences, or (b) want to use/create theory in order to study media. A third group consists of people interested in communication, especially communication theory, because sensory meanings come before language. A fourth group is sensory studies scholars.

Since the turn of the century, scholarly interest in the senses has increased in the social sciences and humanities. One of the products of this increased ← 3 | 4 → interest is that scholars in disciplines such as anthropology, sociology, history, and geography have gathered together and formed a field called sensory studies. These sensory studies scholars are interested in how culture influences the senses and how the senses influence culture. Their articles about the senses and culture often appear in the journal The Senses and Society, which launched in 2006. Philosophy for Multisensory Communication and Media is an expansion of sensory studies to communication and media.

In the remainder of this Introduction, I explain why philosophy is useful for building theory. I also explain relativism, phenomenology, and aesthetics. At the end is a brief overview of the rest of the book.

Using Philosophy to Build Theory

When an area of investigation is unsettled or when dissatisfaction occurs with an existing set of theories, scholars are particularly likely to turn to philosophers for assistance. In The Structure of Scientific Revolutions, Thomas Kuhn (1922–1997) writes, “Scientists have turned to philosophical analysis as a device for unlocking the riddles of their field” (1962/1970: 88). Philosophers are useful in these circumstances because they can propose new—although perhaps incomplete and not scientifically tested—answers to important questions. Scientists themselves may be hesitant to speculate because they are accustomed to building theories by using empirical evidence. In other words, science leads to a slow evolution of ideas, but philosophy can dramatically disrupt old ideas and replace them with new ones.

Philosophy is also useful for building theory because it is “next to” both science and art, writes Brian Massumi (Canadian philosopher). He proposes a continuum with philosophy on the left, science in the middle, and art on the right (Massumi, 2002: 252). Science and philosophy are connected because they inform each other and challenge each other. Scientists test philosophical ideas, and philosophers use scientific findings in order to create new ideas. In fact, science and philosophy need each other. Without science, philosophers’ ideas might be unacceptable to the general population. And “without philosophy, scientists wouldn’t have any problems to solve” (Biggs, Matthen, & Stokes, 2015: 3). Moreover, science and philosophy are often entangled. At any given moment, what we read is probably a mixture of well-supported scientific fact and less-supported explanations pushed forward by one or more philosophers. The connection between science and philosophy ← 4 | 5 → is especially strong when scientists want to understand something that they cannot measure directly because they cannot see, hear, or touch a phenomenon. In studies of the brain, for example, scientists study inputs to the brain, such as materials that people perceive, and they study outputs from the brain, such as people’s behavior, but the human brain itself could not be studied directly until brain scans became possible. Since the brain is still largely a mystery, philosophers have a lot to say about perception, thinking, feeling, and meanings.

Art and philosophy are connected because both stimulate new ways of thinking and feeling. In fact, philosophers Gilles Deleuze, Dennis Dutton, Frances Dyson, and Mark B.N. Hansen, describe artworks and their ideas together in the same book. When they are together it is easier to understand both the artworks and the philosophers’ ideas. In addition, Erin Manning (Canadian philosopher) and Susan Kozel (choreographer who writes about philosophy, dance, and performing arts) make interactive art installations in order to develop their philosophical ideas. Philosophy is thought that is felt, writes Massumi (2002), while art is feelings that are thought. In other words, philosophy makes us feel, while art makes us think.

Philosophers’ ideas are like mutations, writes Gilles Deleuze (1925–1995). Both mutations and philosophers’ ideas lead to change. Sometimes the change helps us, and so ideas become “permanent,” or at least they remain until another mutation of ideas comes along. Other times the new ideas die out. Even though some ideas die, it is important that philosophers continue to create new ideas, because otherwise our behaviors will never change; they won’t evolve. This same line of thinking also applies to artists. Artists’ ideas are also like mutations that lead to change that enable us to think and act differently.

Philosophers meddle, intrude, hamper, and prevent the normal mode of thinking, write Erin Manning and Brian Massumi. These two Canadian philosophers use the metaphor of dropping stones in a pond to explain how philosophers’ ideas interfere (Manning & Massumi, 2014). One stone produces a ripple pattern. If another stone is dropped in the same pond, it creates another ripple pattern. Where the two ripples intersect, new complex patterns form, which are different from the ripples caused by either stone. Philosophy is the second stone dropped into a pond; philosophy creates the new complex patterns. Philosophy, therefore, is a practice of creative destruction; it destroys (more like ruptures) old ways of thinking and creates new ones. ← 5 | 6 →

Susanne Langer (1895–1985) is a good example of philosophical meddling. She challenged a central belief of positivism and behaviorism—that only literal, scientific language is capable of articulating thought and having objective significance. Instead, as long ago as 1942, Langer argued that artworks directly present emotional and cognitive meanings that are just as valuable—even objective—as the meanings of words and sentences (Langer, 1942).

Philosophical concepts are designed to open up whole new “planes of thought,” writes Deleuze (1968/1994). They resemble metaphors, he writes, because they connect an idea in one area with an idea in a completely different area of life. Because they are making connections, philosophical concepts are not useful on their own. A whole network of new connections is involved when we use a philosophical idea. Everyday concepts, on the other hand, are generalizations about one area of life. We use everyday concepts to reduce the complexity in that area of life. Social scientists also make generalizations. They report frequencies and means in their articles instead of discussing each person’s response to a survey question. Although everyday concepts are familiar, we sometimes forget their exact meaning and use them in multiple situations, some of which are inappropriate. In fact, we sometimes use everyday concepts so that we do not have to think, writes Deleuze (1968/1994).

Some of the philosophers mentioned in this book, including Marshall McLuhan (1911–1980) and Vilém Flusser (1920–1991), are known as speculative thinkers because of the originality of their ideas. Neither man systematically developed his ideas, but both men’s ideas were prophetic. Flusser, for example, used metaphors such as the “web” and the “net” to describe how technology will connect us long before the rise of the Internet. McLuhan talked about the “global village,” “media convergence,” “extensions of the central nervous system,” and interactivity back in the 1950s and 1960s.

For Massumi, philosophy is speculative thinking. Philosophy, he writes, “is an utterly speculative undertaking. It is an utterly contrived thought-fiction…. It is prefunctional” (2002: 241). In addition, he believes philosophy keeps wonder in the world. Most of the time we are concerned with utility and profitability, but not when reading and writing philosophy. Of course, these qualities—speculation, wonder, and thought-fiction—are what make philosophy suitable for creating revolutionary theories.

In summary, without philosophers we become accustomed to standard ways of thinking. In fact, Gilles Deleuze believes that we wallow in the inertia of common sense. On the other hand, by questioning old assumptions and ← 6 | 7 → developing new ones, by interfering and speculating, and by creating theoretical problems as well as the concepts to solve those problems, philosophers’ writings are likely to stimulate the type of thinking needed for the creation of new theories.

Realism and Relativism

Realists are philosophers associated with Positivism, Post-Positivism, Empiricism, Objectivism, Functionalism, and Behaviorism, which are collectively known as the Dominant Paradigm. These philosophers believe that a single, true reality exists. This single reality has real objects with real attributes. They believe that we are capable of directly, or objectively, perceiving the attributes of objects. Realists argue that we do not initially interpret these sensations. Only during the first stage of information processing by our brain does interpretation begin.

In order to discover the real attributes of objects, Realists argue that we must avoid subjective experiences of those objects. Take apples, for example. Realists who want to study apples without a subjective experience might create and use an instrument. If this instrument comes with specific instructions for operation, and if everyone follows the instructions, then the instrument is thought to provide objective information about the apple. If the instrument yields numerical data, then Realists can say apples have an attribute that is, for example, 4.55. Of course, this number is an abstraction of the direct experience of an apple, and this number provides less information than is possible with direct experience, but the number seems objective. Since a number represents the attribute of the apples, this attribute and its number can easily be compared with other numbers representing other attributes unrelated to apples. Moreover, any of the numbers can be used in various mathematical equations, so a Realist could say that as a particular attribute of apples increases, another attribute—xxxxx—decreases. As a result, Realists believe they can explain how reality works.

Relativists, by contrast, are associated with Phenomenology, Perspectivism, Contextualism, Interpretivism, Hermeneutics, and Social Constructionism, which are collectively known as the Alternative Paradigm. Relativists agree with Realists that reality is real. But Relativists deny that we directly, objectively, perceive reality. Every time we perceive, we do so from our particular vantage point, with our particular sensory capabilities. If someone ← 7 | 8 → perceived an object from a different vantage point, or if an organism with different sensory capabilities perceived an object, then their perceptions would differ from our perceptions. In other words, it is impossible to take the individual human out of perception.

Not only do we lack direct (objective) access to reality; we also lack direct (objective) understanding of reality, because our understanding depends on our access. What science does, argue Relativists, is provide rules so that we can agree upon a way to access and understand reality. Research, therefore—with its claims of objectivity—is based upon agreement. Researchers’ objectivity is not based upon the elimination of subjectivity. Researchers achieve reliability relatively easily because they can specify rules and train other researchers to follow those rules. Researchers, however, cannot so easily achieve validity. Achieving validity is particularly challenging if researchers begin with concepts rather than experience.

Relativists present a less reassuring depiction of reality than Realists. With Relativists, reality changes when groups of people change their minds about how to construct reality. In addition, multiple realities can exist simultaneously when different groups of people have different ways of constructing reality. For example, different religions, ethnicities, social classes, economic classes, political parties, and so on, have different perspectives on what we should do and how we should go about doing it.

Phenomenology

I found that some types of philosophy were more useful for creating multisensory media theories than others. Analytical philosophy, for example, was not useful. It focuses on mind and language, write cognitive linguist George Lakoff (U.S. cognitive linguist at the University of California, Berkeley) and philosopher Mark Johnson (U.S. philosopher at the University of Oregon). Lakoff and Johnson believe the assumptions of analytical philosophy are: (a) to analyze language is to analyze thought; (b) linguistic meaning is determined by what it can correspond to in the world; (c) a sentence is true if the words fit the state of affairs in the world; (d) all meaning is literal; and (e) all meaning is disembodied (Lakoff & Johnson, 1999: 443). In addition, analytic philosophers are not very concerned about emotions or feelings. They believe emotions and feelings are noncognitive, so they see little role for them in meaning. Analytical philosophy, therefore, has little to do with multisensory ← 8 | 9 → perception, embodied meanings, conceptual metaphors, or multisensory media theory.

Phenomenologists, on the other hand, describe their immediate, yet subjective, sensory experience as it occurs, writes Maurice Merleau-Ponty (1908–1961). In other words, phenomenologists try not to think about what others have previously said about something—an apple, for example, when they perceive that apple. Phenomenologists also try not to remember their previous experiences of apples. Students taking a painting class are similar to phenomenologists. Students look at a specific apple and paint the shape they see from their perspective (perhaps high and to the side). They see the colors of apple in that particular light and they use those colors when they paint the apple. But if there is no apple in the room, then students must remember apples in general. As a result, they paint the concept of an apple, not a specific apple. For example, they might paint something round and red with a short stem. If there is no apple in the room, then painters do not resemble phenomenologists.

Much of the time we follow unconscious routines. For example, when I first wake up in the morning, still groggy, I find the way to the bathroom out of habit and I wash my face. I don’t think about any of this; I just do it. When something goes wrong, or something changes, or I find myself in a new situation, then my experience is no longer immediate. Instead, my experience is the result of thinking and planning.