Building Bridges for Multimodal Research

| 9 →

Preface

The idea for this volume emerged from my work at the University of Bremen, Germany, being employed both in the English and General as well as German Linguistic Department in the Faculty of Linguistics and Literary Science. Teaching and researching multimodality there has always been and still is a challenge: there are considerable differences between national and international approaches to this emerging field, sometimes difficulties in transcending disciplinary borders, and a huge diversity in and of the study programmes in the various departments.

I therefore thank the University of Bremen for acknowledging my ideas of first bringing together the different perspectives in a conference. Both the conference and this book have been supported by the Institutional Strategy of the University of Bremen, funded by the German Excellence Initiative. The First Bremen Conference on Multimodality took place in September 2014 and multimodalists from all over the world and various disciplines presented their work. A selection of these presentations is now available in this volume and I thank all contributors for their cooperation and dedication to this book throughout the whole process.

Special thanks go to the editors of the series Sprache – Medien – Innovationen for their openness to and interest in the publication of this collection within the series’ context. This decision in itself already helps to bridge the gap between the difficulties and differences of interdisciplinary work in the humanities in Germany and elsewhere that are addressed in this book.

I am also grateful to John Bateman for his continuous guidance and encouragement throughout the last seven years in Bremen and his never-tiring enthusiasm for our research, and to Ognyan Seizov as a direct colleague in our department for his assistance and careful reading of some of the texts. I deeply appreciate their expert advice and constant support in being a multimodalist.

Bremen, September 2015

Janina Wildfeuer

| 11 →

I Theoretical Reflections on the Notion and Work of Multimodality

| 13 →

Janina Wildfeuer

Bridging the Gap between Here and There: Combining Multimodal Analysis from International Perspectives¹

1 Introduction

Multimodal analysis and, more generally, its closely related notion of multimodality represent an extremely fashionable and highly discussed research area in the humanities (and beyond) that has experienced rapid evolution and advancements over the last 15 to 20 years. Since the groundbreaking work of Gunther Kress and Theo van Leeuwen from the 1990s and early 2000s (Kress/van Leeuwen 1996, 2001, 2006; see also Kress 2010), many theoretical, methodological, and analytical developments across a multitude of disciplines have been put forth by researchers from all over the world. Several collections (cf., among others, Ventola/Charles/Kaltenbacher 2004; Eckkrammer/Held 2006; Royce/Bowcher 2007; O’Halloran/Smith 2011; Maiorani/Christie 2014, Norris/Maier 2014; Norris forthcoming), introductions (cf., e.g., Machin 2007; Serafini 2014), and handbooks for multimodal analysis (cf. Jewitt 2009, 2014), multimodality in human interaction (cf. Müller et al. 2013, 2014), language in a multimodal environment (cf. Klug/Stöckl forthcoming) as well as several further monographs mostly focusing on more specific aspects of multimodal analysis (see for example the book series Routledge Studies of Multimodality; see also Stöckl 2004; Fricke 2012; Bateman/Schmidt 2012; Sindoni 2013; Bateman 2014 – to name just a few) represent the multitude of interests and the range of topics subsumed under this keyword.

Understood as an approach to the simultaneous analysis of all semiotic resources in an artefact, multimodality is seen in these works as one of the most influential concepts for the semiotisation of diverse forms of communications, providing a range of frameworks for the detailed analysis of meaning construction within and across several modes² (cf. Bucher 2007: 49). Nevertheless, the ← 13 | 14 → starting points of these frameworks and their theoretical grounding could not be more diverse and, in part, ambiguous. Ideas of a theory of multimodality – if at all available – are heterogeneously widespread; multiple views of the definition of modality and even mediality coexist; and theoretical determinations and methodological examinations develop individually and not necessarily with mutual consideration or respect. This leads to a wealth of partly conflicting ideas that often differ from each other already in terms of their general basis, namely the in- or exclusion of, and focus on, language as a (or, for some, the central) semiotic resource (see Section 2 for further details).

As a researcher at a German university, working in linguistics in the context of several language departments (with an institutional distinction between German, English and also General Linguistics), these differences have played a significant role in the author’s personal work in the context of multimodality and have led to several discussions about, for example, hard-won legitimisations for research projects or the still existing need for permanently establishing this topic within the realm of German Linguistics. Positioned somewhere in the middle of more linguistically and also often intradisciplinarily focused research on the one hand and inter- and transdisciplinary approaches to the analysis of all kinds of communicative artefacts on the other, toggling between the two views of multimodality and working towards its acceptance and appreciation has long been a formidable, at times insurmountable task. The strong motivation to overcome this situation lies at the heart of the idea of building bridges across the gap between the national perspective in Germany and a more general and internationally well-established basis in multimodal analysis.

It was these concerns that gave the starting point for the First Bremen Conference on Multimodality that took place in September 2014 at the University of Bremen, Germany. The title and motto of the conference, Bridging the Gap between Here and There: Combining Multimodal Analysis from International Perspectives, indicated this idea from the outset and thus opened the field for multimodalists from various disciplines all over the world. The differences in national and international perspectives were a motivation for discussion and analysis of several multimodal theories and artefacts. By making explicit which aspects of description, terminology, and methodology vary and which difficulties result from these variations in daily academic life, the conference activated the discussion about the status quo and opened the view for new approaches to the analysis of multimodal ← 14 | 15 → documents. The diversity of the more than 20 presentations and the contributing researchers’ various backgrounds (more than 15 countries of the world were represented) demonstrated that the idea of bringing together work on multimodality under this motto is a worthwhile and successful undertaking.3

The aim of the present collection is to collect the various approaches and analyses presented at the conference in order to make their ideas and innovations available to a broader audience not only situated within the narrow context of multimodality, but also to researchers generally interested in disciplines that go beyond the traditional study of language, such as media and communication studies, psychology, computer and information science, and others. Bringing together the different angles on approaching multimodality is intended to provide some sorely needed awareness of the vulnerability of this kind of work and its debates and conflicts. Each of the approaches in focus will present an opportunity to overcome this situation and to bridge the gap between two or more research foci in order to demonstrate that work on multimodality not only engages in conflicts, but also opens up considerable potential for cooperation and collaboration.

Why this is still a necessary undertaking will be discussed in the following section, focusing again and in further detail on the conflicts currently prominent in multimodality research. The overview of the papers that is given in Section 3 along with a short classification and evaluation in terms of their placement within the broad context of multimodality will then provide a first step towards a systematic approach of building bridges to circumvent these conflicts and differences.

2 Multimodality diverse

As van Leeuwen outlines, the term multimodality already dates back at least to the 1920s, as the then-emerging field of the psychology of perception started thinking about the effects that various sensory perceptions created on and with each other (cf. van Leeuwen 2011: 549). Its more recent use in communication studies and linguistics represents a broadened application of the original notion and shares with it the idea of multiple resources or senses working together. Although the ← 15 | 16 → disciplines generally did not always agree in their use of the term, the idea behind it has long been of validity for and within the humanities (and also beyond), and van Leeuwen relates this back to general developments in public communication from the 1920s onwards (e.g., the rise of film and television and the shift in writing to a more illustrative style). Besides the four schools of linguistics he mentions to describe the beginning engagement with communicative modes other than language (cf. van Leeuwen 2011: 550), the philosopher and semiotician Charles Sanders Peirce had already called attention to the fact that the view of language as a sign system is not a satisfying and sufficient approach to semiotic entities in general (cf. Peirce 1909: 8–10). Peirce argues that it is also necessary to differentiate fundamentally individual sensory skills and impressions and to take into consideration the various dimensions of (verbal) signs in connection with other modalities. It is these observations that are common to almost all approaches to the description of communicative artefacts, even though their theoretical determinations and methodological elaborations develop differently.

In the following, we will outline particularly the difficulties that these observations bring with themselves in certain contexts and with regard to newly emerging fields of research within already established parameters, especially within the partly narrow contexts of linguistics on the one hand and with regard to more trans- and interdisciplinary steps transcending the borders of certain research sectors on the other. Particularly in contrast to van Leeuwen’s discussion that linguists took notice of the significant changes in communication and changed their general focus of interest, it is by no means self-evident for all linguistic areas to associate with these changes and to open the field for other modes. This counts in particular for the discipline of German Linguistics, which has become interested in multimodality only recently (see Section 2.1). Conversely, the need for trans- and interdisciplinary bridges to overcome conflicts and internal trench battles such as those within the disciplines of linguistics, for example, raises specific challenges within certain rigid structures. In particular with regard to the development of the so-called multimodal turn not only in the humanities (cf. Jewitt 2014: 4), there is no single discipline or field of study that can be considered to have a leading position in this research area – a bewildering and simultaneously intriguing fact that also contributes to multimodal study’s continuously evolving diversity.

2.1 National and international differences

Whereas the study of multimodality is still seen as “very new and […] a very creative endeavour” (van Leeuwen 2014: 22), the topic has yet been established within the disciplines of linguistics as well as communication and media studies ← 16 | 17 → in the broad international context. Conferences with renowned speakers from all over the world (e.g., the International Conference on Multimodality (ICOM) in London 2012, Hong Kong 2014, South Africa 2016) as well as specific topical panels at international conferences take place regularly. There are specific BA and MA programmes offered at several universities, partly accompanied by particular research centres or specialisation programmes (e.g., the MODE Research Centre at the Institute of Education, University College London; the Multimodal Analysis Lab at the University of Singapore; the Multimodal Research Centre at Auckland University of Technology, etc.).

The situation in Germany, on the other hand, is different. Only recently, and mainly in the last five years, the topic receives wider recognition, and workshops and smaller conferences call for further attention and discussion. So far, there are almost no specific BA or MA programmes, and only a small number of professorial chairs are directly attached to multimodal research. This is often due to the generally strict differentiation between language and literary studies on the one hand and media and communication studies on the other, which is still the norm at most universities in Germany and which seems to restrict communication analysis to the examination of language and, only secondly, its accompanying features and resources. Although study programmes on film, for example, are widely established, the interest in its specific features of multimodality is still in its infancy.

Nevertheless, accounts concerning the notion of multimodality in various artefacts have arisen in the last years from several perspectives and disciplines. Accordingly, however, they tend to deal with the general idea of broadening verbal analysis by including other semiotic entities4 – a perspective which is not in all aspects similar to the starting point of the founding fathers of the multimodality paradigm who underline that language should no longer be at the centre of all interest in communicative analyses (cf. Kress/van Leeuwen 1998, 2001). Although the notion of a general multimodality of language, for example, is now dominant in most of these discussions, the role of language in German descriptions and examinations is still of particular importance and it is, for instance, not normally the case that non- or almost-non-verbal artefacts are in the limelight. ← 17 | 18 →

For instance, Fricke (2012, 2013), as a representative of an often-quoted approach to multimodality in Germany, takes account of this general multimodality of language by describing the different manifestations in written and spoken language. In her opinion, linguistic multimodality – which, as it seems, has to be differentiated from other types of multimodality – is based on the “structural and/or functional integration into a matrix code (code integration) or the manifestation of one and the same code in different media (code manifestation)” (Fricke 2013: 736). In contrast to many other accounts in German linguistics, her concept does not restrict human communication to the monomodal resources of either spoken or written language but also includes the interpretation of – in particular – gesture and facial expression in a comprehensive linguistic analysis. In this analysis, however, language is always the underlying main semiotic entity. It is, thus, a very linguacentric determination, however suitable for the analysis of gestures as a significant part of verbal communication. Indeed, it is not Fricke’s intention to apply her perspective to other multimodal practices, and her account cannot be used as a generally valid approach to multimodality.5 Similar work has been done by Cornelia Müller and her colleagues (Müller 2009; Müller/Cienki 2009; Müller et al. 2013, 2014) in the research group “Dynamic Multimodal Communication”, which – according to the project’s website – sees multimodal communication as “first and foremost language in use in face-to-face communication” (International Research Network on Dynamic Multimodal Communication, DMC).⁶

Other media artefacts such as film, comics or websites, for example, can therefore not be described with the help of Fricke’s characteristics of multimodal language or the definition given by the DMC, since they neither feature a superordinate modality, which occurs in several different realisations, nor represent the narrow context of face-to-face communication. Instead, they connect more than one code and more than one sensory modality and they are – in contrast to language – not an instance of multimedia, but mostly of multimodality (for film and comics for example, see Bateman/Schmidt 2011; Wildfeuer 2014; Bateman/Wildfeuer 2014). ← 18 | 19 → The specific multimodal character of these artefacts is constituted by the integration of various semiotic resources, which together construct meaning and therefore also have to be analysed together, i.e. in their intersemiotic interplay. More and more accounts in the German landscape seem to finally incorporate this basic assumption in their work, going beyond the borders of traditional linguistics and mainly applying broader discourse analytical frameworks (cf., e.g., Stöckl 2006; Müller 2012; Klug forthcoming). Such analyses of film and comics as well as picture books, photographs, or online discourses as multimodal texts then also follow the assumption that language alone no longer determines the meaning of a communicative artefact but that it is always part of a bigger, multimodal ensemble.

This assumption has always been the basic prerequisite of the internationally dominant conceptualisation of multimodality which has mainly been established by the works of Gunther Kress and Theo van Leeuwen: It is “the use of several semiotic modes in the design of a semiotic product or event” in which “language is no longer the central semiotic mode” (Kress/van Leeuwen 2001: 2, 20; see also Kress/van Leeuwen 1998: 73). Other modes can similarly take up the position and functions of language without being language themselves. This concept of multimodality generally sees all forms of texts – and not language in principle – as artefacts consisting of various semiotic resources (see also Mavers in this volume for further aspects of this definition). It is thus a more global perspective than the one taken up by most of the German accounts, and at the same time it is not narrowed down to the discipline of linguistics or communication studies. Accounts in the Anglo-Saxon tradition often come from broader contexts and include other disciplines and more interdisciplinarily focused research in general. They analyse concrete text forms of several types that use images, gesture, facial expression, music, camera work, etc. and represent a potentially endless number of artefacts that unite these different modalities. In these approaches, the term multimodality is used categorically and in order to be promoted as a constitutive feature of (also non-verbal) texts. Hans-Jürgen Bucher, a further representative of multimodal research in Germany, sees this as a change in the point of view of all forms of communication – and demonstrates at the same time that this change has not yet fully been processed in the context of German research. More as an invitation and call for further work, he explains that each analysis of a communicative artefact should be targeted on multimodality and should show how meaning is created out of the different modes (cf. Bucher 2011: 11) – an aim that has already been described by Kress/van Leeuwen in the 1990s.

For the analysis of complex multimodal artefacts, this orientation is definitely necessary to describe the functional interplay of the individual modalities and ← 19 | 20 → to highlight respective textual criteria with regard to their coherent meaning-making strategies – a task that has, for example, long, but not successfully been undertaken with the focus on filmic language in traditional film semiotics (cf., e.g., Metz 1974). More recent discussions in this and other contexts no longer deal with the comparison of individual modalities with regard to language or their description due to the sensory channel through which they are perceived, as it has for example been done by Hartmut Stöckl. He – as a further researcher in multimodality in the German speaking countries, being based in Salzburg, Austria7 – provides a detailed list of resources available in audio-visual texts (cf. Stöckl 2004: 13). Instead, questions concerning relationships between modes and their interplay or the narrative structure of the text are in the foreground, often analysed either with regard to processes of reception and interpretation, as examined in further detail by Bucher (2007, 2011), for example, or with regard to an application of already established frameworks of discourse analysis or stylistics, as pursued by Fraas et al. (2013) or Meier (2014) – to name only a few further works in the German context.

In general, multimodality in Germany is definitely still an emerging, new field of research which still has to fight with disciplinary borders and internal power struggles about the legitimacy of media analyses that go beyond the main topics of linguistic examination and open up the field for new and different research questions. Cautiously speaking, this seems – in comparison to the profound developments in the international context – to have restrained the German scientific landscape in both progress in and demand for further research on multimodality. However, the growing interest in conferences on this topic as well as the many publications that arise from several perspectives show that German linguistics finally seems to be overcoming its traditional restrictions and limitations, and it will open up to more interdisciplinary work – a step that is, as the following section will show, crucial for a successful anchoring of this topic in Germany’s academic landscape.

2.2 Multi-, inter- and transdisciplinary multimodality

Determining multimodality as a basic feature of communicative artefacts is perhaps the simplest and at the same time most convincing use of this term in the ← 20 | 21 → broadest possible context. It, however, does not say much about the emerging field of research behind this context, its theories, and methodologies as well as the many analytical applications, for which the term is used. In the first edition of the Routledge Handbook of Multimodality from 2009, Carey Jewitt lists different references to multimodality as “a theory, a perspective, a methodological application or a field of enquiry” (Jewitt 2009a: 127), and she concedes that the term indeed does not describe a coherent and unambiguously defined concept. Rather, and not euphemistically at all, this list reflects the abovementioned problematic variety of concepts behind this notion, when it comes to the definition of subject for an – albeit still emerging – discipline.

Although the Routledge Handbook gives a very good overview of and feeling for the diversity and interconnectedness of the different accounts brought together, there is, however, simply no overarching or summarising definition of multimodality, which would help classify or organise the diverse uses in its outline. Instead, this use alternates even within several individual accounts and it is, thus, not without reason that Charles Forceville, in his review of the handbook, explicitly criticises this point of uncertainty and openness and calls for „further subdivision of work [and] for systematic rigour“ in the context of multimodal research (cf. Forceville 2010: 2607).

Six years later, the situation has hardly changed, and Jewitt also elaborates on this again in the second edition of the handbook – this time, however, cautiously hedging: “It could be argued that, strictly speaking, multimodality refers to a field of application rather than a theory, although the on-going development of theories that account for the multimodal is an imperative to support high-quality research“ (Jewitt 2014: 2). Both quotations and their diachronic development demonstrate that the continuing debate about multimodality has not yet come to an end, but rather characterises the multimodal landscape as an extensive field of quite diverse research. This diversity is, indeed, one of the constituting features not only of the field and its contexts of investigation, but also of all the work combining under this keyword, which makes it a particularly challenging and exciting endeavour. It will therefore not be the aim of this introduction to add another attempt of definition and discussion of the notion of multimodality – exactly because of this original particularity. Instead, and with regard to the various contributions following this introduction, we take over Jewitt’s description of the “field of application” and leave the term multimodality itself as open as possible for further contemplation, re-evaluation, and specification of what its application can look like.

For this, the multimodalists brought together in this book mainly orientate towards the following principal questions, which were also central in the conference’s call for papers: ← 21 | 22 →

– How is multimodality defined throughout various concepts in linguistics, literary science, media studies or the authors’ home discipline? Which differences exist between these concepts?

– Which terminological specifications or analytical improvements are needed to bring the concepts together?

– Which (analytical) methods can help to bridge the gap between the varying concepts and how?

It will become evident in the following chapters that, in response to these questions, the authors either follow various already well-established notions of multimodality – as for example the communication theoretical concept initiated by Kress/van Leeuwen (2001), the linguacentric one provided by Fricke (see above), and the multicode concept as described by Mitchell/Thomas (1995) and Posner (1986) – or they provide their own ideas and definitions. Frequently used terms in the context, such as multi- and intermodality, intersemiosis or intersemiotic interplay, as well as crossmodal meaning-making, will also play a role in the various discussions, either addressed from a theoretical or a methodological perspective, or via a concrete empirical or analytical approach.

These perspectives as responses to the abovementioned questions demonstrate a further central feature of multimodality’s overarching context: its multi- and interdisciplinarity. As a matter of fact, the various approaches’ origins are as diverse as the use of the notion of multimodality itself – and these two facts of course mutually influence each other. The book’s variety as further represented by the list of contributors is thereby only one example of the multitude of disciplines that contribute to the discussion and further development of what has been called a “serious academic perspective” (see Mavers in this volume). Although a certain number of the contributors work in the broad field of linguistics and communication studies – which still seem to be the main areas of interest for all questions concerning multimodality – there are also representatives from further disciplines in the humanities, such as argumentation theory, aesthetics, medieval studies and musicology, political science, and the broad context of media studies, as well as from more technically oriented disciplines such as computer and information science, (neuro-)psychology, and engineering.

Their different starting points as well as the common ideas leading to similar research questions corroborate the statement this book also makes: multimodality is interdisciplinary in itself and there is absolutely no need for any discipline to claim the leading role for itself. Instead, it is the aim of this book to demonstrate that the different disciplines do not compete against each other but rather gain from their reciprocal influence and mutual exchange. Thus, instead of jostling for ← 22 | 23 → position in the sometimes vast landscape, the various disciplines will share their ideas and questions, and each of them will bring in certain theoretical or terminological specifications or methodological and analytical peculiarities, which will then help bridge the described gaps between the various applications.

Therefore, in contrast to other handbooks and introductions (see Section 1) that try to define the common idea of multimodal research and to describe coherent methodologies, this book elaborates on ways of bridging and mediating between the various disciplines, definitions, and methods. Each chapter demonstrates the specific value of an interdisciplinary approach by showing concrete example analyses based on certain terminological or theoretical assumptions. The authors partly re-examine existing theoretical frameworks and/or highlight and apply new methodologies that have not yet been used within the multimodal context.