Multimodal Literacy in Education

Sonja Starc, Nickolas Komninos

Introduction

We are delighted to introduce the comprehensive multilingual volume titled ‘Case Studies in Multimodal Texts in Schools’. This book explores the diverse applications of multimodal studies across various educational settings, drawing from compelling case studies spanning Singapore, Greece, Italy, the UK, Spain, and Slovenia. It does this also maintaining three chapters in Slovene to improve their impact in the Slovene educational context. The origin of these contributions stems from the December 2022 symposium titled ‘Multimodal Texts in School’, organised by the University of Udine, Italy and the University of Primorska, Slovenia where scholars engaged in rigorous discussions around analytical methodologies such as Systemic Functional Grammar and Corpus-Assisted Multimodal Discourse Analysis. The symposium addressed a spectrum of inquiries concerning the role of multimodal texts in educational contexts, reflecting the global diversity of educational practices and challenges. The resulting collection comprises ten insightful chapters, each delving into distinct themes using the aforementioned analytical frameworks.

This book embarks on a journey through the practical applications of multimodality in classrooms, offering invaluable insights garnered from real-world experiences. We have meticulously organised the chapters around three overarching themes: theory and policy, educational processes and classroom case studies, and comparative analysis alongside alternative approaches to multimodality.

The integration of multimodal studies into curricula has witnessed a significant surge in recent years, echoing the seminal work of the New London Group in the 1990s. As literacy paradigms evolve, encompassing a spectrum of modalities, educators grapple with the notion of ‘multiliteracy’ (Cope & Kalantzis, 2000, 2015; Kress, 2000, 2003) in the digital age. Hence, these contributions signify a pivotal advancement in the realm of multimodality, language, and linguistics, extending established research into practical applications within educational settings.

Each chapter encapsulates a distinctive facet of multimodal pedagogy, offering nuanced perspectives on its implementation and impact. From examining student agency in creating multimodal texts to sensitising learners to multimodality and genre, these case studies exemplify the dynamic interplay between theory and practice.

Fei Victor Lim’s chapter, titled ‘From Literacy to Multiliteracies: Engaging with Multimodal Texts in Singapore Schools’, delves into the expanding landscape of literacy curricula worldwide. Departing from traditional language-centric approaches, this chapter illuminates Singapore’s pioneering strides in integrating multimodal meaning-making into the English Language syllabus. Through a design-based research pro- ject, Lim offers insights into how educators and students in Singaporean classrooms navigate and harness the potentials of multimodal texts. By synthesising empirical findings with pedagogical reflections, Lim not only underscores the significance of multiliteracies but also presents actionable recommendations to enrich pedagogical practices and refine the assessment of multimodal literacy skills.

Elorza Izascun presents a compelling exploration in the chapter titled ‘A Six-Stage Sequential Frame of Migration Representations in Children’s Picture Books’. This chapter delves into the intricate portrayal of migration in children’s literature, conceptualising it as a dynamic narrative unfolding across six sequential stages or episodes. Drawing upon a meticulous content analysis of thirty migration-themed picture books, Izascun identifies and delineates these stages, arguing for their pivotal role in shaping the narrative potential of migration representations. By offering a structured framework, Izascun’s work not only enhances our understanding of how writers and illustrators navigate the complexities of migration narratives but also provides a valuable benchmark for evaluating the myriad portrayals of migration in children’s picture books, encompassing both forced and non-forced migration scenarios.

In her chapter titled ‘Student Agency in the Creation of Multimodal Texts at the Ancient Agora of Athens: A Political Project’, Sophia Diamantopoulou explores how primary students engage with history through multimodal texts. The project, set at the Ancient Agora Museum in Athens, aims to connect 10-year-old learners with their history curriculum. Diamantopoulou analyses the students’ creative outputs, combining imagery and text, to illustrate how they actively shape their learning experiences. Within this hybrid space, where school-based instruction intersects with museum resources, students exercise agency by selecting and transforming key resources into meaningful expressions. By adopting a multimodal social semiotic perspective, Diamantopoulou highlights the transformative potential of such initiatives, emphasising students’ active role in constructing knowledge and deepening their understanding of historical concepts.

In her chapter titled ‘Rethinking the Use of Predominantly Pictorial Semiotic Sources in Primary School Textbooks for Slovene Language’, Sonja Starc evaluates the effectiveness of visual aids in educational materials. She argues that while images are valuable for conveying abstract concepts, their overreliance without accompanying verbal explanations poses challenges. Starc’s study focuses on a textbook sample for 10–11-year-old Slovene students, revealing that while images are easily understood by most students (70 %), there is a lack of contextual understanding without verbal explanations. This results in superficial comprehension and potential misconceptions. Furthermore, students’ written expressions often lack formal structure and omit key thematic elements, leading to ambiguity and unclear communication. This linguistic deficiency, coupled with confusion between semiotic systems and media, highlights gaps in language education objectives. To address these issues, Starc suggests textbooks incorporate more verbal explanations to deepen understanding and language development. Balancing pictorial and verbal elements is essential for wide-ranging comprehension, linguistic proficiency, and critical thinking skills among students. This underscores the crucial role of textbooks in shaping linguistic competence and calls for a nuanced approach in their design to effectively meet language education objectives.

Styliani Karatza’s chapter, titled ‘Applying CFRIDiL Guidelines to Create a Digital Poster in a Primary School Context’, explores the practical application of the Common Framework of Reference for Intercultural Digital Literacies (CFRIDiL). This framework offers clear guidelines for proficiency and intercultural awareness in multimodal digital literacies. Karatza begins by outlining the rationale and methodology of CFRIDiL, emphasising its three dimensions: multimodal orchestration, digital technologies, and intercultural communication, along with a fourth dimension encompassing transversal skills. The chapter then delves into empirical research conducted with primary school students, focussing on the creation of a digital poster on sustainability using CFRIDiL descriptors. The findings of this case study are discussed, highlighting the differences between the initial and final versions of the poster. Additionally, Karatza provides CFRIDiL-based self-assessment grids, checklists for teachers, and task objectives to aid in the implementation of CFRIDiL guidelines in primary school contexts.

In his chapter titled ‘Sensitising Students to Multimodality and Genre: A Diamesic and Diachronic Approach’, Nickolas Komninos explores methods to enhance university students’ understanding of multimodality and genre. Students were tasked with compiling corpora of obituaries, divided both diachronically and diamesically, and analysed using Bhatia’s genre analysis approach. This involved examining language patterns with Coh-Metrix, text patterns with a Time-based moves annotation, and multimodality through transcription techniques. Komninos evaluates student progress through questionnaires and semi-structured interviews, revealing insights into three key areas: the socio-anthropological dimension of newsworthiness, genre analysis of obituaries, and tools for assessing student multimodal-genre awareness. This chapter sheds light on effective strategies for sensitising students to the complexities of multimodality and genre.

In their chapter titled ‘Audio Description as an Effective Learning Tool in the L1 and L2 Language Class: Theory and Applications in Primary School Settings’, Taylor and Perego explore the educational potential of Audio Description (AD). Originally designed for visually impaired individuals, AD involves transforming visual images into verbal descriptions. However, the authors argue that AD can benefit language learners of all ages and proficiency levels. The chapter focuses on the pedagogical possibilities of both screen and art AD, demonstrating how AD-based activities can be integrated into language classes. By offering examples from a small pilot focus-group activity, Taylor and Perego illustrate the versatility of AD in catering to diverse individual needs. They discuss how AD texts can be adapted to different levels of complexity, from ‘enriched’ to ‘easy-to-understand’ versions, to accommodate varying learning abilities. Overall, Taylor and Perego advocate for the incorporation of AD into language education, highlighting its potential to enhance comprehension, linguistic skills, and overall learning experience in both first-language (L1) and second-language (L2) contexts, particularly in primary school settings.

In their chapter titled ‘Adapted Picturebook Aiding Reading Development in Special Needs Children’, Baloh and Križnik underscore the significance of reading for children with special needs and explore strategies to make reading more accessible to them. The chapter emphasises the importance of reading for children with disabilities, highlighting the role of picture books in fostering holistic reading and comprehension skills. To address the specific needs of children with disabilities, Baloh and Križnik discuss the adaptation of the picture book Žiga spaget Goes to the Wide World by Aksinja Kermauner. The text is augmented with symbols of Augmentative and Alternative Communication in the Slovene language. The chapter aims to investigate how children with special needs perceive and interact with the adapted picture book, as well as the challenges they encounter during the reading process. Through research findings presented in the chapter, Baloh and Križnik provid insights into the acceptance of the adapted picture book by children with special needs and identify potential difficulties they face. This chapter contributes to the understanding of reading development in special needs children and offers valuable insights for educators and practitioners working in this field.

In her chapter titled ‘One Text, Two Picturebooks’, Cvetka Sokolov delves into the intricate relationship between text and illustration in children’s literature. She highlights the pivotal role of the illustrator as the first reader of the text, emphasising their responsibility to translate verbal language into visual representations while infusing their personal experiences and values into the interpretation. Focussing on the fairy tale ‘Three Spells for the Night’, Sokolov examines the distinct approaches taken by two students, Eliza Atanasovska and Klara Jan, from the Academy of Fine Arts and Design at the University of Ljubljana. The chapter explores the extent to which the illustrations in their respective picture books either reproduce the text literally or enhance and expand upon it. These contrasting visual interpretations offer valuable insights into the creative process and the varied perspectives brought by individual illustrators. Additionally, Sokolov discusses the didactic potential of reading the two picture books at different educational levels, underscoring the importance of recognising and appreciating the differences between visual interpretations of the same text.

In her chapter titled ‘The Close-Up of the Face and Its Literary Parallels’, Barbara Zorman conducts a comparative analysis of subjectivity representation in the youth film and the novel Ko zorijo jagode (Strawberry Time) by R. Ranfl (1978) and B. Jurca (1974) respectively and its impact on youth perceptions of gender roles. Drawing upon the works of Balázs, Deleuze, Metz, Elsaesser, and Doane, Zorman examines how the film uses close-ups of the protagonist’s face and body positioning within the narrative space to convey subjectivity. The chapter explores the emergence of a new type of protagonist in ‘Strawberry Time’, one that actively resists being portrayed as a passive figure or a ‘semantically empty’ surface. Zorman analyses key moments in the film where the protagonist ‘returns the gaze’ to the viewers, inviting them to participate actively in the creation of narrative meaning. By comparing the filmic and literary representations of subjectivity, Zorman offers insights into the interplay between visual and textual narratives, shedding light on how different mediums engage audiences and shape narrative experiences.

In this collection of diverse chapters, we have delved into the multifaceted realm of multimodal texts in educational settings, spanning various contexts and methodologies. Each chapter offers unique insights into the application and implications of multimodality in different educational landscapes, shedding light on its role in shaping learning experiences and outcomes. Through empirical studies, theoretical explorations, and practical applications, contributors have illuminated the intricate interplay between various semiotic resources, including images, text, sound, and gesture. They have demonstrated how these resources can be harnessed to foster deeper understanding, critical thinking, and creativity among students.