MULTIMODAL TEXT AND MEDIA RESOURCE: SEMIOTIC, COGNITIVE AND PEDAGOGICAL INTERPRETATIONS
Keywords:
Keywords: multimodal text, media resource, mode, social semiotics, dual coding, multimedia learning, multiliteracies, primary school.Abstract
Abstract. The article examines how the concepts of multimodal text and media resource are defined in three research traditions and what each definition implies for teaching reading in primary school. Social semiotics treats a multimodal text as a single message built from several modes, each with its own meaning potential. Cognitive psychology explains how verbal and pictorial information is processed and when their combination helps or hinders understanding. Pedagogy, drawing on the multiliteracies framework, regards the multimodal text as an object of teaching rather than an illustration of it. On the basis of this analysis, working definitions of both concepts are proposed for primary reading literacy lessons, together with a distinction between a text in which the image repeats the words and a text in which it adds meaning.
References
1. Cope, B., & Kalantzis, M. (2009). “Multiliteracies”: New literacies, new learning. Pedagogies: An International Journal, 4(3), 164–195. https://doi.org/10.1080/15544800903076044
2. Jewitt, C. (Ed.). (2014). The Routledge handbook of multimodal analysis (2nd ed.). Routledge.
3. Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Routledge.
4. Kress, G., & van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). Routledge.
5. Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University Press.
6. Nikolajeva, M., & Scott, C. (2001). How picturebooks work. Garland.
7. Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.
8. Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. Teachers College Press.
9. Takacs, Z. K., Swart, E. K., & Bus, A. G. (2015). Benefits and pitfalls of multimedia and interactive features in technology-enhanced storybooks: A meta-analysis. Review of Educational Research, 85(4), 698–739. https://doi.org/10.3102/0034654315568142
10. The New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60–92. https://doi.org/10.17763/haer.66.1.17370n67v22j160u
11. Unsworth, L. (2001). Teaching multiliteracies across the curriculum: Changing contexts of text and image in classroom practice. Open University Press.
