A Multimodal Social Semiotic Perspective
Søren Vigild Poulsen, University of Southern Denmark & Gunhild Kvåle, University of Agder
What we strive at in this article is providing a viable theoretical-analytical approach that serves to underscore our research interest: the relationship between multimodal meaning potentials of semiotic surfaces, various dimensions of the technology “below” these surfaces, and the social practices in which a given social media is embedded in and contributing to shaping. We have found that the following seven dimensions are the most crucial dimensions in our thinking, and we will, therefore, structure our paper around it: i) practice, ii) multimodality, iii) the social, iv) medium, v) the material, vi) the historical, and vii) the critical. Other dimensions may be added, and the order and significance of each of the seven dimensions will vary in an analysis of specific social media technologies. But as a whole, the dimensions provide an apt perspective for the multimodal social semiotic study of social media.
Adami, Elisabetta (2014) Retwitting, Reposting, Repinning; Reshaping Identities Online: Towards a Social Semiotic Multimodal Analysis of Digital Remediation. LEA : Lingue e Letterature d’Oriente e d’Occidente 3 (3): 223-243.
Djonov, E. & Van Leeuwen, T. (2011). The semiotics of texture: from tactile to visual. Visual Communication, 10 (4), p. 541–564.
Djonov, E. & Van Leeuwen, T. (2012). Normativity and software: A multimodal social semiotic approach. In S. Norris (ed.), Multimodality and Practice: Investigating Theory-in-Practice-through-Method (p. 119–137). New York: Routledge.
Djonov, E. & Van Leeuwen, T. (2013a). Between the grid and composition: Layout in PowerPoint’s design and use. Semiotica (197), p. 1–34.
Djonov E. & van Leeuwen, T. (2013b): Bullet points, New Writing and the Marketization of Public Discourse. A critical multimodal perspective. In E. Djonov & Z. Sumin (eds.) Critical Multimodal Studies of Popular Discourse (p. 232–250). London: Routledge.
Jones, R. H. (2009). Technology and sites of display. In Jewitt, Carey (ed.) The Routledge Handbook of multimodal analysis. (pp. 114-126). London: Routledge.
Kress, G. & van Leeuwen, T. (2001). Multimodal Discourse. The Modes and Media of Contemporary Communication. London: Arnold.
Kvåle, G. (2016). Software as Ideology. A Multimodal Critical Discourse Analysis of Microsoft Word and SmartArt. Journal of Language and Politics – Special Issue on Multimodality, Politics and Ideology 15 (3), p. 259–273.
O’Halloran, K. (2009). Historical changes in the semiotic landscape: from calculation to computation. In C. Jewitt (ed.) The Routledge Handbook of Multimodal Analysis (p. 98–113). London: Routledge.
van Leeuwen, T. (2005). Introducing Social Semiotics. London: Routledge.
van Leeuwen, T. & Djonov, E. (2013). Multimodality and software. In C. Chapelle (ed.), Encyclopedia of Applied Linguistics. (unpaginated). Oxford: Wiley-Blackwell.
van Leeuwen, T., Djonov, E., & O’Halloran, K. L. (2013). “David Byrne really does love PowerPoint”: art as research on semiotics and semiotic technology. Social Semiotics 23 (3), p. 409-423.
Zappavigna, M. (2013) “Enacting identity in microblogging through ambient affiliation”. Discourse & Communication.
Zappavigna, M. (2016), `Social media photography: Construing subjectivity in Instagram images’, Visual Communication 15(3), 271-292.
Zhao, S., Djonov, E., & van Leeuwen, T. (2014). Semiotic technology and practice: a multimodal social semiotic approach to PowerPoint. In Text & Talk 34 (3), p. 349-375.
Zhao, S & Zappavigna, M (in pre) Intersubjectivity and the Selife: a social semiotic analysis of social photography from Instagram to “hipster” magazines New Media and Society