header image

Alignment sequences in multimodal interaction

Bert Oben & Geert Brône (Leuven)

Saturday, May 19th
Buchanan A201

This paper ties in with two recent developments in cognitive-functional approaches to language. First, there is a growing interest in explicitly incorporating dialogicity in cognitive discourse models (Langacker 2001, Verhagen 2005, Fried & Östman 2005, Du Bois 2011). Second, there is growing awareness that usage-based models in linguistics need to adopt a multimodal perspective (Mittelberg 2007, Sweetser 2007, Cienki & Müller 2008). Both strands serve as an empirical operationalization of the non-reductionist foundation of cognitive-functional linguistics, which holds that all aspects of the linguistic usage event that contribute to the meaning potential (including intonation, gesture, posture, etc.) should in principle be incorporated.

In this paper we focus on the process of synchronization where interlocutors engaged in face-to-face interaction continuously and increasingly use and reuse each others’ linguistic and non-linguistic behaviour. This process has been approached from different theoretical perspectives (e.g. mechanistic processes vs. joint actions) and has alternatively been labelled interactive alignment (Pickering & Garrod 2004), conceptual pact (Brennan & Clark 1996), resonance (Du Bois 2011), and mimicry (Kimbara 2006). In the majority of studies, however, the perspective on this phenomenon has been largely monodimensional (focus on alignment behaviour in one semiotic channel or on one linguistic level) and restricted to minimal contexts (pairs of utterances rather than longer stretches of resonance). In order to arrive at a more fine-grained multimodal picture, we conducted a corpus study using the MIMIC-corpus (MultIModal Interaction Corpus, Brône & Oben 2011). This allowed us to incorporate various meaning-making cues, ranging from intonation in the acoustic channel, to gesture from the visual channel, and gaze generated by eye-trackers.

The focus of this paper is on how multimodal alignment sequences emerge, persist and ‘die out’ in the interaction. First, we deal with the differences in distribution between the linguistic and non-linguistic channels in forming alignment sequences. We address questions such as “is non-linguistic alignment more restricted to simple pairs than linguistic alignment?” and “can linguistic alignment more easily cross the boundaries of one conversational sequence than non-linguistic alignment?”. Second, we look at distributional patterns within each of the semiotic channels, and the role of different levels of representation. For instance, for the linguistic channel we address whether different levels of linguistic alignment (lexical, morphological, syntactic) behave differently, both in terms of frequency and persistence in and across conversational sequences. The results of the corpus-based study reveal both high degrees of interaction between channels/levels (what we label ‘clustered alignment’) and significant differences in durability or persistence.

(See the attached PDF for references)

Download PDF

a place of mind, The University of British Columbia

Faculty of Arts - Department of Englsh
397 - 1873 East Mall,
Vancouver, BC, V6T 1Z1, Canada

Emergency Procedures | Accessibility | Contact UBC  | © Copyright The University of British Columbia