Joint modelling of speech and gaze dynamics for predicting turn-taking outcomes in multi-party conversation

Smooth speaker transitions are fundamental to effective conversation and rely on an inter-
locutor’s ability to predict when to enter the conversation. This ability depends on accurately
interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to
take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-
party settings, with multiple interlocutors available. This study models how gaze and speech
jointly signal conversational floor changes in free four-person dialogue. Using the GaMMA
corpus, we trained logistic regression models to classify floor-transfer outcomes based on multi-
modal features extracted before each turn-taking event. Predictors included gaze features such
as behavioural motifs and contrasts, entropy, gaze-based addressee identity, mutual gaze, and
supporting speech features derived from speaker loudness. Results show that gaze features cap-
ture predictive structure, and that combining them with loudness improves performance (ROC
AUC = 0.76 ± 0.04). Loudness reflected speaker control, while gaze dispersion and addressing
indexed listener readiness and competitive entry. Performance remained robust across noise
conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking
dynamics. These findings demonstrate that multimodal modelling can reveal how visual and
acoustic behaviors jointly forecast conversational control in natural interaction.