The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters

Angela Tinwell, Mark Grimshaw, Deborah Abdel Nabi

Research output: Contribution to journal › Journal article › Research › peer-review

628 Downloads (Pure)

Abstract

This study investigates if the Uncanny Valley phenomenon is increased for realistic, human-like characters with an asynchrony of lip movement during speech. An experiment was conducted in which 113 participants rated, a human and a realistic, talking-head, human-like, virtual character over a range of onset asynchronies for both perceived familiarity and human-likeness. The results show that virtual characters were regarded as more uncanny (less familiar and human-like) than humans and that increasing levels of asynchrony increased perception of uncanniness. Interestingly, participants were more sensitive to the uncanny in characters when the audio stream preceded the visual stream than with asynchronous footage where the video stream preceded the audio stream. This paper considers possible psychological explanations as to why the magnitude and direction of an asynchrony of speech dictates magnitude of perceived uncanniness and the implications of this in character design.
Original languageEnglish
JournalInternational Journal of Mechanisms and Robotic Systems
Volume2
Issue number2
Pages (from-to)97-110
Number of pages24
ISSN2047-7244
DOIs
Publication statusPublished - 2015

Fingerprint

Experiments

Keywords

  • onset asynchrony; audio visual speech; Uncanny Valley; virtual characters; realism; human-like; lip-sync; video games

Cite this

Tinwell, Angela ; Grimshaw, Mark ; Abdel Nabi, Deborah. / The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters. In: International Journal of Mechanisms and Robotic Systems. 2015 ; Vol. 2, No. 2. pp. 97-110.
@article{8e146ae6a8d14f04a0558ac812447ed9,
title = "The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters",
abstract = "This study investigates if the Uncanny Valley phenomenon is increased for realistic, human-like characters with an asynchrony of lip movement during speech. An experiment was conducted in which 113 participants rated, a human and a realistic, talking-head, human-like, virtual character over a range of onset asynchronies for both perceived familiarity and human-likeness. The results show that virtual characters were regarded as more uncanny (less familiar and human-like) than humans and that increasing levels of asynchrony increased perception of uncanniness. Interestingly, participants were more sensitive to the uncanny in characters when the audio stream preceded the visual stream than with asynchronous footage where the video stream preceded the audio stream. This paper considers possible psychological explanations as to why the magnitude and direction of an asynchrony of speech dictates magnitude of perceived uncanniness and the implications of this in character design.",
keywords = "onset asynchrony; audio visual speech; Uncanny Valley; virtual characters; realism; human-like; lip-sync; video games",
author = "Angela Tinwell and Mark Grimshaw and {Abdel Nabi}, Deborah",
year = "2015",
doi = "10.1504/IJMRS.2015.068991",
language = "English",
volume = "2",
pages = "97--110",
journal = "International Journal of Mechanisms and Robotic Systems",
issn = "2047-7244",
publisher = "Inderscience Publishers",
number = "2",

}

Tinwell, A, Grimshaw, M & Abdel Nabi, D 2015, 'The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters', International Journal of Mechanisms and Robotic Systems, vol. 2, no. 2, pp. 97-110. https://doi.org/10.1504/IJMRS.2015.068991

The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters. / Tinwell, Angela; Grimshaw, Mark; Abdel Nabi, Deborah.

In: International Journal of Mechanisms and Robotic Systems, Vol. 2, No. 2, 2015, p. 97-110.

Research output: Contribution to journal › Journal article › Research › peer-review

TY - JOUR

T1 - The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters

AU - Tinwell, Angela

AU - Grimshaw, Mark

AU - Abdel Nabi, Deborah

PY - 2015

Y1 - 2015

N2 - This study investigates if the Uncanny Valley phenomenon is increased for realistic, human-like characters with an asynchrony of lip movement during speech. An experiment was conducted in which 113 participants rated, a human and a realistic, talking-head, human-like, virtual character over a range of onset asynchronies for both perceived familiarity and human-likeness. The results show that virtual characters were regarded as more uncanny (less familiar and human-like) than humans and that increasing levels of asynchrony increased perception of uncanniness. Interestingly, participants were more sensitive to the uncanny in characters when the audio stream preceded the visual stream than with asynchronous footage where the video stream preceded the audio stream. This paper considers possible psychological explanations as to why the magnitude and direction of an asynchrony of speech dictates magnitude of perceived uncanniness and the implications of this in character design.

AB - This study investigates if the Uncanny Valley phenomenon is increased for realistic, human-like characters with an asynchrony of lip movement during speech. An experiment was conducted in which 113 participants rated, a human and a realistic, talking-head, human-like, virtual character over a range of onset asynchronies for both perceived familiarity and human-likeness. The results show that virtual characters were regarded as more uncanny (less familiar and human-like) than humans and that increasing levels of asynchrony increased perception of uncanniness. Interestingly, participants were more sensitive to the uncanny in characters when the audio stream preceded the visual stream than with asynchronous footage where the video stream preceded the audio stream. This paper considers possible psychological explanations as to why the magnitude and direction of an asynchrony of speech dictates magnitude of perceived uncanniness and the implications of this in character design.

KW - onset asynchrony; audio visual speech; Uncanny Valley; virtual characters; realism; human-like; lip-sync; video games

U2 - 10.1504/IJMRS.2015.068991

DO - 10.1504/IJMRS.2015.068991

M3 - Journal article

VL - 2

SP - 97

EP - 110

JO - International Journal of Mechanisms and Robotic Systems

JF - International Journal of Mechanisms and Robotic Systems

SN - 2047-7244

IS - 2

ER -

Tinwell A, Grimshaw M, Abdel Nabi D. The Effect of Onset Asynchrony in Audio Visual Speech and the Uncanny Valley in Virtual Characters. International Journal of Mechanisms and Robotic Systems. 2015;2(2):97-110. https://doi.org/10.1504/IJMRS.2015.068991

Access to Document