Factors affecting inter-rater agreement in human classification of eye movements: a comparison of three datasets

Lee Friedman*, Vladyslav Prokopenko, Shagen Djanian, Dmytro Katrychuk, Oleg V Komogortsev

*Kontaktforfatter

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

1 Citationer (Scopus)

Abstract

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters.

OriginalsprogEngelsk
TidsskriftBehavior Research Methods
Vol/bind55
Udgave nummer1
Sider (fra-til)417-427
Antal sider11
ISSN1554-351X
DOI
StatusUdgivet - jan. 2023

Fingeraftryk

Dyk ned i forskningsemnerne om 'Factors affecting inter-rater agreement in human classification of eye movements: a comparison of three datasets'. Sammen danner de et unikt fingeraftryk.

Citationsformater