The Trained Panel Method and Its Application in HCI Research

User interfaces utilising multiple modalities or even multisensory feedback are more common, creating the need for evaluation techniques that can consider various quality dimensions. This paper demonstrates how the trained panel method can support the design and evaluation of physical or complex technological artefacts by mapping out design spaces based on their descriptive attributes. It is an expert-based method, and the goal is to derive a comprehensive description of a sample of existing artefacts or prototypes. The method entails training as well as multiple feedback sessions to ensure consensus among panel participants. We describe the advantages and limitations of the method by presenting how it was applied to identify important attributes in the design or evaluation of smartwatches. Apart from the specific case described in detail, we also discuss how and in what context the trained panel method can provide value in HCI research and practice.