TY - JOUR
T1 - Assessing individual contributions to Collaborative Problem Solving
T2 - A network analysis approach
AU - Swiecki, Zachari
AU - Ruis, A. R.
AU - Farrell, Cayley
AU - Shaffer, David Williamson
PY - 2020/3
Y1 - 2020/3
N2 - Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. However, current methods for measuring the CPS processes of individuals, such as coding and counting, treat these processes as sets of isolated and independent events. In contrast, Epistemic Network Analysis (ENA) models how the contributions of a given individual relate to the contributions of others. This article examines the communications of air defense warfare teams from an experiment comparing two different computer-based decision support systems, using this data to ask whether ENA provides a more ecologically valid quantitative model of CPS than coding and counting. Qualitative analysis showed that commanders using one system asked questions to understand the tactical situation, while commanders using an experimental system focused more on actions in response to the tactical situation. Neither of the coding and counting approaches we tested corroborated these findings with statistically significant results. In contrast, ENA created models of the individual contributions of commanders that (a) showed statistical differences between commanders using the two systems to corroborate the qualitative analysis, and (b) revealed differences in individual performance. This suggests that ENA is a more powerful tool for CPS assessment than coding and counting approaches.
AB - Collaborative Problem Solving (CPS) is an interactive, interdependent, and temporal process. However, current methods for measuring the CPS processes of individuals, such as coding and counting, treat these processes as sets of isolated and independent events. In contrast, Epistemic Network Analysis (ENA) models how the contributions of a given individual relate to the contributions of others. This article examines the communications of air defense warfare teams from an experiment comparing two different computer-based decision support systems, using this data to ask whether ENA provides a more ecologically valid quantitative model of CPS than coding and counting. Qualitative analysis showed that commanders using one system asked questions to understand the tactical situation, while commanders using an experimental system focused more on actions in response to the tactical situation. Neither of the coding and counting approaches we tested corroborated these findings with statistically significant results. In contrast, ENA created models of the individual contributions of commanders that (a) showed statistical differences between commanders using the two systems to corroborate the qualitative analysis, and (b) revealed differences in individual performance. This suggests that ENA is a more powerful tool for CPS assessment than coding and counting approaches.
KW - Assessment
KW - Coding and counting
KW - Collaborative Problem Solving
KW - Epistemic Network Analysis
KW - Measurement
UR - http://www.scopus.com/inward/record.url?scp=85077654541&partnerID=8YFLogxK
U2 - 10.1016/j.chb.2019.01.009
DO - 10.1016/j.chb.2019.01.009
M3 - Journal article
AN - SCOPUS:85077654541
SN - 0747-5632
VL - 104
JO - Computers in Human Behavior
JF - Computers in Human Behavior
M1 - 105876
ER -