TY - JOUR
T1 - Artificial Intelligence in Clinical Decision Support
T2 - Challenges for Evaluating AI and Practical Implications
AU - Magrabi, Farah
AU - Ammenwerth, Elske
AU - McNair, Jytte Brender
AU - De Keizer, Nicolet F
AU - Hyppönen, Hannele
AU - Nykänen, Pirkko
AU - Rigby, Michael
AU - Scott, Philip J
AU - Vehko, Tuulikki
AU - Wong, Zoie Shui-Yee
AU - Georgiou, Andrew
N1 - Georg Thieme Verlag KG Stuttgart.
PY - 2019/8
Y1 - 2019/8
N2 - OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance.METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed.CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.
AB - OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance.METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed.CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.
KW - Artificial Intelligence
KW - Decision Support Systems, Clinical
KW - Evaluation Studies as Topic
KW - Machine Learning
KW - Program Evaluation/methods
UR - http://www.scopus.com/inward/record.url?scp=85071700730&partnerID=8YFLogxK
U2 - 10.1055/s-0039-1677903
DO - 10.1055/s-0039-1677903
M3 - Journal article
C2 - 31022752
SN - 0943-4747
VL - 28
SP - 128
EP - 134
JO - Yearbook of Medical Informatics
JF - Yearbook of Medical Informatics
IS - 1
ER -