Automatic Metamorphic Test Oracles for Action-Policy Testing

Jan Eisenhut, Álvaro Torralba, Maria Christakis, Jörg Hoffmann

Research output: Contribution to book/anthology/report/conference proceedingArticle in proceedingResearchpeer-review

Abstract

Testing is a promising way to gain trust in learned action policies p. Prior work on action-policy testing in AI planning formalized bugs as states t where p is sub-optimal with respect to a given testing objective. Deciding whether or not t is a bug is as hard as (optimal) planning itself. How can we design test oracles able to recognize some states t to be bugs efficiently? Recent work introduced metamorphic oracles which compare policy behavior on state pairs (s, t) where t is easier to solve; if p performs worse on t than on s, we know that t is a bug. Here, we show how to automatically design such oracles in classical planning, based on simulation relations between states. We introduce two oracle families of this kind: first, morphing query states t to obtain suitable s; second, maintaining and comparing upper bounds on h* across the states encountered during testing. Our experiments on ASNet policies show that these oracles can find bugs much more quickly than the existing alternatives, which are search-based; and that the combination of our oracles with search-based ones almost consistently dominates all other oracles.

Original languageEnglish
Title of host publicationProceedings of the Thirty-Third International Conference on Automated Planning and Scheduling
Number of pages9
PublisherAAAI Press
Publication date2023
Pages109-117
ISBN (Electronic)978-1-57735-881-7
DOIs
Publication statusPublished - 2023
Event33rd International Conference on Automated Planning and Scheduling, ICAPS 2023 - Prague, Czech Republic
Duration: 8 Jul 202313 Jul 2023

Conference

Conference33rd International Conference on Automated Planning and Scheduling, ICAPS 2023
Country/TerritoryCzech Republic
CityPrague
Period08/07/202313/07/2023
SeriesProceedings International Conference on Automated Planning and Scheduling, ICAPS
Number1
Volume33
ISSN2334-0835

Bibliographical note

Publisher Copyright:
Copyright © 2023, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

Fingerprint

Dive into the research topics of 'Automatic Metamorphic Test Oracles for Action-Policy Testing'. Together they form a unique fingerprint.

Cite this