Automatic Instance Generation for Classical Planning

The benchmarks from previous International Planning Com-petitions (IPCs) are the de-facto standard for evaluating plan-ning algorithms. The IPC set is both a collection of planningdomains and a selection of instances from these domains.Most of the domains come with a parameterized generatorthat generates new instances for a given set of parameter val-ues. Due to the steady progress of planning research some ofthe instances that were generated for past IPCs are inadequatefor evaluating current planners. To alleviate this problem, weintroduce Autoscale, an automatic tool that selects instancesfor a given domain. Autoscale takes into account constraintsfrom the domain designer as well as the performance of cur-rent planners to generate an instance set of appropriate diffi-culty, while avoiding too much bias with respect to the con-sidered planners. We show that the resulting benchmark setis superior to the IPC set and has the potential of improvingempirical evaluation of planning research.