Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019

Zhengying Liu, Adrien Pavao, Zhen Xu, Sergio Escalera, Fabio Ferreira, Isabelle Guyon, Sirui Hong, Frank Hutter, Rongrong Ji, Julio C. Junior, Ge Li, Marius Lindauer, Luo Zhipeng, Meysam Madadi, Thomas Nierhoff, Kangning Niu, Chunguang Pan, Danny Stoll, Sebastien Treger, Wang JinPeng Wang, Chengling Wu, Youcheng Xiong, Arber Zela, Yang Zhang

Research output: Contribution to journalJournal articleResearch

13 Citations (Scopus)

Abstract

This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a ‘`meta-learner’', ‘`data ingestor’', ‘`model selector’', ‘`model/learner’', and ‘`evaluator’'. This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free 'AutoDL self-service''.
Original languageEnglish
Article number9415128
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume43
Issue number9
Pages (from-to)3108-3125
Number of pages18
ISSN1939-3539
DOIs
Publication statusPublished - 2021
Externally publishedYes

Keywords

  • AutoML
  • Benchmark testing
  • Computer architecture
  • Deep Learning
  • Deep learning
  • Hyperparameter Optimization
  • Internet
  • Meta-learning
  • Model Selection
  • Neural Architecture Search
  • Task analysis
  • Tensors
  • Videos

Fingerprint

Dive into the research topics of 'Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019'. Together they form a unique fingerprint.

Cite this