Abstract
The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity. DARTS leverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handled with gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes. The vanilla DARTS assumes the optimized magnitudes reflect the importance of operations, while more recent works find this naive assumption leads to poor generalization and is without any theoretical guarantees. In this work, we leverage influence functions, the functional derivatives of the loss function, to theoretically reveal the operation selection part in DARTS and estimate the candidate operation importance by approximating its influence on the supernet with Taylor expansions. We show the operation strength is not only related to the magnitude but also second-order information, leading to a fundamentally new criterion for operation selection in DARTS, named Influential Magnitude. Empirical studies across different tasks on several spaces show that vanilla DARTS and its variants can avoid most failures by leveraging the proposed theory-driven operation selection criterion.
Originalsprog | Engelsk |
---|---|
Titel | Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022 |
Redaktører | S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh |
Forlag | Neural Information Processing Systems Foundation |
Publikationsdato | 2022 |
ISBN (Elektronisk) | 9781713871088 |
Status | Udgivet - 2022 |
Begivenhed | 36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, USA Varighed: 28 nov. 2022 → 9 dec. 2022 |
Konference
Konference | 36th Conference on Neural Information Processing Systems, NeurIPS 2022 |
---|---|
Land/Område | USA |
By | New Orleans |
Periode | 28/11/2022 → 09/12/2022 |
Navn | Advances in Neural Information Processing Systems |
---|---|
Vol/bind | 35 |
ISSN | 1049-5258 |
Bibliografisk note
Funding Information:This work was partially supported by Independent Research Fund Denmark under agreements 8022-00246B and 8048-00038B, the VILLUM FONDEN under agreement 34328, and the Innovation Fund Denmark centre, DIREC.
Publisher Copyright:
© 2022 Neural information processing systems foundation. All rights reserved.