BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule

Miao Zhang, Shirui Pan, Xiaojun Chang, Steven Su, Jilin Hu, Gholamreza Haffari, Bin Yang

Publikation: Bidrag til bog/antologi/rapport/konference proceedingKonferenceartikel i proceedingForskningpeer review

1 Downloads (Pure)

Abstrakt

Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost through weight sharing and continuous relaxation. However, more recent works find that existing differentiable NAS techniques struggle to outperform naive baselines, yielding deteriorative architectures as the search proceeds. Rather than directly optimizing the architecture parameters, this paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. By leveraging the natural-gradient variational inference (NGVI), the architecture distribution can be easily optimized based on existing codebases without incurring more memory and computational consumption. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability. The experimental results on NAS benchmark datasets confirm the significant improvements the proposed framework can make. In addition, instead of simply applying the argmax on the learned parameters, we further leverage the recently-proposed training-free proxies in NAS to select the optimal architecture from a group architectures drawn from the optimized distribution, where we achieve state-of-the-art results on the NAS-Bench-201 and NAS-Bench-1shot1 benchmarks. Our best architecture in the DARTS search space also obtains competitive test errors with 2.37%, 15.72%, and 24.2% on CIFAR-10, CIFAR-100, and ImageNet, respectively.
OriginalsprogEngelsk
TitelProceedings - 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022
Antal sider10
Publikationsdato2022
Sider11861-11870
ISBN (Trykt)978-1-6654-6947-0
ISBN (Elektronisk)978-1-6654-6946-3
DOI
StatusUdgivet - 2022
BegivenhedIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) - New Orleans, USA
Varighed: 18 jun. 202224 jun. 2022

Konference

KonferenceIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Land/OmrådeUSA
ByNew Orleans
Periode18/06/202224/06/2022

Fingeraftryk

Dyk ned i forskningsemnerne om 'BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule'. Sammen danner de et unikt fingeraftryk.

Citationsformater