Soft Dropout and Its Variational Bayes Approximation

Jiyang Xie, Zhanyu Ma, Guoqiang Zhang, Jing-Hao Xue, Zheng-Hua Tan, Jun Guo

Publikation: Bidrag til bog/antologi/rapport/konference proceedingKonferenceartikel i proceedingForskningpeer review

Abstrakt

Soft dropout, a generalization of standard 'hard' dropout, is introduced to regularize the parameters in neural networks and prevent overfitting. We replace the 'hard' dropout mask following a Bernoulli distribution with the 'soft' mask following a beta distribution to drop the hidden nodes in different levels. The soft dropout method can introduce continuous mask coefficients in the interval of [0, 1], rather than only zero and one. Meanwhile, in order to implement the adaptive dropout rate via adaptive distribution parameters, we respectively utilize the half-Gaussian distributed and the half-Laplace distributed variables to approximate the beta distributed masks and apply a variation of variational Bayes optimization called stochastic gradient variational Bayes (SGVB) algorithm to optimize the distribution parameters. In the experiments, compared with the standard soft dropout with fixed dropout rate, the adaptive soft dropout method generally improves the performance. In addition, the proposed soft dropout and its adaptive versions achieve performance improvement compared with the referred methods on both image classification and regression tasks.

OriginalsprogEngelsk
Titel2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)
Antal sider6
ForlagIEEE
Publikationsdato5 dec. 2019
Sider1-6
Artikelnummer8918818
ISBN (Trykt)978-1-7281-0825-4
ISBN (Elektronisk)978-1-7281-0824-7
DOI
StatusUdgivet - 5 dec. 2019
Begivenhed2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) - Pittsburgh, USA
Varighed: 13 okt. 201916 okt. 2019

Konference

Konference2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)
LandUSA
ByPittsburgh
Periode13/10/201916/10/2019
NavnIEEE International Workshop on Machine Learning for Signal Processing (MLSP). Proceedings.

Fingeraftryk Dyk ned i forskningsemnerne om 'Soft Dropout and Its Variational Bayes Approximation'. Sammen danner de et unikt fingeraftryk.

  • Citationsformater

    Xie, J., Ma, Z., Zhang, G., Xue, J-H., Tan, Z-H., & Guo, J. (2019). Soft Dropout and Its Variational Bayes Approximation. I 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) (s. 1-6). [8918818] IEEE. IEEE International Workshop on Machine Learning for Signal Processing (MLSP). Proceedings. https://doi.org/10.1109/MLSP.2019.8918818