Compression of dnns using magnitude pruning and nonlinear information bottleneck training

As Deep Neural Networks (DNNs) have achieved state-of- the-art performance in various scientific fields and applica- tions, the memory and computational complexity of DNNs have increased concurrently. The increased complexity re- quired by DNNs prohibits them from running on platforms with limited computational resources. This has sparked a re- newed interest in parameter pruning. We propose to replace the standard cross-entropy objective – typically used in clas- sification problems – with the Nonlinear Information Bottle- neck (NIB) objective to improve the accuracy of a pruned net- work. We demonstrate, that our proposal outperforms cross- entropy combined with global magnitude pruning for high compression rates on VGG-nets trained on CIFAR10. With approximately 97% of the parameters pruned, we obtain an accuracy of 87.63% and 88.22% for VGG-16 and VGG-19, respectively, where the baseline accuracy is 91.5% for the un- pruned networks. We observe that the majority of biases are pruned completely, and pruning parameters globally outper- forms layer-wise pruning.