Please use this identifier to cite or link to this item:
|Title:||Deep network regularization via bayesian inference of synaptic connectivity||Authors:||Partaourides, Charalampos
Chatzis, Sotirios P.
|Issue Date:||1-Jan-2017||Publisher:||Springer Verlag||Source:||Advances in Knowledge Discovery and Data Mining, 2017, Pages 30-41||Abstract:||Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are widely popular among practitioners. However, a drawback of such approaches consists in the fact that their postulated probability of random unit/connection omission is a constant that must be heuristically selected based on the obtained performance in some validation set. To alleviate this burden, in this paper we regard the DNN regularization problem from a Bayesian inference perspective: We impose a sparsity-inducing prior over the network synaptic weights, where the sparsity is induced by a set of Bernoulli-distributed binary variables with Beta (hyper-)priors over their prior parameters. This way, we eventually allow for marginalizing over the DNN synaptic connectivity for output generation, thus giving rise to an effective, heuristics-free, network regularization scheme. We perform Bayesian inference for the resulting hierarchical model by means of an efficient Black-Box Variational inference scheme. We exhibit the advantages of our method over existing approaches by conducting an extensive experimental evaluation using benchmark datasets.||URI:||http://ktisis.cut.ac.cy/handle/10488/10120||ISBN:||9783319574530||Rights:||© 2017, Springer International Publishing AG|
|Appears in Collections:||Κεφάλαια βιβλίων/Book chapters|
Show full item record
checked on Aug 23, 2017
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.