Please use this identifier to cite or link to this item:
|Title:||Deep network regularization via bayesian inference of synaptic connectivity||Authors:||Partaourides, Charalampos
Chatzis, Sotirios P.
|Keywords:||Bayesian networks;Data mining;Hierarchical systems;Inference engines;Iterative methods||Category:||Electrical Engineering - Electronic Engineering - Information Engineering||Field:||Engineering and Technology||Issue Date:||1-Jan-2017||Publisher:||Springer Verlag||Source:||Advances in Knowledge Discovery and Data Mining, 2017, Pages 30-41||metadata.dc.doi:||10.1007/978-3-319-57454-7_3||Abstract:||Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are widely popular among practitioners. However, a drawback of such approaches consists in the fact that their postulated probability of random unit/connection omission is a constant that must be heuristically selected based on the obtained performance in some validation set. To alleviate this burden, in this paper we regard the DNN regularization problem from a Bayesian inference perspective: We impose a sparsity-inducing prior over the network synaptic weights, where the sparsity is induced by a set of Bernoulli-distributed binary variables with Beta (hyper-)priors over their prior parameters. This way, we eventually allow for marginalizing over the DNN synaptic connectivity for output generation, thus giving rise to an effective, heuristics-free, network regularization scheme. We perform Bayesian inference for the resulting hierarchical model by means of an efficient Black-Box Variational inference scheme. We exhibit the advantages of our method over existing approaches by conducting an extensive experimental evaluation using benchmark datasets.||URI:||http://ktisis.cut.ac.cy/handle/10488/10120||ISBN:||9783319574530||Rights:||© 2017, Springer International Publishing AG||Type:||Book Chapter|
|Appears in Collections:||Κεφάλαια βιβλίων/Book chapters|
Show full item record
Page view(s) 5075
checked on Feb 23, 2019
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.