Please use this identifier to cite or link to this item: http://ktisis.cut.ac.cy/handle/10488/10120
Title: Deep network regularization via bayesian inference of synaptic connectivity
Authors: Partaourides, Charalampos 
Chatzis, Sotirios P. 
Keywords: Bayesian networks
Data mining
Hierarchical systems
Inference engines
Iterative methods
Issue Date: 1-Jan-2017
Publisher: Springer Verlag
Source: Advances in Knowledge Discovery and Data Mining, 2017, Pages 30-41
Abstract: Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are widely popular among practitioners. However, a drawback of such approaches consists in the fact that their postulated probability of random unit/connection omission is a constant that must be heuristically selected based on the obtained performance in some validation set. To alleviate this burden, in this paper we regard the DNN regularization problem from a Bayesian inference perspective: We impose a sparsity-inducing prior over the network synaptic weights, where the sparsity is induced by a set of Bernoulli-distributed binary variables with Beta (hyper-)priors over their prior parameters. This way, we eventually allow for marginalizing over the DNN synaptic connectivity for output generation, thus giving rise to an effective, heuristics-free, network regularization scheme. We perform Bayesian inference for the resulting hierarchical model by means of an efficient Black-Box Variational inference scheme. We exhibit the advantages of our method over existing approaches by conducting an extensive experimental evaluation using benchmark datasets.
URI: http://ktisis.cut.ac.cy/handle/10488/10120
ISBN: 9783319574530
Rights: © 2017, Springer International Publishing AG
Appears in Collections:Κεφάλαια βιβλίων/Book chapters

Show full item record

Page view(s)

12
Last Week
0
Last month
2
checked on Aug 23, 2017

Google ScholarTM

Check


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.