Please use this identifier to cite or link to this item:
Title: Deep network regularization via bayesian inference of synaptic connectivity
Authors: Partaourides, Charalampos 
Chatzis, Sotirios P. 
Keywords: Bayesian networks;Data mining;Hierarchical systems;Inference engines;Iterative methods
Category: Electrical Engineering - Electronic Engineering - Information Engineering
Field: Engineering and Technology
Issue Date: 1-Jan-2017
Publisher: Springer Verlag
Source: Advances in Knowledge Discovery and Data Mining, 2017, Pages 30-41
metadata.dc.doi: 10.1007/978-3-319-57454-7_3
Abstract: Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are widely popular among practitioners. However, a drawback of such approaches consists in the fact that their postulated probability of random unit/connection omission is a constant that must be heuristically selected based on the obtained performance in some validation set. To alleviate this burden, in this paper we regard the DNN regularization problem from a Bayesian inference perspective: We impose a sparsity-inducing prior over the network synaptic weights, where the sparsity is induced by a set of Bernoulli-distributed binary variables with Beta (hyper-)priors over their prior parameters. This way, we eventually allow for marginalizing over the DNN synaptic connectivity for output generation, thus giving rise to an effective, heuristics-free, network regularization scheme. We perform Bayesian inference for the resulting hierarchical model by means of an efficient Black-Box Variational inference scheme. We exhibit the advantages of our method over existing approaches by conducting an extensive experimental evaluation using benchmark datasets.
ISBN: 9783319574530
Rights: © 2017, Springer International Publishing AG
Type: Book Chapter
Appears in Collections:Κεφάλαια βιβλίων/Book chapters

Show full item record

Page view(s) 50

Last Week
Last month
checked on Feb 23, 2019

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.