Please use this identifier to cite or link to this item: http://ktisis.cut.ac.cy/handle/10488/10120
Title: Deep network regularization via bayesian inference of synaptic connectivity
Authors: Partaourides, Charalampos 
Chatzis, Sotirios P. 
Keywords: Bayesian networks;Data mining;Hierarchical systems;Inference engines;Iterative methods
Category: Electrical Engineering - Electronic Engineering - Information Engineering
Field: Engineering and Technology
Issue Date: 1-Jan-2017
Publisher: Springer Verlag
Source: Advances in Knowledge Discovery and Data Mining, 2017, Pages 30-41
metadata.dc.doi: 10.1007/978-3-319-57454-7_3
Abstract: Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic examples of such regularizers, that are widely popular among practitioners. However, a drawback of such approaches consists in the fact that their postulated probability of random unit/connection omission is a constant that must be heuristically selected based on the obtained performance in some validation set. To alleviate this burden, in this paper we regard the DNN regularization problem from a Bayesian inference perspective: We impose a sparsity-inducing prior over the network synaptic weights, where the sparsity is induced by a set of Bernoulli-distributed binary variables with Beta (hyper-)priors over their prior parameters. This way, we eventually allow for marginalizing over the DNN synaptic connectivity for output generation, thus giving rise to an effective, heuristics-free, network regularization scheme. We perform Bayesian inference for the resulting hierarchical model by means of an efficient Black-Box Variational inference scheme. We exhibit the advantages of our method over existing approaches by conducting an extensive experimental evaluation using benchmark datasets.
URI: http://ktisis.cut.ac.cy/handle/10488/10120
ISBN: 9783319574530
Rights: © 2017, Springer International Publishing AG
Type: Book Chapter
Appears in Collections:Κεφάλαια βιβλίων/Book chapters

Show full item record

Page view(s) 50

75
Last Week
1
Last month
8
checked on Feb 23, 2019

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.