Please use this identifier to cite or link to this item: http://ktisis.cut.ac.cy/handle/10488/8190
Title: Sparse bayesian recurrent neural networks
Authors: Chatzis, Sotirios P. 
Keywords: Recurrent neural networks
RNN
Bayesian regression
Issue Date: 2015
Source: European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2015, Porto, Portugal, 7-11 September
Abstract: Recurrent neural networks (RNNs) have recently gained renewed attention from the machine learning community as effective methods for modeling variable-length sequences. Language modeling, handwriting recognition, and speech recognition are only few of the application domains where RNN-based models have achieved the state-of- the-art performance currently reported in the literature. Typically, RNN architectures utilize simple linear, logistic, or softmax output layers to perform data modeling and prediction generation. In this work, for the first time in the literature, we consider using a sparse Bayesian regression or classification model as the output layer of RNNs, inspired from the automatic relevance determination (ARD) technique. The notion of ARD is to continually create new components while detecting when a component starts to overfit, where overfit manifests itself as a precision hyperparameter posterior tending to infinity. This way, our method manages to train sparse RNN models, where the number of effective (“active”) recurrently connected hidden units is selected in a data-driven fashion, as part of the model inference procedure. We develop efficient and scalable training algorithms for our model under the stochastic variational inference paradigm, and derive elegant predictive density expressions with computational costs comparable to conventional RNN formulations. We evaluate our approach considering its application to challenging tasks dealing with both regression and classification problems, and exhibit its favorable performance over the state-of-the-art.
URI: http://172.16.21.8/jspui/handle/10488/8190
Appears in Collections:Δημοσιεύσεις σε συνέδρια/Conference papers

Files in This Item:
File Description SizeFormat 
Chatzis.pdf176.51 kBAdobe PDFView/Open
Show full item record

Page view(s)

1
checked on Jan 30, 2017

Download(s) 10

3
checked on Jan 30, 2017

Google ScholarTM

Check


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.