Deep Transformer Neural Networks with Stochastic Competition

Abstract

Transformers have become one of the most successful architectures in deep learning, experiencing a steady rise in popularity. These advanced networks have revolutionized the field of Natural Language Processing (NLP) and are extending their influence into new domains within artificial intelligence and beyond. The recent rise of large language models, fundamentally reliant on Transformer architectures, highlights their effectiveness and underscores their transformative impact. This thesis delves into exploring further capabilities of this deep learning framework by incorporating stochastic methodologies as an essential component of Transformer networks. Our primary focus is on leveraging stochastic competition techniques, proven to be highly advantageous in various contexts, as the cornerstone for developing highperforming models. Instead of focusing on the extensively researched application areas such as NLP, our research pivots to exploring two distinct and significantly different fields: i) Sign Language Translation and ii) Tabular Data Modeling.