Classification of Instagram photos: Topic modelling vs transfer learning

Abstract

The existence of pre-trained deep learning models for image classification, such as those trained on the well-known Resnet-50 architecture, allows for easy application of transfer learning to several domains including image retrieval. Recently, we proposed topic modelling for the retrieval of Instagram photos based on the associated hashtags. In this paper we compare content-based image classification, based on transfer learning, with the classification based on topic modelling of Instagram hashtags for a set of 24 different concepts. The comparison was performed on a set of 1944 Instagram photos, 81 per concept. Despite the excellent performance of the pre-trained deep learning models, it appears that text-based retrieval, as performed by the topic models of Instagram hashtags, stills perform better.