Feature extraction for tweet classification: do the humans perform better?

Abstract

Sentiment analysis of Twitter data became a research trend the last decade. Thanks to the Twitter API, massive amounts of tweets, relating to a topic of interest, can be collected in real time. Performing sentiment analysis of these tweets can be used to conduct social sensing and opinion mining. For instance, forecasting elections is a primary area in which sentiment analysis of tweets has been extensively applied the last few years. Sentiment analysis of Twitter data presents important challenges compared to the similar task of text classification. Tweets are limited to 140 characters; thus, the conveyed message is compressed and often context-dependent. The tweets are informal and unstructured, usually lacking grammatical soundness and use of a standard lexicon. On the other hand, tweets are usually annotated by their authors regarding their topic and sentiment with the aid of hashtags and emoticons. Identifying appropriate features for sentiment analysis of tweets remains an open research area since text indexing methods face the sparseness problem while POS tagging methods fail due to the lack of grammatical structure of tweets. Character based features, i.e., n-grams of characters, are currently getting popular because they are language independent. However, their effectiveness remains quite low. In this paper, we argue that tokens used by humans for sentiment analysis of tweets are probably the best feature set one can use for that purpose. We compare several automatically extracted features with the features (tokens) used by humans for tweet classification, under a machine learning framework. The results show that the manually indicated tokens combined with a Decision Tree classifier outperform any other feature set-classification algorithm combination. The manually annotated dataset that was used in our experiments is publicly available for anyone who wishes to use it.