Bias by design in large language model-based review annotation: a cross-model analysis of ChatGPT-5, Qwen-3, and Gemini-3-flash
Date Issued
April 2026
Author(s)
Andreou, Christiana
Abstract
Large Language Models (LLMs) are increasingly used to analyse online reviews, mainly in sentiment analysis and aspect-based annotation. Despite offering efficient solutions, these models raise concerns about their annotation reliability and their ability to interpret data in ways consistent with human interpretation
This study examines how different LLMs label and interpret reviews from the hospitality industry. This study uses the Hotel Reviews: Aspects, Sentiments and Topics (HRAST) dataset which includes 16,813 reviews (23,114 sentences) collected from Booking.com, including both real and synthetic data. It compares the annotations of ChatGPT-5, Qwen-3, and Gemini-3-flash with human annotations.
The findings show that the models differ in their performance. ChatGPT-5 tends to assign more neutral labels, while Qwen-3 and Gemini-3-flash are closer to human evaluations. Agreement is higher in explicit, concrete aspects (e.g., room, location, price) but lower in more abstract categories (e.g., experience, comfort, atmosphere). In addition, synthetic data increases agreement between LLMs and human annotations but reduces interpretability. The findings show that annotation bias is not random but related to how models are designed and how data is constructed.
This study examines how different LLMs label and interpret reviews from the hospitality industry. This study uses the Hotel Reviews: Aspects, Sentiments and Topics (HRAST) dataset which includes 16,813 reviews (23,114 sentences) collected from Booking.com, including both real and synthetic data. It compares the annotations of ChatGPT-5, Qwen-3, and Gemini-3-flash with human annotations.
The findings show that the models differ in their performance. ChatGPT-5 tends to assign more neutral labels, while Qwen-3 and Gemini-3-flash are closer to human evaluations. Agreement is higher in explicit, concrete aspects (e.g., room, location, price) but lower in more abstract categories (e.g., experience, comfort, atmosphere). In addition, synthetic data increases agreement between LLMs and human annotations but reduces interpretability. The findings show that annotation bias is not random but related to how models are designed and how data is constructed.
Affiliation
File(s)![Thumbnail Image]()
Name
CHRISTIANA ANDREOU.MSC.2026.ABSTRACT.pdf
Size
155.92 KB
Format
Adobe PDF
Checksum (MD5)
2740b7f05164b7127847affadc08bc71