An Evaluation of Apache Jena and Apache Hive for Data Meshes: Using Semantic Enrichment and LLMs for data products recommendations - Photiou, Artemis (My University, 2025)
This thesis investigates two key challenges in the world of Data Lakes: The semantic organization of metadata and the automated generation of meaningful data products using an LLM. The first part of the thesis evaluates which system is most suitable for implementing a metadata enrichment mechanism, specifically comparing Apache Hive and Apache Jena in terms of scalability, query efficiency and storage performance. Experimental results demonstrate that Hive outperforms Jena in large scale environments. The second part of the thesis proposes a framework in which an LLM autonomously suggests data products by providing to the LLM a user-defined concept, metadata retrieved from Hive and sample records from HDFS. The purpose of this study is to examine whether the LLM can act as a domain expert by reasoning over structured and unstructured input and by performing external web searches. Specifically, the framework is evaluated across two domains and three complexity levels, measuring the precision and quality of the suggested data products. The results show that while the LLM performs well in simple scenarios, its effectiveness declines as concept complexity and dataset pool size increases.
Keywords: Semantic Enrichment; Apache Hive; Apache Jena; Data Lakes; Large Language Models