Video representation and retrieval using spatio-temporal descriptors and region relations

Abstract

This paper describes a novel methodology for video summarization and representation. The video shots are processed in space-time as 3D volumes of pixels. Pixel regions with consistent color and motion properties are extracted from these 3D volumes by a space-time segmentation technique based on a novel machine learning algorithm. Each region is then described by a high-dimensional point whose components represent the average position, motion velocity and color of the region. Subsequently, the spatio-temporal relations of the regions are deduced and a concise, graph-based description of them is generated. This graph-based description of the video shot's content, along with the region centroids, comprises a concise yet powerful description of the video-shot and is used for retrieval applications. The retrieval problem is formulated as an inexact graph matching problem between the data video shots and the query input which is also a video segment. Experimental results on action recognition and video retrieval are illustrated and discussed