Comparative Analysis of Various Textual-Visual Models for Self-Attentive Query Focused Video Summarization - Sheetal Girase (International Journal of Intelligent Systems and Applications in Engineering, 2024)
The exponential growth of video data presents a significant challenge in extracting pertinent information from it. Video summarization aims to address this issue by extracting essential information from video data in order to facilitate the exploration of videos. Given the subjective nature of determining "relevant information" in a video based on user preferences, it is imperative to establish a mechanism that takes into account the users' preferences during the process of generating a summary. One approach that can be employed is to enable users to input a query. Rather than generating a predetermined and inflexible summary for a given video input, this study has investigated a method of generating a video summary that caters to the preferences of the user. Query Focused Video Summarization (QFVS) is regarded as a supervised learning problem in the context of the YouTube Dataset [4]. It aims to produce a summary based on user inputs, specifically the video and the textual query. The query relevance of frames from the video is determined by mapping them to a shared multimodal semantic embedding space. By utilising our attention network and encoder, we have successfully enhanced the accuracy rate from 61.91% [4] to 74.60%. Extensive experiments were conducted utilising deep learning models, specifically ResNet34 and DenseNet, to extract image features. Additionally, word2vec and GloVe were employed for word mappings. The integration of textual and image features is employed for diverse experimental purposes.