Multimedia Understanding meets Social Media Analysis

Social media is emerging as the predominant communication platform. Globally, the number of active users is estimated to be 4.62
billion (58.4% of the total population). As a consequence, the need for automatic processing, understanding and monitoring of
communication patterns has significantly increased. Detection of misinformation, polarization and malicious communities is also a
crucial step to identify hoaxes and monitor online content. To address these issues, the research community has proposed social
media analysis algorithms, which, so far, have been primarily based on graph and network methods.
In this context, online content is becoming increasingly available in mixed modalities (text, images, videos, etc.). The convergence of
Computer Vision (CV) and Natural Language Processing (NLP) has made it possible to empower textual and image understanding,
and, more recently, to link textual and visual information to enable multi-modal understanding and retrieval. However, these recent
efforts have been seldom applied to social media analysis, missing the benefit of developing innovative approaches to jointly
understand text and other modalities, and provide an effective understanding of communication patterns. Indeed, there is a lack of a
unified social media analysis methodology, which provides a seamless integration between network analysis and multimodal
processing of visual and textual data.

The MUSMA project will make a radical change by investigating and developing innovative analysis models that can jointly process
and understand textual and visual content simultaneously. MUSMA will enable:
- processing, understanding, selection and monitoring of online content, to extract relevant information on a set of specific topics or
subjects;
- misinformation and manipulated content detection, through the development of AI techniques specifically designed to identify the
network-wide phenomena (e.g. emerging communities and viral content) and credibility of online content;
- analysis of the main drivers of information consumption, the dynamics of information and misinformation flow and ranking
information sources according to their topical influence.
At the core of the project lies a new unifying synergy between Network Science, NLP and CV, using supervised neural networks
(going beyond convolutive autoencoders, Transformer-based NNs, Capsules and graph-based networks) and symbolic
representations.
This 2-year project brings together the research experiences and expertise of three internationally-recognized research teams: the
AILAB of the University of Udine, the Data and Complexity for Society Lab at Roma Sapienza, and AImageLab at UNIMORE,
encompassing NLP, vision and network science. The project proposes foundational research with direct practical and industrial
exploitation. We foresee an enormous potential benefit for the society and as well as in paving the way to new research directions in
several areas of AI.