Egocentric Vision for Detecting Social Relationships

Social interactions are so natural that we rarely stop wondering who is interacting with whom or which people are gathering into a group and who are not. Nevertheless, humans naturally do that neglecting that the complexity of this task increases when only visual cues are available. Different situations need different behaviors: while we accept to stand in close proximity to strangers when we attend some kind of public event, we would feel uncomfortable in having people we do not know close to us when we have a coffee. In fact, we rarely exchange mutual gaze with people we are not interacting with, an important clue when trying to discern different social clusters.

We address the problem of partitioning people in a video sequence into socially related groups from an egocentric vision (from now on, ego-vision) perspective. Human behavior is by no means random: when interacting with each other we generally stand in determined positions to avoid occlusions in our group, stand close to those we interact with and organize orientations to naturally be focused on the subjects we are interest in. Distance between individuals and mutual orientations assume clear significance and must be interpreted according to the situation. F-formation theory describes patterns that humans create naturally while interacting with each other and it can be used to understand whether a gathering of people forms a group or not based on the mutual distances and orientations of the subjects in the scene.

Publications

1	Alletto, Stefano; Serra, Giuseppe; Calderara, Simone; Cucchiara, Rita "Understanding social relationships in egocentric vision" PATTERN RECOGNITION, vol. 48, pp. 4082 -4096 , 2015 | DOI: 10.1016/j.patcog.2015.06.006 Journal
2	Alletto, Stefano; Serra, Giuseppe; Cucchiara, Rita "Egocentric Object Tracking: An Odometry-Based Solution" International Conference on Image Analysis and Processing - ICIAP 2015, vol. 9280, Genova, pp. 687 -696 , 5-11 September 2015, 2015 | DOI: 10.1007/978-3-319-23234-8_63 Conference
3	Alletto, Stefano; Serra, Giuseppe; Calderara, Simone; Cucchiara, Rita "Head Pose Estimation in First-Person Camera Views" International Conference on Pattern Recognition, Stockholm, Sweden, pp. 4188 -4193 , 24-28 Aug. 2014, 2014 | DOI: 10.1109/ICPR.2014.718 Conference
4	Alletto, Stefano; Serra, Giuseppe; Calderara, Simone; Solera, Francesco; Cucchiara, Rita "From Ego to Nos-Vision: Detecting Social Relationships in First-Person Views" 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Columbus, Ohio, 23-28 June 2014, 2014 | DOI: 10.1109/CVPRW.2014.91 Conference

Video Demo