Behavior lEarning in Surveilled Areas with Feature Extraction

This project is unique since it aims at combining two main areas of research, Computer Vision and Machine Learning, in an application of automatic surveillance for people detection and tracking and abnormal behavior recognition. Computer Vision (CV) and Machine Learning (ML) have been used jointly for many different applications but either using ML as a tool for computer vision applications or using CV as a case study to proof theoretical advances in ML. The project aims at exploring how visual features can be automatically extracted from video using computer vision techniques and exploited by a classifier (generated by machine learning) to detect and identify suspicious people behavior in public places in real time. In this sense, CV and ML are jointly developed and studied to provide a better mix of innovative techniques.

Justification of the proposed project is based on two issues of major concern to the state of Israel:

1. the need for intelligent surveillance in public and commercial areas that are susceptible to terrorist attacks;
2. lack of automatic and intelligent decision support in existing surveillance systems.

More specifically, the objectives of the project are:

1. to achieve a better understanding of which visual features can be used for
   1. analyzing people activity
   2. characterizing people shape;
2. to suitably adapt ML techniques such as HMM, SVM or methods for "novelty detection" in order to infer from the visual features extracted the behavior of the people and possible classifying it as normal or abnormal;
3. develop a first simple prototype in a specific scenario that can be considered as a threat for security.

The machine learning research is carried out at the Hebrew University's machine learning lab utilizing its long experience in temporal pattern recognition and computational learning methods. Following the meeting in June 2007 in Jerusalem, we decided to focus in the available time for the project on one particular behavior which is both well defined and threatening: people who leave objects behind them (such as luggage in airports). The machine learning component is based on the following phases:

1. constructing a generative statistical model of human gait on the basis of the features provided by the CV group. Such a model is an adaptation of an oscillatory dynamic model we developed in the past (Singer and Tishby 1994), where different points on the walking person are assumed having a drifted oscillatory motion with characteristic frequency and relative phases;
2. this basic Oscillatory Gait Model (OGM) is then plugged as the output of a state of an HMM, yielding a complete statistical model of regular gait;
3. detecting deviations (irregularities) in the relative phases and amplitudes of the OGM to capture irregular behavior, e.g. halting, bending, leaving objects, etc. The output of such a statistical model can be classified using likelihood ratio tests or standard classifiers as SVM to improve confidence;
4. We also carried work on detecting statistical irregularities in multivariate correlated data, as another component of the project.

Video Demo