Unimore logo AImageLab

Spot the Difference

F. Landi, R. Bigazzi, M. Cornia, S. Cascianelli, L. Baraldi, R. Cucchiara

 

Spot the Difference is a dataset of 2D semantics occupancy maps (SOMs) in which the objects can be added, removed, and rearranged while the area and the position of architectural elements do not change. In fact, using semantic annotations in Gibson and Matterport3D dataset we consider categories that have a high probability of being displaced or removed in the real world and ignore non-movable semantic categories such as fireplaces, columns, and stairs.

The dataset contains also exploration episodes to train and evaluate an agent that is required to find the differences between the actual state of the map and an outdated version of it.

 

Paper Download GitHub

 

Please cite with the following BibTeX:

@inproceedings{landi2022spot,
    title={Spot the Difference: A Novel Task for Embodied Agents in Changing Environments},
    author={Landi, Federico and Bigazzi, Roberto and Cornia, Marcella and Cascianelli, Silvia and Baraldi, Lorenzo and Cucchiara, Rita},
    booktitle={International Conference on Pattern Recognition},
    year={2022}
}

Dataset info

The semantic occupancy maps are generated using Habitat Simulator on the environments of Matterport3D and Gibson Tiny datasets. The size of the maps is 961x961 on Gibson environments and 2001x2001 on Matterport3D environments, where each pixel represents the occupancy state of a 5cm x 5cm real-world area.

For the generation of alternative SOMs, we start from the 90 building scans of Matterport3D (MP3D) dataset and from the 5 building scans of Gibson Tiny.  For each building, we compute the number of floors in each building and compute the SOM for each floor. We create 10 alternative versions of each SOM. In this step, we discard floors that have few semantic objects (e.g., empty rooftops) or that are not fully navigable by the agent. As a result, we retain 249 floors belonging to 81 different buildings of Matterport3D and 13 floors in 5 building of Gibson, thus generating a total of 2490+130 different semantic occupancy maps for these floors. Finally, we split the dataset into train, validation, and test subsets. 


Operations on Each Semantic Category

        Matterport3D

Idx   Category              Action
0 Void No Operation
1 Wall No Operation
2 Floor No Operation
3 Chair Displacement
4 Door No Operation
5 Table Displacement
6 Picture No Operation
7 Cabinet Removal
8 Cushion Overlap Removal
9 Window No Operation
10 Sofa Displacement
11 Bed Displacement
12 Curtain No Operation
13 Chest of Drawers Displacement
14 Plant Displacement
15 Sink Empty
16 Stairs No Operation
17 Ceiling No Operation
18 Toilet Removal
19 Stool Displacement
20 Towel Overlap Removal
21 Mirror No Operation
22 TV Monitor Removal
23 Shower Removal
24 Column No Operation
25 Bathtub Removal
26 Counter Removal
27 Fireplace No Operation
28 Lighting No Operation
29 Beam No Operation
30 Railing No Operation
31 Shelving Removal
32 Blinds No Operation
33 Gym Equipment Displacement
34 Seating Removal
35 Board Panel No Operation
36 Furniture Displacement
37 Appliances Removal
38 Clothes Overlap Removal
39 Objects Overlap Removal
40 Misc Overlap Removal
41 Unlabeled No Operation

                    

        Gibson Tiny

Idx   Category        Action
0 Chair Displacement
1 Couch Displacement
2 Potted Plant Removal
3 Bed Displacement
4 Toilet Removal
5 TV Removal
6 Dining Table Displacement
7 Oven Removal
8 Sink Removal
9 Refrigerator Removal
10 Book Overlap Removal
11 Clock Removal
12 Vase Removal
13 Cup Overlap Removal
14 Bottle Overlap Removal
15 Bench Removal
16 Appliances Removal
17 Objects Overlap Removal
18 Misc Overlap Removal
19 Void No Operation

                    


Exploration Episodes

        Matterport3D Training

Scan   Floors # Episodes
HxpKQynjfin   0 81967
gTV8FGcVJC9   0,1,2,3,4,6,10,11 77186
29hnd4uzFmX   0,1,2,3 81967
5LpN3gDmAk7   0,1,2,3 81885
SN83YJsR3w2   0,1,2,3,7,8,10,12 81438
VzqfbhrpDEA   0,1,3,6 81641
D7N2EKCX4Sj   0,1,2,3,5,6 81830
5q7pvUzZiYa   0,1,2,3,4 81967
ac26ZMwG7aT   0,1 81967
r47D5H71a5s   0,1 81965
Pm6F8kyY3z2   0 81967
8WUmhLawc2A   0,1,2 81967
82sE5b5pLXE   0,1,2 80682
mJXqzFtmKg4   0,1,2 81967
i5noydFURQK   0,1 81120
V2XKFyX4ASd   0,1,2,3,4,5,7 81129
759xd9YjKW5   0,1,2,3 81913
r1Q1Z4BcV1o   0 81812
S9hNv5qa7GM   0,1 81967
1LXtFkjw3qL   0,1,2,3,4,5,6 81967
PuKPg4mmafe   0 81940
EDJbREhghzL   0,1,3 64755
ur6pFq6Qu1A   0,1 81967
B6ByNegPMKs   0 81951
b8cTxDM8gDG   0,1,2,8,11 73307
17DRP5sb8fy   0 81967
YmJkqBEsHnH   0 80780
ULsKaCPVFJR   0,1,2 81967
XcA2TqTSSAj   0,2,3,5,6,8,9,11,12    60679
sKLMLpTHeUy   0,1,2,4 79736
ZMojNkEp431   0,1,2 81967
e9zR4mvMWw7     0,1,2 80193
JeFG25nYj2p   0,1 81967
uNb9QFRL6hY   1,4,5,6 59613
p5wJjkQkbXX   0,1,2,3 81967
Vvot9Ly1tCj   0,3 78115
E9uDoFAP3SH   0,1,5,6 81914
qoiz87JEwZ2   0,1,2,3 81967
VFuaQ6m2Qom   0,1,2,4,5,6 81758
VLzqgDo317F   0,1,2 81396
kEZ7cmS4wCh   0,1,2,3,7 69135
7y3sRwLe3Va   0,1,2,5 81386
VVfe2KiqLaN   0,1,2 81967
2n8kARJN3HM   0,1,2,4 81076
PX4nDJXEHrG   0,1,2,3,4,5 79151
Uxmj2M2itWa   0,1,3,4 49942
pRbA3pwrgk9   0,2,3,7,9,11 53295
cV4RVeZvu5T   0,1,2,3 81038
sT4fr6TAbpF   0 81625
GdvgFV5R1Z5   0 81967
JF19kD82Mey   0,1,2 81927
JmbYfDe2QKZ   0,1 81489
s8pcmisQ38h   0,1,2 80428
1pXnuDYAj8r   0,1,2,5 81901
jh4fc5c5qoQ   0,1,2 81967
vyrNrziPKCB   0,1,3,4,7 81388
aayBHfsNo7d   0,1,2 81693
rPc6DW4iMge   0,1,3,4 80296
Total: 58   207 4581881

            

        Matterport3D Validation

Scan   Floors    # Episodes
2azQ1b91cZZ   0,1 40
8194nk5LbLH   0 40
EU6Fwq7SyZv   0 30
QUCTc6BB5sX   1 20
TbHJrupSAjP   0,1,2 30
Z6MFQCViBuw   0 40
oLBMNvg9in8   0,1,2,3      50
x8F5xyUWy9e       0,1 30
zsNo4HB9uLZ   0 40
Total: 9   16 320

            

        Matterport3D Test

Scan   Floors # Episodes
2t7WUuJeko7   0 50
5ZKStnWn8Zo   0,1 50
RPmz2sHmrrY   0 50
UwV83HsGsw3   0,1,2,3     50
WYY7iVyf5p8   0,2 30
YFuZgdQ5vWj   1 10
YVUC4YcDtcY   0 50
fzynW3qQPVF   0,1 50
jtcxE69GiFV   0,1 40
pa4otMbVnkk   0,1 50
q9vSo1VnCiC   0 50
rqfALeAoiTq   0,2 20
wc2JMjhGNzB   0,1 50
yqstnuAEVhm   0,1,2 60
Total: 14   26 610

            

       Gibson Test

Scan   Floors # Episodes
Wiconisco   1,2 90
Corozal   0,2,4 90
Collierville   0,1,2 80
Markleeville   0,1 90
Darden   0,1,2 100
Total: 5   13 450


Contacts

If you have any doubts about our dataset, please use the public issues section on this github repo. Alternatively, drop us an e-mail at roberto.bigazzi [at] unimore.it.


Acknowledgment

This work was supported by the “European Training Network on PErsonalized Robotics as SErvice Oriented applications” (PERSEO) MSCA-ITN-2020 project (G.A. 955778).