Spot the Difference
F. Landi, R. Bigazzi, M. Cornia, S. Cascianelli, L. Baraldi, R. Cucchiara
Spot the Difference is a dataset of 2D semantics occupancy maps (SOMs) in which the objects can be added, removed, and rearranged while the area and the position of architectural elements do not change. In fact, using semantic annotations in Gibson and Matterport3D dataset we consider categories that have a high probability of being displaced or removed in the real world and ignore non-movable semantic categories such as fireplaces, columns, and stairs.
The dataset contains also exploration episodes to train and evaluate an agent that is required to find the differences between the actual state of the map and an outdated version of it.
Please cite with the following BibTeX:
@inproceedings{landi2022spot,
title={Spot the Difference: A Novel Task for Embodied Agents in Changing Environments},
author={Landi, Federico and Bigazzi, Roberto and Cornia, Marcella and Cascianelli, Silvia and Baraldi, Lorenzo and Cucchiara, Rita},
booktitle={International Conference on Pattern Recognition},
year={2022}
}
Dataset info
The semantic occupancy maps are generated using Habitat Simulator on the environments of Matterport3D and Gibson Tiny datasets. The size of the maps is 961x961 on Gibson environments and 2001x2001 on Matterport3D environments, where each pixel represents the occupancy state of a 5cm x 5cm real-world area.
For the generation of alternative SOMs, we start from the 90 building scans of Matterport3D (MP3D) dataset and from the 5 building scans of Gibson Tiny. For each building, we compute the number of floors in each building and compute the SOM for each floor. We create 10 alternative versions of each SOM. In this step, we discard floors that have few semantic objects (e.g., empty rooftops) or that are not fully navigable by the agent. As a result, we retain 249 floors belonging to 81 different buildings of Matterport3D and 13 floors in 5 building of Gibson, thus generating a total of 2490+130 different semantic occupancy maps for these floors. Finally, we split the dataset into train, validation, and test subsets.
Operations on Each Semantic Category
Matterport3D
| Idx | Category | Action |
| 0 | Void | No Operation |
| 1 | Wall | No Operation |
| 2 | Floor | No Operation |
| 3 | Chair | Displacement |
| 4 | Door | No Operation |
| 5 | Table | Displacement |
| 6 | Picture | No Operation |
| 7 | Cabinet | Removal |
| 8 | Cushion | Overlap Removal |
| 9 | Window | No Operation |
| 10 | Sofa | Displacement |
| 11 | Bed | Displacement |
| 12 | Curtain | No Operation |
| 13 | Chest of Drawers | Displacement |
| 14 | Plant | Displacement |
| 15 | Sink | Empty |
| 16 | Stairs | No Operation |
| 17 | Ceiling | No Operation |
| 18 | Toilet | Removal |
| 19 | Stool | Displacement |
| 20 | Towel | Overlap Removal |
| 21 | Mirror | No Operation |
| 22 | TV Monitor | Removal |
| 23 | Shower | Removal |
| 24 | Column | No Operation |
| 25 | Bathtub | Removal |
| 26 | Counter | Removal |
| 27 | Fireplace | No Operation |
| 28 | Lighting | No Operation |
| 29 | Beam | No Operation |
| 30 | Railing | No Operation |
| 31 | Shelving | Removal |
| 32 | Blinds | No Operation |
| 33 | Gym Equipment | Displacement |
| 34 | Seating | Removal |
| 35 | Board Panel | No Operation |
| 36 | Furniture | Displacement |
| 37 | Appliances | Removal |
| 38 | Clothes | Overlap Removal |
| 39 | Objects | Overlap Removal |
| 40 | Misc | Overlap Removal |
| 41 | Unlabeled | No Operation |
Gibson Tiny
| Idx | Category | Action |
| 0 | Chair | Displacement |
| 1 | Couch | Displacement |
| 2 | Potted Plant | Removal |
| 3 | Bed | Displacement |
| 4 | Toilet | Removal |
| 5 | TV | Removal |
| 6 | Dining Table | Displacement |
| 7 | Oven | Removal |
| 8 | Sink | Removal |
| 9 | Refrigerator | Removal |
| 10 | Book | Overlap Removal |
| 11 | Clock | Removal |
| 12 | Vase | Removal |
| 13 | Cup | Overlap Removal |
| 14 | Bottle | Overlap Removal |
| 15 | Bench | Removal |
| 16 | Appliances | Removal |
| 17 | Objects | Overlap Removal |
| 18 | Misc | Overlap Removal |
| 19 | Void | No Operation |
Exploration Episodes
Matterport3D Training
| Scan | Floors | # Episodes | |
| HxpKQynjfin | 0 | 81967 | |
| gTV8FGcVJC9 | 0,1,2,3,4,6,10,11 | 77186 | |
| 29hnd4uzFmX | 0,1,2,3 | 81967 | |
| 5LpN3gDmAk7 | 0,1,2,3 | 81885 | |
| SN83YJsR3w2 | 0,1,2,3,7,8,10,12 | 81438 | |
| VzqfbhrpDEA | 0,1,3,6 | 81641 | |
| D7N2EKCX4Sj | 0,1,2,3,5,6 | 81830 | |
| 5q7pvUzZiYa | 0,1,2,3,4 | 81967 | |
| ac26ZMwG7aT | 0,1 | 81967 | |
| r47D5H71a5s | 0,1 | 81965 | |
| Pm6F8kyY3z2 | 0 | 81967 | |
| 8WUmhLawc2A | 0,1,2 | 81967 | |
| 82sE5b5pLXE | 0,1,2 | 80682 | |
| mJXqzFtmKg4 | 0,1,2 | 81967 | |
| i5noydFURQK | 0,1 | 81120 | |
| V2XKFyX4ASd | 0,1,2,3,4,5,7 | 81129 | |
| 759xd9YjKW5 | 0,1,2,3 | 81913 | |
| r1Q1Z4BcV1o | 0 | 81812 | |
| S9hNv5qa7GM | 0,1 | 81967 | |
| 1LXtFkjw3qL | 0,1,2,3,4,5,6 | 81967 | |
| PuKPg4mmafe | 0 | 81940 | |
| EDJbREhghzL | 0,1,3 | 64755 | |
| ur6pFq6Qu1A | 0,1 | 81967 | |
| B6ByNegPMKs | 0 | 81951 | |
| b8cTxDM8gDG | 0,1,2,8,11 | 73307 | |
| 17DRP5sb8fy | 0 | 81967 | |
| YmJkqBEsHnH | 0 | 80780 | |
| ULsKaCPVFJR | 0,1,2 | 81967 | |
| XcA2TqTSSAj | 0,2,3,5,6,8,9,11,12 | 60679 | |
| sKLMLpTHeUy | 0,1,2,4 | 79736 | |
| ZMojNkEp431 | 0,1,2 | 81967 | |
| e9zR4mvMWw7 | 0,1,2 | 80193 | |
| JeFG25nYj2p | 0,1 | 81967 | |
| uNb9QFRL6hY | 1,4,5,6 | 59613 | |
| p5wJjkQkbXX | 0,1,2,3 | 81967 | |
| Vvot9Ly1tCj | 0,3 | 78115 | |
| E9uDoFAP3SH | 0,1,5,6 | 81914 | |
| qoiz87JEwZ2 | 0,1,2,3 | 81967 | |
| VFuaQ6m2Qom | 0,1,2,4,5,6 | 81758 | |
| VLzqgDo317F | 0,1,2 | 81396 | |
| kEZ7cmS4wCh | 0,1,2,3,7 | 69135 | |
| 7y3sRwLe3Va | 0,1,2,5 | 81386 | |
| VVfe2KiqLaN | 0,1,2 | 81967 | |
| 2n8kARJN3HM | 0,1,2,4 | 81076 | |
| PX4nDJXEHrG | 0,1,2,3,4,5 | 79151 | |
| Uxmj2M2itWa | 0,1,3,4 | 49942 | |
| pRbA3pwrgk9 | 0,2,3,7,9,11 | 53295 | |
| cV4RVeZvu5T | 0,1,2,3 | 81038 | |
| sT4fr6TAbpF | 0 | 81625 | |
| GdvgFV5R1Z5 | 0 | 81967 | |
| JF19kD82Mey | 0,1,2 | 81927 | |
| JmbYfDe2QKZ | 0,1 | 81489 | |
| s8pcmisQ38h | 0,1,2 | 80428 | |
| 1pXnuDYAj8r | 0,1,2,5 | 81901 | |
| jh4fc5c5qoQ | 0,1,2 | 81967 | |
| vyrNrziPKCB | 0,1,3,4,7 | 81388 | |
| aayBHfsNo7d | 0,1,2 | 81693 | |
| rPc6DW4iMge | 0,1,3,4 | 80296 | |
| Total: 58 | 207 | 4581881 |
Matterport3D Validation
| Scan | Floors | # Episodes | |
| 2azQ1b91cZZ | 0,1 | 40 | |
| 8194nk5LbLH | 0 | 40 | |
| EU6Fwq7SyZv | 0 | 30 | |
| QUCTc6BB5sX | 1 | 20 | |
| TbHJrupSAjP | 0,1,2 | 30 | |
| Z6MFQCViBuw | 0 | 40 | |
| oLBMNvg9in8 | 0,1,2,3 | 50 | |
| x8F5xyUWy9e | 0,1 | 30 | |
| zsNo4HB9uLZ | 0 | 40 | |
| Total: 9 | 16 | 320 |
Matterport3D Test
| Scan | Floors | # Episodes | |
| 2t7WUuJeko7 | 0 | 50 | |
| 5ZKStnWn8Zo | 0,1 | 50 | |
| RPmz2sHmrrY | 0 | 50 | |
| UwV83HsGsw3 | 0,1,2,3 | 50 | |
| WYY7iVyf5p8 | 0,2 | 30 | |
| YFuZgdQ5vWj | 1 | 10 | |
| YVUC4YcDtcY | 0 | 50 | |
| fzynW3qQPVF | 0,1 | 50 | |
| jtcxE69GiFV | 0,1 | 40 | |
| pa4otMbVnkk | 0,1 | 50 | |
| q9vSo1VnCiC | 0 | 50 | |
| rqfALeAoiTq | 0,2 | 20 | |
| wc2JMjhGNzB | 0,1 | 50 | |
| yqstnuAEVhm | 0,1,2 | 60 | |
| Total: 14 | 26 | 610 |
Gibson Test
| Scan | Floors | # Episodes | |
| Wiconisco | 1,2 | 90 | |
| Corozal | 0,2,4 | 90 | |
| Collierville | 0,1,2 | 80 | |
| Markleeville | 0,1 | 90 | |
| Darden | 0,1,2 | 100 | |
| Total: 5 | 13 | 450 |
Contacts
If you have any doubts about our dataset, please use the public issues section on this github repo. Alternatively, drop us an e-mail at roberto.bigazzi [at] unimore.it.
Acknowledgment
This work was supported by the “European Training Network on PErsonalized Robotics as SErvice Oriented applications” (PERSEO) MSCA-ITN-2020 project (G.A. 955778).