Synthetic

We collected data using ROS and Gazebo for simulating a synthetic environment in which the robot performs random pick-n-place motions. We record 2 runs with different initialization seeds. Each run contains 20 recording sequences composed of 10 pick-n-place motions split equally between left and right robot arm. Each sequence is recorded by 3 anchor cameras (center, left, right) which are randomly moved within a sphere of 1m diameter.

Real

We collected data using ROS and a time-of-flight Microsoft Kinect One (second version) sensor capturing the moving robot from 3 camera positions (center, left, right). We record 20 pick-n-place sequences from each camera location and split them equally between right and left arm.

Synthetic

The synthetic part contains 10FPS videos with RGB (1980x1080) and depth (512x424) frames. In total there are 40 sequences with over 350k annotated frames.

Real

The real part contains 15FPS videos with RGB (1980x1080) and depth (512x424) frames. In total there are 20 sequences with over 20k annotated frames.

Synthetic

0: "head",                         8:  "left upper shoulder",
1: "right upper shoulder",         9:  "left lower shoulder",
2: "right lower shoulder",         10: "left upper elbow",
3: "right upper elbow",            11: "left lower elbow",
4: "right lower elbow",            12: "left upper forearm",
5: "right upper forearm",          13: "left lower forearm",
6: "right lower forearm",          14: "left wrist",
7: "right wrist",                  15: "base"

The synthetic annotations contain information about the 3D robot joints positions, the pick-n-place locations and the camera positions. They are stored in JSON files with the structure described below:

{
    "joints": {
        "world": {                 # coordinates w.r.t. world origin
            "head": {
                "translation.z": float,
                "translation.x": float,
                "translation.y": float,
                "rotation.w": float,
                "rotation.z": float,
                "rotation.x": float,
                "rotation.y": float
            }
            ⋮
            "base": {
                "translation.x": float,
                "translation.y": float,
                "translation.z": float,
                "rotation.w": float,
                "rotation.z": float,
                "rotation.x": float,
                "rotation.y": float
            }
        }
        "base": {                  # coordinates w.r.t. robot base
            ⋮
        }
        "kinect_v2_c": {           # coordinates w.r.t. right camera
            ⋮
        }
        "kinect_v2_b": {           # coordinates w.r.t. left camera
            ⋮
        }
        "kinect_v2_a": {           # coordinates w.r.t. center camera
            ⋮
        }
    }
}

{
    "place": {
        "position": {
            "x": float,
            "y": float,
            "z": float
            }
        "orientation": {
            "x": float,
            "y": float,
            "z": float,
            "w": float
            }
    }
    "pick": {
        "position": {
            "x": float,
            "y": float,
            "z": float
            }
        "orientation": {
            "x": float,
            "y": float,
            "z": float,
            "w": float
            }
    }
}

{
    "position": {
        "x": float,
        "y": float,
        "z": float
        }
    "orientation": {
        "x": float,
        "y": float,
        "z": float,
        "w": float
        }
}

{
    "rgb": {
        "fx": float,
        "fy": float,
        "cx": float,
        "cy": float
        }
    "depth": {
        "fx": float,
        "fy": float,
        "cx": float,
        "cy": float
        }
    "tx_depth_to_rgb": float
}

Real

0: "head",                         8:  "left upper shoulder",
1: "right upper shoulder",         9:  "left lower shoulder",
2: "right lower shoulder",         10: "left upper elbow",
3: "right upper elbow",            11: "left lower elbow",
4: "right lower elbow",            12: "left upper forearm",
5: "right upper forearm",          13: "left lower forearm",
6: "right lower forearm",          14: "left wrist",
7: "right wrist",                  15: "base"

The real annotations contain information about the 3D robot joints positions and camera positions. The joints are saved as TXT files (one file per joint) while the camera info in a JSON file with the structure described below:

# coordinates w.r.t. robot base
timestamp, tx, ty, tz, rx, ry, rz, rw

{
    "rgb": {
        "fx": float,
        "fy": float,
        "cx": float,
        "cy": float
        }
    "depth": {
        "fx": float,
        "fy": float,
        "cx": float,
        "cy": float
        }
    "tx_depth_to_rgb": float
    "cam_center": {
        "tr_camera_to_base": [float, float, float],
        "tilt_angle": float,
        "offset": float
    }
    "cam_left": {
        ⋮
    }
    "cam_right": {
        ⋮
    }
}

Synthetic

The synthetic data are split into 28 train sequences, 4 validation sequences and 8 test sequences. Each sequences is sampled every 10 frames.

Real

The real sequences represent a single test split. Each sequence is sampled every 5 frames obtaining a test of 4032 images.