# Data Format This document describes the database schema used in nuScenes. All annotations and meta data (including calibration, maps, vehicle coordinates etc.) are covered in a relational database. The database tables are listed below. Every row can be identified by its unique primary key `token`. Foreign keys such as `sample_token` may be used to link to the `token` of the table `sample`. Please refer to the [tutorial](https://www.nuscenes.org/nuimages#tutorial) for an introduction to the most important database tables. ![img](https://tutu.s3.cn-northwest-1.amazonaws.com.cn/open+dataset/md_images/357.svg) ### attribute An attribute is a property of an instance that can change while the category remains the same. Example: a vehicle being parked/stopped/moving, and whether or not a bicycle has a rider. ``` attribute { "token": -- Unique record identifier. "name": -- Attribute name. "description": -- Attribute description. } ``` ### calibrated_sensor Definition of a particular sensor (lidar/radar/camera) as calibrated on a particular vehicle. All extrinsic parameters are given with respect to the ego vehicle body frame. All camera images come undistorted and rectified. ``` calibrated_sensor { "token": -- Unique record identifier. "sensor_token": -- Foreign key pointing to the sensor type. "translation": [3] -- Coordinate system origin in meters: x, y, z. "rotation": [4] -- Coordinate system orientation as quaternion: w, x, y, z. "camera_intrinsic": [3, 3] -- Intrinsic camera calibration. Empty for sensors that are not cameras. } ``` ### category Taxonomy of object categories (e.g. vehicle, human). Subcategories are delineated by a period (e.g. `human.pedestrian.adult`). ``` category { "token": -- Unique record identifier. "name": -- Category name. Subcategories indicated by period. "description": -- Category description. "index": -- The index of the label used for efficiency reasons in the .bin label files of nuScenes-lidarseg. This field did not exist previously. } ``` ### ego_pose Ego vehicle pose at a particular timestamp. Given with respect to global coordinate system of the log's map. The ego_pose is the output of a lidar map-based localization algorithm described in our paper. The localization is 2-dimensional in the x-y plane. ``` ego_pose { "token": -- Unique record identifier. "translation": [3] -- Coordinate system origin in meters: x, y, z. Note that z is always 0. "rotation": [4] -- Coordinate system orientation as quaternion: w, x, y, z. "timestamp": -- Unix time stamp. } ``` ### instance An object instance, e.g. particular vehicle. This table is an enumeration of all object instances we observed. Note that instances are not tracked across scenes. ``` instance { "token": -- Unique record identifier. "category_token": -- Foreign key pointing to the object category. "nbr_annotations": -- Number of annotations of this instance. "first_annotation_token": -- Foreign key. Points to the first annotation of this instance. "last_annotation_token": -- Foreign key. Points to the last annotation of this instance. } ``` ### lidarseg Mapping between nuScenes-lidarseg annotations and sample_datas corresponding to the lidar pointcloud associated with a keyframe. ``` lidarseg { "token": -- Unique record identifier. "filename": -- The name of the .bin files containing the nuScenes-lidarseg labels. These are numpy arrays of uint8 stored in binary format using numpy. "sample_data_token": -- Foreign key. Sample_data corresponding to the annotated lidar pointcloud with is_key_frame=True. } ``` ### log Information about the log from which the data was extracted. ``` log { "token": -- Unique record identifier. "logfile": -- Log file name. "vehicle": -- Vehicle name. "date_captured": -- Date (YYYY-MM-DD). "location": -- Area where log was captured, e.g. singapore-onenorth. } ``` ### map Map data that is stored as binary semantic masks from a top-down view. ``` map { "token": -- Unique record identifier. "log_tokens": [n] -- Foreign keys. "category": -- Map category, currently only semantic_prior for drivable surface and sidewalk. "filename": -- Relative path to the file with the map mask. } ``` ### sample A sample is an annotated keyframe at 2 Hz. The data is collected at (approximately) the same timestamp as part of a single LIDAR sweep. ``` sample { "token": -- Unique record identifier. "timestamp": -- Unix time stamp. "scene_token": -- Foreign key pointing to the scene. "next": -- Foreign key. Sample that follows this in time. Empty if end of scene. "prev": -- Foreign key. Sample that precedes this in time. Empty if start of scene. } ``` ### sample_annotation A bounding box defining the position of an object seen in a sample. All location data is given with respect to the global coordinate system. ``` sample_annotation { "token": -- Unique record identifier. "sample_token": -- Foreign key. NOTE: this points to a sample NOT a sample_data since annotations are done on the sample level taking all relevant sample_data into account. "instance_token": -- Foreign key. Which object instance is this annotating. An instance can have multiple annotations over time. "attribute_tokens": [n] -- Foreign keys. List of attributes for this annotation. Attributes can change over time, so they belong here, not in the instance table. "visibility_token": -- Foreign key. Visibility may also change over time. If no visibility is annotated, the token is an empty string. "translation": [3] -- Bounding box location in meters as center_x, center_y, center_z. "size": [3] -- Bounding box size in meters as width, length, height. "rotation": [4] -- Bounding box orientation as quaternion: w, x, y, z. "num_lidar_pts": -- Number of lidar points in this box. Points are counted during the lidar sweep identified with this sample. "num_radar_pts": -- Number of radar points in this box. Points are counted during the radar sweep identified with this sample. This number is summed across all radar sensors without any invalid point filtering. "next": -- Foreign key. Sample annotation from the same object instance that follows this in time. Empty if this is the last annotation for this object. "prev": -- Foreign key. Sample annotation from the same object instance that precedes this in time. Empty if this is the first annotation for this object. } ``` ### sample_data A sensor data e.g. image, point cloud or radar return. For sample_data with is_key_frame=True, the time-stamps should be very close to the sample it points to. For non key-frames the sample_data points to the sample that follows closest in time. ``` sample_data { "token": -- Unique record identifier. "sample_token": -- Foreign key. Sample to which this sample_data is associated. "ego_pose_token": -- Foreign key. "calibrated_sensor_token": -- Foreign key. "filename": -- Relative path to data-blob on disk. "fileformat": -- Data file format. "width": -- If the sample data is an image, this is the image width in pixels. "height": -- If the sample data is an image, this is the image height in pixels. "timestamp": -- Unix time stamp. "is_key_frame": -- True if sample_data is part of key_frame, else False. "next": -- Foreign key. Sample data from the same sensor that follows this in time. Empty if end of scene. "prev": -- Foreign key. Sample data from the same sensor that precedes this in time. Empty if start of scene. } ``` ### scene A scene is a 20s long sequence of consecutive frames extracted from a log. Multiple scenes can come from the same log. Note that object identities (instance tokens) are not preserved across scenes. ``` scene { "token": -- Unique record identifier. "name": -- Short string identifier. "description": -- Longer description of the scene. "log_token": -- Foreign key. Points to log from where the data was extracted. "nbr_samples": -- Number of samples in this scene. "first_sample_token": -- Foreign key. Points to the first sample in scene. "last_sample_token": -- Foreign key. Points to the last sample in scene. } ``` ### sensor A specific sensor type. ``` sensor { "token": -- Unique record identifier. "channel": -- Sensor channel name. "modality": {camera, lidar, radar} -- Sensor modality. Supports category(ies) in brackets. } ``` ### visibility The visibility of an instance is the fraction of annotation visible in all 6 images. Binned into 4 bins 0-40%, 40-60%, 60-80% and 80-100%. ``` visibility { "token": -- Unique record identifier. "level": -- Visibility level. "description": -- Description of visibility level. } ```