Using your data
Read Sentinel MCAP recordings, message schemas, and episode metadata.
Sentinel records each teleoperation episode as a ROS 2 bag using MCAP storage. A recording can contain per-tick training records, robot state, commands, XR input, encoded camera frames, task context, labels, and system events.
The exact topic set comes from the robot's data-collection configuration. Inspect the bag instead of assuming every setup records the same topics.
ros2 bag info /datasets/session_20260718_143052/episode_001.finalizedDirectory structure
On a daemon-managed install, recordings live on the robot host at:
/var/lib/sentinel/datasets/ # mounted into the runtime container at /datasetsLegacy launcher setups use the configured dataset_recording.output_dir instead.
Episodes are grouped into timestamped recording sessions, under a task directory when a task is configured:
/var/lib/sentinel/datasets/
└── pick-place/ # task prefix — only when tasks are configured
└── session_20260718_143052/
├── episode_001.finalized/
│ ├── episode_001_0.mcap
│ ├── metadata.yaml
│ ├── manifest.json # written by the daemon: identity + per-file sha256
│ └── .reported # marker: the platform acknowledged this episode
└── episode_002.finalized/
├── episode_002_0.mcap
├── episode_002_1.mcap
└── metadata.yaml- Session names use
session_YYYYMMDD_HHMMSSin the robot's local time; timestamps inside records are UTC. - Episode numbers are zero-padded within a session.
- An active recording uses the
.in_progresssuffix. - A completed recording normally uses the
.finalizedsuffix. manifest.jsonand.reportedappear after finalization; they are additive —metadata.yamlremains the ground truth.- MCAP files split when they reach the configured maximum size. The common default is 1 GiB.
On daemon-managed installs, finalized episodes sync to cloud storage automatically. Reading this directory directly is still supported — the daemon uploads the same files unchanged. Don't move them by hand; the platform will keep listing the episode as unsynced.
Failed-episode handling is configurable. A failed episode can be finalized, deleted, or left in progress for manual review. A discarded episode is deleted.
Common recorded topics
Topic names that contain {namespace} or {camera_name} vary by configuration.
| Data | Typical topic | Message type |
|---|---|---|
| Action samples — one aligned training record per control tick | /{namespace}/sentinel/control/commands/action_sample | sentinel_msgs/msg/ActionSample |
| Executed samples — what actually ran, post-safety | /{namespace}/sentinel/control/commands/executed_sample | sentinel_msgs/msg/ExecutedSample |
| Policy provenance and native actions | /{namespace}/sentinel/policy/action_debug | sentinel_msgs/msg/PolicyActionDebug |
| Safe joint commands | /{namespace}/sentinel/control/commands/joint_safe | trajectory_msgs/msg/JointTrajectory |
| Measured joints | /{namespace}/sentinel/robot/state/joint_states/measured | sensor_msgs/msg/JointState |
| Gripper state | /{namespace}/sentinel/robot/state/gripper_states/measured | sentinel_msgs/msg/GripperState |
| Gripper commands | /{namespace}/sentinel/control/commands/gripper | sentinel_msgs/msg/GripperCommand |
| Encoded camera frames | /sentinel/vision/output/{camera_name}/encoded | sentinel_msgs/msg/EncodedImage |
| XR tracking and controls | /sentinel/xr/input/xr_input | sentinel_msgs/msg/XRInput |
| System state | /{namespace}/sentinel/system/state/current | sentinel_msgs/msg/SystemState |
| Task and operator context | /sentinel/agent/session_context | sentinel_msgs/msg/SessionContext |
| Episode and system events | /sentinel/events | sentinel_msgs/msg/Event |
Control-command topics gain a _{capability_id} suffix when a namespace exposes more than one capability. Sentinel always records the session-context and event topics; they preserve task, operator, label, and event changes inside the MCAP timeline.
Message definitions
All messages use ROS 2 CDR serialization. Standard robot state and command topics use standard ROS 2 message packages. Sentinel-specific topics use sentinel_msgs.
Decode recorded video
The following ROS 2 node decodes sentinel_msgs/msg/EncodedImage with PyAV and republishes each decoded frame as sensor_msgs/msg/CompressedImage.
Install the Python dependencies in the ROS 2 environment that contains sentinel_msgs:
python3 -m pip install av Pillow#!/usr/bin/env python3
from io import BytesIO
import av
import rclpy
from rclpy.node import Node
from rclpy.qos import (
DurabilityPolicy,
HistoryPolicy,
QoSProfile,
ReliabilityPolicy,
)
from sensor_msgs.msg import CompressedImage
from sentinel_msgs.msg import EncodedImage
CODECS = {
"h264": "h264",
"h265": "hevc",
"vp8": "vp8",
"vp9": "vp9",
"av1": "av1",
}
class EncodedImageDecoder(Node):
def __init__(self) -> None:
super().__init__("encoded_image_decoder")
self.declare_parameter(
"input_topic", "/sentinel/vision/output/right_wrist/encoded"
)
self.declare_parameter(
"output_topic", "/sentinel/vision/output/right_wrist/decoded/compressed"
)
input_topic = self.get_parameter("input_topic").value
output_topic = self.get_parameter("output_topic").value
video_qos = QoSProfile(
reliability=ReliabilityPolicy.BEST_EFFORT,
durability=DurabilityPolicy.VOLATILE,
history=HistoryPolicy.KEEP_LAST,
depth=5,
)
self.publisher = self.create_publisher(
CompressedImage, output_topic, video_qos
)
self.subscription = self.create_subscription(
EncodedImage, input_topic, self.decode, video_qos
)
self.decoder = None
self.stream_key = None
self.frames_decoded = 0
self.get_logger().info(f"Decoding {input_topic}")
self.get_logger().info(f"Publishing {output_topic}")
def decode(self, msg: EncodedImage) -> None:
codec_name = CODECS.get(msg.encoding.lower())
if codec_name is None:
self.get_logger().error(f"Unsupported encoding: {msg.encoding}")
return
stream_key = (codec_name, msg.width, msg.height)
if stream_key != self.stream_key:
self.decoder = av.CodecContext.create(codec_name, "r")
self.stream_key = stream_key
self.get_logger().info(
f"Initialized {codec_name} decoder for {msg.width}x{msg.height}"
)
packet = av.Packet(bytes(msg.data))
packet.pts = msg.pts_ns
packet.dts = msg.dts_ns if msg.dts_ns else msg.pts_ns
try:
frames = self.decoder.decode(packet)
except Exception as error:
# Playback may begin before an IDR frame. Decoding recovers at the
# next keyframe.
self.get_logger().debug(f"Waiting for a decodable frame: {error}")
return
for frame in frames:
jpeg = BytesIO()
frame.to_image().save(jpeg, format="JPEG", quality=90)
output = CompressedImage()
output.header = msg.header
output.format = "jpeg"
output.data = jpeg.getvalue()
self.publisher.publish(output)
self.frames_decoded += 1
if self.frames_decoded == 1:
self.get_logger().info("Published the first decoded frame")
def main() -> None:
rclpy.init()
node = EncodedImageDecoder()
try:
rclpy.spin(node)
except KeyboardInterrupt:
pass
finally:
node.destroy_node()
rclpy.shutdown()
if __name__ == "__main__":
main()Start the decoder with the encoded topic from the bag:
python3 encoded_image_decoder.py --ros-args \
-p input_topic:=/sentinel/vision/output/right_wrist/encoded \
-p output_topic:=/sentinel/vision/output/right_wrist/decoded/compressedPlay the episode in another terminal:
ros2 bag play /datasets/session_20260718_001151/episode_003.finalizedView /sentinel/vision/output/right_wrist/decoded/compressed with RViz, rqt_image_view, or another sensor_msgs/msg/CompressedImage subscriber.
A decoder needs an IDR frame before it can reconstruct dependent frames. If playback starts between keyframes, the first few packets may not produce an image. Decoding begins when the next IDR frame arrives.
Episode metadata
All episode summary metadata is stored in the bag's standard metadata.yaml file. ROS bag information and Sentinel attribution are both children of rosbag2_bagfile_information. Time-varying task context and events remain as messages inside the MCAP timeline.
rosbag2_bagfile_information:
version: 5
storage_identifier: mcap
duration:
nanoseconds: 18484502184
starting_time:
nanoseconds_since_epoch: 1784333771770314226
message_count: 15583
topics_with_message_count:
- topic_metadata:
name: /sentinel/events
type: sentinel_msgs/msg/Event
serialization_format: cdr
message_count: 33
- topic_metadata:
name: /sentinel/vision/output/overhead/encoded
type: sentinel_msgs/msg/EncodedImage
serialization_format: cdr
message_count: 541
compression_format: ""
compression_mode: ""
relative_file_paths:
- episode_003_0.mcap
files:
- path: episode_003_0.mcap
starting_time:
nanoseconds_since_epoch: 1784333771770314226
duration:
nanoseconds: 18484502184
message_count: 15583
custom_data:
recording_session_id: session_20260718_001151
episode_seq: 3
started_at: 2026-07-18T00:16:11Z
ended_at: 2026-07-18T00:16:30Z
outcome: successEach entry in topics_with_message_count also includes the recorded publisher QoS profile. The shortened example above shows two topics; the real list contains every topic present in the bag.
Standard rosbag fields
| Field | Meaning |
|---|---|
duration.nanoseconds | Total recorded duration |
starting_time.nanoseconds_since_epoch | Bag start time as nanoseconds since the Unix epoch |
message_count | Total number of messages across all recorded topics |
topics_with_message_count | Topic names, ROS types, serialization formats, QoS profiles, and per-topic message counts |
relative_file_paths | MCAP files that belong to the episode |
files | Per-file start time, duration, and message count |
Sentinel fields in custom_data
| Field | Meaning |
|---|---|
recording_session_id | Recording session that contains the episode |
episode_seq | Episode number within that session |
started_at | UTC episode start time in ISO 8601 format |
ended_at | UTC episode end time in ISO 8601 format |
outcome | Episode outcome, such as success or failure |
task_id | Selected task identifier, when present |
task_version | Version of the selected task |
prompt | Task prompt captured at episode start |
operator_id | Authenticated operator attributed to the episode, when present |
Task and operator fields are optional. They appear in custom_data when an episode starts with an active task or authenticated operator context.
Preserve alignment when converting
- Use source message timestamps rather than file order alone.
- Keep
recording_session_idandepisode_seqwith every converted episode. - Preserve task context and structured events instead of reconstructing labels from video.
- Use
sequence_numberto detect dropped encoded video frames. - Ignore unknown metadata fields so readers remain compatible with additive schema changes.
Do not treat an .in_progress directory as a completed episode. It may still be recording or may have been retained after a failed finalization.
Next step
Review upcoming workflows
See which autonomy, fleet, and API workflows are still in development.