Sentinel

Using your data

Read Sentinel MCAP recordings, message schemas, and episode metadata.

Sentinel records each teleoperation episode as a ROS 2 bag using MCAP storage. A recording can contain per-tick training records, robot state, commands, XR input, encoded camera frames, task context, labels, and system events.

The exact topic set comes from the robot's data-collection configuration. Inspect the bag instead of assuming every setup records the same topics.

ros2 bag info /datasets/session_20260718_143052/episode_001.finalized

Directory structure

On a daemon-managed install, recordings live on the robot host at:

/var/lib/sentinel/datasets/    # mounted into the runtime container at /datasets

Legacy launcher setups use the configured dataset_recording.output_dir instead.

Episodes are grouped into timestamped recording sessions, under a task directory when a task is configured:

/var/lib/sentinel/datasets/
└── pick-place/                        # task prefix — only when tasks are configured
    └── session_20260718_143052/
        ├── episode_001.finalized/
        │   ├── episode_001_0.mcap
        │   ├── metadata.yaml
        │   ├── manifest.json          # written by the daemon: identity + per-file sha256
        │   └── .reported              # marker: the platform acknowledged this episode
        └── episode_002.finalized/
            ├── episode_002_0.mcap
            ├── episode_002_1.mcap
            └── metadata.yaml
  • Session names use session_YYYYMMDD_HHMMSS in the robot's local time; timestamps inside records are UTC.
  • Episode numbers are zero-padded within a session.
  • An active recording uses the .in_progress suffix.
  • A completed recording normally uses the .finalized suffix.
  • manifest.json and .reported appear after finalization; they are additive — metadata.yaml remains the ground truth.
  • MCAP files split when they reach the configured maximum size. The common default is 1 GiB.

On daemon-managed installs, finalized episodes sync to cloud storage automatically. Reading this directory directly is still supported — the daemon uploads the same files unchanged. Don't move them by hand; the platform will keep listing the episode as unsynced.

Failed-episode handling is configurable. A failed episode can be finalized, deleted, or left in progress for manual review. A discarded episode is deleted.

Common recorded topics

Topic names that contain {namespace} or {camera_name} vary by configuration.

DataTypical topicMessage type
Action samples — one aligned training record per control tick/{namespace}/sentinel/control/commands/action_samplesentinel_msgs/msg/ActionSample
Executed samples — what actually ran, post-safety/{namespace}/sentinel/control/commands/executed_samplesentinel_msgs/msg/ExecutedSample
Policy provenance and native actions/{namespace}/sentinel/policy/action_debugsentinel_msgs/msg/PolicyActionDebug
Safe joint commands/{namespace}/sentinel/control/commands/joint_safetrajectory_msgs/msg/JointTrajectory
Measured joints/{namespace}/sentinel/robot/state/joint_states/measuredsensor_msgs/msg/JointState
Gripper state/{namespace}/sentinel/robot/state/gripper_states/measuredsentinel_msgs/msg/GripperState
Gripper commands/{namespace}/sentinel/control/commands/grippersentinel_msgs/msg/GripperCommand
Encoded camera frames/sentinel/vision/output/{camera_name}/encodedsentinel_msgs/msg/EncodedImage
XR tracking and controls/sentinel/xr/input/xr_inputsentinel_msgs/msg/XRInput
System state/{namespace}/sentinel/system/state/currentsentinel_msgs/msg/SystemState
Task and operator context/sentinel/agent/session_contextsentinel_msgs/msg/SessionContext
Episode and system events/sentinel/eventssentinel_msgs/msg/Event

Control-command topics gain a _{capability_id} suffix when a namespace exposes more than one capability. Sentinel always records the session-context and event topics; they preserve task, operator, label, and event changes inside the MCAP timeline.

Message definitions

All messages use ROS 2 CDR serialization. Standard robot state and command topics use standard ROS 2 message packages. Sentinel-specific topics use sentinel_msgs.

Decode recorded video

The following ROS 2 node decodes sentinel_msgs/msg/EncodedImage with PyAV and republishes each decoded frame as sensor_msgs/msg/CompressedImage.

Install the Python dependencies in the ROS 2 environment that contains sentinel_msgs:

python3 -m pip install av Pillow

#!/usr/bin/env python3

from io import BytesIO

import av
import rclpy
from rclpy.node import Node
from rclpy.qos import (
    DurabilityPolicy,
    HistoryPolicy,
    QoSProfile,
    ReliabilityPolicy,
)
from sensor_msgs.msg import CompressedImage
from sentinel_msgs.msg import EncodedImage


CODECS = {
    "h264": "h264",
    "h265": "hevc",
    "vp8": "vp8",
    "vp9": "vp9",
    "av1": "av1",
}


class EncodedImageDecoder(Node):
    def __init__(self) -> None:
        super().__init__("encoded_image_decoder")

        self.declare_parameter(
            "input_topic", "/sentinel/vision/output/right_wrist/encoded"
        )
        self.declare_parameter(
            "output_topic", "/sentinel/vision/output/right_wrist/decoded/compressed"
        )

        input_topic = self.get_parameter("input_topic").value
        output_topic = self.get_parameter("output_topic").value

        video_qos = QoSProfile(
            reliability=ReliabilityPolicy.BEST_EFFORT,
            durability=DurabilityPolicy.VOLATILE,
            history=HistoryPolicy.KEEP_LAST,
            depth=5,
        )

        self.publisher = self.create_publisher(
            CompressedImage, output_topic, video_qos
        )
        self.subscription = self.create_subscription(
            EncodedImage, input_topic, self.decode, video_qos
        )

        self.decoder = None
        self.stream_key = None
        self.frames_decoded = 0

        self.get_logger().info(f"Decoding {input_topic}")
        self.get_logger().info(f"Publishing {output_topic}")

    def decode(self, msg: EncodedImage) -> None:
        codec_name = CODECS.get(msg.encoding.lower())
        if codec_name is None:
            self.get_logger().error(f"Unsupported encoding: {msg.encoding}")
            return

        stream_key = (codec_name, msg.width, msg.height)
        if stream_key != self.stream_key:
            self.decoder = av.CodecContext.create(codec_name, "r")
            self.stream_key = stream_key
            self.get_logger().info(
                f"Initialized {codec_name} decoder for {msg.width}x{msg.height}"
            )

        packet = av.Packet(bytes(msg.data))
        packet.pts = msg.pts_ns
        packet.dts = msg.dts_ns if msg.dts_ns else msg.pts_ns

        try:
            frames = self.decoder.decode(packet)
        except Exception as error:
            # Playback may begin before an IDR frame. Decoding recovers at the
            # next keyframe.
            self.get_logger().debug(f"Waiting for a decodable frame: {error}")
            return

        for frame in frames:
            jpeg = BytesIO()
            frame.to_image().save(jpeg, format="JPEG", quality=90)

            output = CompressedImage()
            output.header = msg.header
            output.format = "jpeg"
            output.data = jpeg.getvalue()
            self.publisher.publish(output)

            self.frames_decoded += 1
            if self.frames_decoded == 1:
                self.get_logger().info("Published the first decoded frame")


def main() -> None:
    rclpy.init()
    node = EncodedImageDecoder()
    try:
        rclpy.spin(node)
    except KeyboardInterrupt:
        pass
    finally:
        node.destroy_node()
        rclpy.shutdown()


if __name__ == "__main__":
    main()

Start the decoder with the encoded topic from the bag:

python3 encoded_image_decoder.py --ros-args \
  -p input_topic:=/sentinel/vision/output/right_wrist/encoded \
  -p output_topic:=/sentinel/vision/output/right_wrist/decoded/compressed

Play the episode in another terminal:

ros2 bag play /datasets/session_20260718_001151/episode_003.finalized

View /sentinel/vision/output/right_wrist/decoded/compressed with RViz, rqt_image_view, or another sensor_msgs/msg/CompressedImage subscriber.

A decoder needs an IDR frame before it can reconstruct dependent frames. If playback starts between keyframes, the first few packets may not produce an image. Decoding begins when the next IDR frame arrives.

Episode metadata

All episode summary metadata is stored in the bag's standard metadata.yaml file. ROS bag information and Sentinel attribution are both children of rosbag2_bagfile_information. Time-varying task context and events remain as messages inside the MCAP timeline.

rosbag2_bagfile_information:
  version: 5
  storage_identifier: mcap
  duration:
    nanoseconds: 18484502184
  starting_time:
    nanoseconds_since_epoch: 1784333771770314226
  message_count: 15583
  topics_with_message_count:
    - topic_metadata:
        name: /sentinel/events
        type: sentinel_msgs/msg/Event
        serialization_format: cdr
      message_count: 33
    - topic_metadata:
        name: /sentinel/vision/output/overhead/encoded
        type: sentinel_msgs/msg/EncodedImage
        serialization_format: cdr
      message_count: 541
  compression_format: ""
  compression_mode: ""
  relative_file_paths:
    - episode_003_0.mcap
  files:
    - path: episode_003_0.mcap
      starting_time:
        nanoseconds_since_epoch: 1784333771770314226
      duration:
        nanoseconds: 18484502184
      message_count: 15583
  custom_data:
    recording_session_id: session_20260718_001151
    episode_seq: 3
    started_at: 2026-07-18T00:16:11Z
    ended_at: 2026-07-18T00:16:30Z
    outcome: success

Each entry in topics_with_message_count also includes the recorded publisher QoS profile. The shortened example above shows two topics; the real list contains every topic present in the bag.

Standard rosbag fields

FieldMeaning
duration.nanosecondsTotal recorded duration
starting_time.nanoseconds_since_epochBag start time as nanoseconds since the Unix epoch
message_countTotal number of messages across all recorded topics
topics_with_message_countTopic names, ROS types, serialization formats, QoS profiles, and per-topic message counts
relative_file_pathsMCAP files that belong to the episode
filesPer-file start time, duration, and message count

Sentinel fields in custom_data

FieldMeaning
recording_session_idRecording session that contains the episode
episode_seqEpisode number within that session
started_atUTC episode start time in ISO 8601 format
ended_atUTC episode end time in ISO 8601 format
outcomeEpisode outcome, such as success or failure
task_idSelected task identifier, when present
task_versionVersion of the selected task
promptTask prompt captured at episode start
operator_idAuthenticated operator attributed to the episode, when present

Task and operator fields are optional. They appear in custom_data when an episode starts with an active task or authenticated operator context.

Preserve alignment when converting

  • Use source message timestamps rather than file order alone.
  • Keep recording_session_id and episode_seq with every converted episode.
  • Preserve task context and structured events instead of reconstructing labels from video.
  • Use sequence_number to detect dropped encoded video frames.
  • Ignore unknown metadata fields so readers remain compatible with additive schema changes.

Do not treat an .in_progress directory as a completed episode. It may still be recording or may have been retained after a failed finalization.

Next step

Review upcoming workflows

See which autonomy, fleet, and API workflows are still in development.