feat(ui): Browser Based UI for Control Robot

[jackvial]

What this does

https://www.youtube.com/watch?v=mSybeh16ma4

Adds a browser based UI for recording datasets, and teleoperation.

How It Works

LeRobot -> Browser UI Commutation

• control_robot.py uses ZeroMQ to publish observations (images and robot state) as JSON to TCP socket on port 5555
• browser_ui_server.py uses ZeroMQ to listen for messages on TCP socket port 5555
• browser_ui_server.py sends observations to the browser using socketio
• browser_ui.html listens for the socketio events and renders the observations

Browser UI -> LeRobot Communication

• browser_ui.html listens for keyboard events or button events and sends those events to browser_ui_server.py
• browser_ui_server.py uses ZeroMQ to send the keyboard events to TCP socket port 5556
• control_robot.py listens for keyboard events on TCP socket port 5556

Why

• Improves cross platform compatibility by moving display rendering and keyboard input handling to the browser.
  • On Ubuntu 24.04 cv.imshow in control_loop kept crashing
  • When accessing ubuntu over SSH from macbook keyboard input doesn't work because pynput in is_headless will always throw
  • When accessing ubuntu over SSH from macbook there is no way to view the camera display.
  • SSH'ing into a linux box with a GPU running on your local network is a pretty common setup so would be great to have better support for this.
• Easily supports UI in headless environments e.g. SSH in from macbook to linux machine with CUDA GPU (as shown in the video). The UI server port can be forwarded over SSH (VSCode remote does this automatically) or you can access the UI by going to the remote machines private IP on your local network
• Encapsulates everything UI related in ControlContext class. This makes it simpler to get visual feedback like if rewards are being assigned.
• UX improvements: Clearly display which phase of recording you are in, which episode you are on, and how much time is remaining in the episode. Easily support adding additional visual feedback e.g. manual reward assignment for training reward classifier for RL based methods like HIL-SERL and TDMPC

How To Test

• Run pip install -e ."[koch, test]" to install the new dependencies flask-socketio and pyzmq
• Start recording or teleoperating e.g. (you do not need to pass display cameras or play sounds)

python lerobot/scripts/control_robot.py record \
    --fps 30 \
    --repo-id $USER/koch_pick_place_lego \
    --num-episodes 50 \
    --warmup-time-s 20 \
    --episode-time-s 30 \
    --reset-time-s 10

• Start the UI server (starts on port 8000) by default
  python lerobot/scripts/browser_ui_server.py
• Got to http://localhost:8000 in your browser
• If accessing a remote headless machine over SSH you can forward port 8000 to the client machine and go to http://localhost:8000 on that machine. If your remote machine is on your local network don't need to forward the port and can got to the UI at http://{remote-machine-private-ip}:8000

For VSCode Users

Add this to your .vscode/launch.json. You might need to change the change the venv path.

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Record",
            "type": "debugpy",
            "request": "launch",
            "program": "${workspaceFolder}/lerobot/scripts/control_robot.py",
            "args": [
                "record",
                "--robot-path", "lerobot/configs/robot/koch_jack.yaml",
                "--fps", "30",
                "--root", "outputs/browser_ui_test_9",
                "--repo-id", "jackvial/browser_ui_test_9",
                "--warmup-time-s", "10",
                "--episode-time-s", "15",
                "--reset-time-s", "5",
                "--num-episodes", "20",
                "--push-to-hub", "0",
                "--single-task", "browser_ui_test_9"
            ],
            "console": "integratedTerminal",
            "justMyCode": true,
            "python": "${workspaceFolder}/lerobot_venv/bin/python3.12",
            "cwd": "${workspaceFolder}",
            "env": {
                "PYTHONPATH": "${workspaceFolder}"
            }
        },
        {
            "name": "Start UI Server",
            "type": "debugpy",
            "request": "launch",
            "program": "${workspaceFolder}/lerobot/scripts/browser_ui_server.py",
            "console": "integratedTerminal",
            "justMyCode": true,
            "python": "${workspaceFolder}/lerobot_venv/bin/python3.12",
            "cwd": "${workspaceFolder}",
            "env": {
                "PYTHONPATH": "${workspaceFolder}"
            }
        }
   ]
}

My Koch Robot Config

lerobot/configs/robot/koch_jack.yaml

_target_: lerobot.common.robot_devices.robots.manipulator.ManipulatorRobot
robot_type: koch
calibration_dir: .cache/calibration/koch_tdmpc_jack
leader_arms:
  main:
    _target_: lerobot.common.robot_devices.motors.dynamixel.DynamixelMotorsBus
    port: /dev/servo_585A007782
    motors:
      # name: (index, model)
      shoulder_pan: [1, "xl330-m077"]
      shoulder_lift: [2, "xl330-m077"]
      elbow_flex: [3, "xl330-m077"]
      wrist_flex: [4, "xl330-m077"]
      wrist_roll: [5, "xl330-m077"]
      gripper: [6, "xl330-m077"]
follower_arms:
  main:
    _target_: lerobot.common.robot_devices.motors.dynamixel.DynamixelMotorsBus
    port: /dev/servo_5837053138
    motors:
      # name: (index, model)
      shoulder_pan: [1, "xl430-w250"]
      shoulder_lift: [2, "xl430-w250"]
      elbow_flex: [3, "xl330-m288"]
      wrist_flex: [4, "xl330-m288"]
      wrist_roll: [5, "xl330-m288"]
      gripper: [6, "xl330-m288"]
cameras:
  main:
    _target_: lerobot.common.robot_devices.cameras.opencv.OpenCVCamera
    camera_index: 0
    fps: 15
    width: 800
    height: 600
  top:
    _target_: lerobot.common.robot_devices.cameras.opencv.OpenCVCamera
    camera_index: 4
    fps: 15
    width: 800
    height: 600
gripper_open_degree: 35.156

[helper2424]

I didn't test, but it's cool. I would help you with it when we finish with Hil-SERL

[jackvial]

I didn't test, but it's cool. I would help you with it when we finish with Hil-SERL

That sounds great, thank you 🙏