Lesson 3: Teaching BonicBot to Track Body Joints

Learning Objective

Teach BonicBot to detect a person’s body pose, visualize all 33 body landmarks, and retrieve a specific joint from the landmark list.

Introduction

BonicBot can now recognize faces, but what if it needs to understand how a person is moving?

Instead of detecting an entire face or object, BonicBot can estimate the position of important body joints. This process is called pose estimation.

When someone stands in front of BonicBot, its pose detection model predicts 33 body landmarks representing major joints and body parts, including:

nose
shoulders
elbows
wrists
hips
knees
ankles

Together, these landmarks form a digital skeleton that describes the person’s posture.

In this lesson, BonicBot will:

detect a person’s pose in real time
draw the complete body skeleton
highlight one specific joint (the right wrist)
display a live status panel showing whether a pose is currently being tracked

You’ll also learn an important programming concept: every landmark has a fixed position in the list. Because the landmarks always appear in the same order, your program can directly access any body joint using its index.

This simple idea becomes the foundation for future lessons, where BonicBot will recognize gestures, movements, and human activities.

Setup: Installing Packages

Before running the code, make sure your computer has the required Python packages installed. Open a terminal and run:


pip install bonicbot-bridge opencv-python numpy

What each package does

Package	Purpose
`bonicbot-bridge`	The official BonicBot SDK. Provides the `BonicBot` class used to connect to the robot, control its camera, enable pose detection, and read back landmarks via `bot.get_pose_keypoints()`.
`opencv-python`	Imported as `cv2`. Used to draw the skeleton lines, joint dots, the highlighted tracked landmark, and the stats panel, and to display the live video window.
`numpy`	Imported as `np`. Backs the image arrays that both `opencv-python` and `bonicbot-bridge` work with internally.

If you already installed these packages in an earlier lesson, you don’t need to reinstall them — the same environment works here too.

If pip install fails, try pip3 install ... instead, or use a virtual environment:


python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy

How to run the program

Find your BonicBot’s IP address. Check the robot’s on-device display, its companion app, or your router’s connected-devices list.
Save the code below into a file, e.g. lesson3_pose.py.
Replace [IP_ADDRESS] in the code with your robot’s actual IP address (see the callout box under the code block).
Make sure your BonicBot is powered on and connected to the same network as your computer, and that a full person (not just a face) can stand in view of its camera.
Run the script:
```
python lesson3_pose.py
```
A window titled “BonicBot Vision” should open, showing a green skeleton over any detected person, a red highlighted joint, and a small stats panel in the top-left corner.
Watch the terminal — a new line is only printed when pose tracking starts or stops.
Press q with the video window focused to stop the stream and exit cleanly.

Don’t have a physical BonicBot? Try it in simulation (optional)

If you don’t have physical access to a BonicBot, you can still work through this lesson using the ROS 2 simulation environment instead of a real robot:

Launch the BonicBot simulation:
```
ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True
```
- use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion are still simulated. This is the easiest way to test detection, since you can just hold whatever you want detected up to your laptop camera.
- use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).
Once the simulation is running, use localhost as the host instead of a physical IP address:
```
with BonicBot(host="localhost") as bot:
```
Everything else in the code — enabling pose detection, drawing the skeleton, reading bot.get_pose_keypoints() — works the same way, since the simulation exposes the same interface as a real robot.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, connecting to its actual IP address is the recommended way to go through this lesson.

Code

Click to view the complete program


import time
import cv2
import numpy as np
from bonicbot_bridge import BonicBot
 
# Standard 33-point MediaPipe Pose topology, hardcoded so no mediapipe
# import is needed — index order matches what the bridge's pose detector
# (which runs MediaPipe on the robot) reports.
POSE_LANDMARK_NAMES = [
    "NOSE", "LEFT_EYE_INNER", "LEFT_EYE", "LEFT_EYE_OUTER",
    "RIGHT_EYE_INNER", "RIGHT_EYE", "RIGHT_EYE_OUTER",
    "LEFT_EAR", "RIGHT_EAR", "MOUTH_LEFT", "MOUTH_RIGHT",
    "LEFT_SHOULDER", "RIGHT_SHOULDER", "LEFT_ELBOW", "RIGHT_ELBOW",
    "LEFT_WRIST", "RIGHT_WRIST", "LEFT_PINKY", "RIGHT_PINKY",
    "LEFT_INDEX", "RIGHT_INDEX", "LEFT_THUMB", "RIGHT_THUMB",
    "LEFT_HIP", "RIGHT_HIP", "LEFT_KNEE", "RIGHT_KNEE",
    "LEFT_ANKLE", "RIGHT_ANKLE", "LEFT_HEEL", "RIGHT_HEEL",
    "LEFT_FOOT_INDEX", "RIGHT_FOOT_INDEX",
]
 
POSE_CONNECTIONS = frozenset([
    (0, 1), (1, 2), (2, 3), (3, 7), (0, 4), (4, 5), (5, 6), (6, 8),
    (9, 10), (11, 12), (11, 13), (13, 15), (15, 17), (15, 19), (15, 21),
    (17, 19), (12, 14), (14, 16), (16, 18), (16, 20), (16, 22), (18, 20),
    (11, 23), (12, 24), (23, 24), (23, 25), (24, 26), (25, 27), (26, 28),
    (27, 29), (28, 30), (27, 31), (28, 32), (29, 31), (30, 32),
])
 
MIN_VISIBILITY = 0.6  # only trust landmarks MediaPipe thinks are actually visible
 
# The ONE landmark we "pull out by index" for the body-language demo.
# Swap this for any index in POSE_LANDMARK_NAMES — e.g. 15 for LEFT_WRIST,
# 12 for RIGHT_SHOULDER — once you're ready to read actual gestures instead
# of just the nose.
TRACKED_LANDMARK_INDEX = 16  # RIGHT_WRIST
 
SKELETON_COLOR = (0, 255, 0)   # joints + connecting lines
TRACKED_COLOR = (0, 0, 255)    # the one landmark pulled out by index
FLASH_DURATION = 12            # frames the border pulse stays visible after a change
 
 
def draw_pose(frame, landmarks):
    """Draw the full skeleton from the landmark list, then highlight the ONE
    landmark pulled out by index (TRACKED_LANDMARK) in a different color."""
    if not landmarks:
        return frame
    h_px, w_px = frame.shape[:2]
 
    def to_px(lm):
        return int(float(lm["x"]) * w_px), int(float(lm["y"]) * h_px)
 
    for a_idx, b_idx in POSE_CONNECTIONS:
        a, b = landmarks[a_idx], landmarks[b_idx]
        if float(a.get("visibility", 1.0)) < MIN_VISIBILITY or float(b.get("visibility", 1.0)) < MIN_VISIBILITY:
            continue
        cv2.line(frame, to_px(a), to_px(b), SKELETON_COLOR, 2)
 
    for lm in landmarks:
        if float(lm.get("visibility", 1.0)) < MIN_VISIBILITY:
            continue
        cv2.circle(frame, to_px(lm), 4, SKELETON_COLOR, -1)
 
    # The specific landmark pulled out by index — bigger dot + label
    tracked = landmarks[TRACKED_LANDMARK_INDEX]
    if float(tracked.get("visibility", 1.0)) >= MIN_VISIBILITY:
        px, py = to_px(tracked)
        cv2.circle(frame, (px, py), 9, TRACKED_COLOR, -1)
        cv2.putText(frame, f"[{TRACKED_LANDMARK_INDEX}] {POSE_LANDMARK_NAMES[TRACKED_LANDMARK_INDEX]}",
                    (px + 12, py - 8), cv2.FONT_HERSHEY_SIMPLEX, 0.6, TRACKED_COLOR, 2)
 
    return frame
 
 
def draw_hud(frame, landmarks, fps, flash_frames_left):
    """Translucent stats panel + a brief border flash whenever pose tracking
    is gained or lost — same pattern as the face-detection script, so the
    console only needs to log on real changes, not every frame."""
    h_px, w_px = frame.shape[:2]
    panel_w, panel_h = 230, 66
    overlay = frame.copy()
    cv2.rectangle(overlay, (10, 10), (10 + panel_w, 10 + panel_h), (0, 0, 0), -1)
    frame = cv2.addWeighted(overlay, 0.55, frame, 0.45, 0)
 
    if landmarks:
        visible = sum(1 for lm in landmarks if float(lm.get("visibility", 1.0)) >= MIN_VISIBILITY)
        status, color = f"Pose: {visible}/{len(landmarks)} visible", (0, 255, 0)
    else:
        status, color = "Pose: none", (0, 0, 255)
 
    cv2.putText(frame, status, (22, 38), cv2.FONT_HERSHEY_SIMPLEX, 0.62, color, 2)
    cv2.putText(frame, f"FPS: {fps:.1f}", (22, 64),
                cv2.FONT_HERSHEY_SIMPLEX, 0.55, (180, 180, 180), 1)
 
    if flash_frames_left > 0:
        border_color = (0, 255, 0) if landmarks else (0, 0, 255)
        cv2.rectangle(frame, (2, 2), (w_px - 3, h_px - 3), border_color, 4)
 
    return frame
 
 
with BonicBot(host='[IP_ADDRESS]') as bot:
    print("📷 Starting camera and streaming...")
    bot.system.start_camera()
    bot.start_camera()
    bot.camera.wait_for_image(timeout=5.0)
 
    print("🧠 Enabling pose detection (BonicBot Bridge)...")
    bot.enable_detection('pose')
 
    print("✅ Live Stream Active. Press 'q' to quit.\n")
 
    prev_time = time.time()
    fps = 0.0
    had_pose_last = None   # sentinel forces one log line on the very first frame
    flash_frames_left = 0
 
    try:
        while True:
            frame = bot.get_image()
            if frame is not None:
                landmarks = bot.get_pose_keypoints()  # full list, or []
 
                now = time.time()
                dt = now - prev_time
                prev_time = now
                if dt > 0:
                    fps = fps * 0.9 + (1.0 / dt) * 0.1
 
                has_pose = len(landmarks) > 0
                if has_pose != had_pose_last:
                    ts = time.strftime("%H:%M:%S")
                    if has_pose:
                        print(f"[{ts}] 🧍 Pose acquired — {len(landmarks)} landmarks "
                              f"(tracking [{TRACKED_LANDMARK_INDEX}] {POSE_LANDMARK_NAMES[TRACKED_LANDMARK_INDEX]})")
                    else:
                        print(f"[{ts}] ⚪ Pose lost")
                    had_pose_last = has_pose
                    flash_frames_left = FLASH_DURATION
 
                if flash_frames_left > 0:
                    flash_frames_left -= 1
 
                display = draw_pose(frame.copy(), landmarks)
                display = draw_hud(display, landmarks, fps, flash_frames_left)
                cv2.imshow("BonicBot Vision", display)
 
            if cv2.waitKey(1) & 0xFF == ord("q"):
                break
    finally:
        bot.disable_detection()
        cv2.destroyAllWindows()
        bot.stop_camera()
        bot.system.stop_camera()

Replace [IP_ADDRESS] with the IP address of your own BonicBot. Every BonicBot may have a different IP address depending on your network configuration.

📌 Example:


with BonicBot(host="172.20.10.2") as bot:

Replace 172.20.10.2 with the IP address assigned to your BonicBot.

Code Walkthrough

Line-by-line explanation

Imports — time handles timestamps and FPS calculation; cv2 (OpenCV) draws the skeleton, tracked joint, and HUD; numpy backs the image arrays; BonicBot connects to the robot.
POSE_LANDMARK_NAMES — A hardcoded list of the 33 standard MediaPipe Pose landmark names, in the exact order the bridge reports them. Having this locally means the script doesn’t need to import mediapipe itself just to get readable names.
POSE_CONNECTIONS — A set of index pairs describing which landmarks should be connected with a line (e.g. shoulder to elbow, elbow to wrist) to visually form a skeleton.
MIN_VISIBILITY — A threshold (0.6) below which a landmark is considered unreliable and skipped when drawing, since MediaPipe reports a per-landmark visibility/confidence score.
TRACKED_LANDMARK_INDEX — The single index (16, RIGHT_WRIST) that the script “pulls out” for special highlighting, demonstrating that any joint can be accessed directly by its fixed position in the landmark list.
SKELETON_COLOR / TRACKED_COLOR / FLASH_DURATION — Drawing colors for the general skeleton vs. the one tracked joint, and how many frames the border-flash effect lasts after pose tracking starts or stops.
draw_pose(frame, landmarks) — Converts each landmark’s normalized x/y to pixel coordinates, draws a line for every connection in POSE_CONNECTIONS (skipping any pair where either landmark is below MIN_VISIBILITY), draws a small dot on every visible landmark, then draws a larger dot and text label on just the one landmark at TRACKED_LANDMARK_INDEX.
draw_hud(frame, landmarks, fps, flash_frames_left) — Draws the same style of semi-transparent stats panel as Lesson 2, showing how many landmarks are currently visible (or “Pose: none”) and the current FPS, plus the same border-flash effect when tracking status changes.
bot.enable_detection('pose') — Switches the robot’s vision pipeline to the pose-estimation model instead of object or face detection.
bot.get_pose_keypoints() — Returns the full list of 33 landmarks for the currently detected person, or an empty list if no one is in view.
had_pose_last = None — Starts as None (rather than True/False) specifically so the very first frame always triggers one log line, establishing the initial tracking state.
Change-triggered logging (if has_pose != had_pose_last:) — Only prints to the terminal when pose tracking is newly acquired or newly lost, rather than every single frame.
Main loop / finally cleanup — Follows the same structure as Lesson 2: grab a frame, get landmarks, update FPS and logging, draw the skeleton and HUD, show the window, check for q, and always disable detection and stop the camera in the finally block regardless of how the loop ends.

Expected Output

Click to see expected output

Visual Output:

Terminal Output:


🧠 Loading pose detection model...
✅ Vision system ready.

[14:21:10] 🧍 Pose acquired — 33 landmarks (tracking [16] RIGHT_WRIST)

The BonicBot Vision Window will display:

The complete body skeleton
A highlighted right wrist
A label showing the tracked landmark
A live pose status
Current processing speed (FPS)
A brief border flash whenever pose tracking starts or stops

🔧 Under the Hood

How does BonicBot estimate a person’s pose?

Every image entering BonicBot’s vision system is analyzed using MediaPipe Pose, a deep learning model that estimates the positions of important body joints.

Instead of producing just one bounding box, the model predicts 33 body landmarks, each containing:

x position
y position
relative depth (z)
visibility score

The program draws lines between these landmarks to create a digital skeleton that follows the person’s movements.

Every landmark always occupies the same location in the list.

For example:

Landmark 0 → Nose
Landmark 11 → Left Shoulder
Landmark 12 → Right Shoulder
Landmark 15 → Left Wrist
Landmark 16 → Right Wrist

Because these positions never change, BonicBot can immediately retrieve any joint simply by accessing its index.

The program also tracks whether a pose is currently visible. Rather than printing messages continuously, it only reports when pose tracking starts or stops, producing a much cleaner event log.

Student Challenge

Instead of highlighting BonicBot’s right wrist, modify the program so it tracks the left wrist or nose.

Observe how the highlighted point moves as you change the tracked landmark.

Hint

Replace:


TRACKED_LANDMARK = mp_pose.PoseLandmark.RIGHT_WRIST

with another landmark, such as:


TRACKED_LANDMARK = mp_pose.PoseLandmark.LEFT_WRIST


TRACKED_LANDMARK = mp_pose.PoseLandmark.NOSE

Reflection Question

BonicBot can accurately locate body joints, but it still doesn’t know what those movements mean.

What additional logic would you need to write if you wanted BonicBot to recognize actions such as waving, raising a hand, or giving a thumbs-up?