Lesson 4: Teaching BonicBot to Recognize Hand Gestures

Learning Objective

Teach BonicBot to recognize hand gestures in real time, visualize the detected hand skeleton, and trigger robot actions based on recognized gestures.

Introduction

BonicBot can now detect faces and track body joints. The next step is teaching it to understand simple hand gestures.

Instead of only detecting where a hand is, BonicBot can recognize specific gestures such as an open palm, thumbs up, peace sign, or fist. These gestures can then be used to control the robot without touching it.

In this lesson, BonicBot uses its built-in gesture detection system to continuously analyze its surroundings. Whenever it recognizes a hand gesture, it displays:

the detected hand skeleton
the fingertips
whether it is the left or right hand
the recognized gesture
the current number of detected hands

To make gesture recognition more reliable, the program waits until the same gesture has been detected for several consecutive frames before accepting it. This process, called debouncing, helps prevent accidental detections caused by temporary hand movements.

Once a gesture has been confirmed, BonicBot can immediately perform an action. In this lesson, an Open Palm makes the robot move forward, while a Thumbs Up safely stops the robot. Additional gestures are left as activities for you to implement later.

Setup: Installing Packages

Before running the code, make sure your computer has the required Python packages installed. Open a terminal and run:


pip install bonicbot-bridge opencv-python numpy

What each package does

Package	Purpose
`bonicbot-bridge`	The official BonicBot SDK. Provides the `BonicBot` class used to connect to the robot, control its camera, enable gesture detection, read hands back via `bot.get_gesture_full()`, and send movement commands like `bot.move()` and `bot.stop()`. Also provides the `bonicbot_bridge.exceptions` module used for safe error handling (`NavigationError`, `BonicBotError`).
`opencv-python`	Imported as `cv2`. Used to draw the hand skeleton, fingertip highlights, gesture labels, and the stats panel, and to display the live video window.
`numpy`	Backs the image arrays that both `opencv-python` and `bonicbot-bridge` work with internally.

If you already installed these packages in an earlier lesson, you don’t need to reinstall them — the same environment works here too.

If pip install fails, try pip3 install ... instead, or use a virtual environment:


python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy

How to run the program

Find your BonicBot’s IP address. Check the robot’s on-device display, its companion app, or your router’s connected-devices list.
Save the code below into a file, e.g. lesson4_gestures.py.
Replace [IP_ADDRESS] in the code with your robot’s actual IP address (see the callout box under the code block).
Make sure your BonicBot is powered on, connected to the same network as your computer, and has clear space in front of it to move — this lesson makes the robot drive forward on an Open Palm gesture.
Run the script:
```
python lesson4_gestures.py
```
A window titled “BonicBot Vision” should open, showing the hand skeleton, highlighted fingertips, the recognized gesture, and a small stats panel in the top-left corner.
Hold an Open Palm steadily to make the robot move forward, and a Thumbs Up to stop it.
Press q with the video window focused to stop the stream and exit cleanly — the finally block will also stop the robot’s motion automatically.

Don’t have a physical BonicBot? Try it in simulation (optional)

If you don’t have physical access to a BonicBot, you can still work through this lesson using the ROS 2 simulation environment instead of a real robot. This is especially useful here since the lesson also drives the robot’s motion, and the simulation lets you see that motion safely in a virtual world instead of on real hardware:

Launch the BonicBot simulation:
```
ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True
```
- use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion are still simulated. This is the easiest way to test detection, since you can just hold whatever you want detected up to your laptop camera.
- use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).
Once the simulation is running, use localhost as the host instead of a physical IP address:
```
with BonicBot(host="localhost") as bot:
```
Everything else in the code — enabling gesture detection, drawing the hand skeleton, and the bot.move() / bot.stop() commands — works the same way, since the simulation exposes the same interface as a real robot, and you’ll see the simulated BonicBot drive forward in the virtual world.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, connecting to its actual IP address is the recommended way to go through this lesson.

Code

Click to view the complete program


 
import time
import cv2
from bonicbot_bridge import BonicBot
from bonicbot_bridge.exceptions import NavigationError, BonicBotError
 
MIN_VISIBILITY = 0.5  # Only trust landmarks with reasonable confidence
GESTURE_HOLD_FRAMES = 8  # Debounce frame buffer for stable tracking
 
SKELETON_COLOR = (0, 255, 0)    # Hand connections
TRACKED_COLOR = (0, 0, 255)     # Fingertips + gesture label
FLASH_DURATION = 12             # Frame duration for border flash
 
# --- Safe Motion Tuning Constants ---
FORWARD_SPEED = 0.15    # m/s, continuous forward velocity
TURN_ANGULAR = 0.3      # rad/s, turning velocity
 
# Standard MediaPipe 21 landmark indices mapping
HAND_LANDMARK_NAMES = [
    "WRIST",
    "THUMB_CMC", "THUMB_MCP", "THUMB_IP", "THUMB_TIP",
    "INDEX_FINGER_MCP", "INDEX_FINGER_PIP", "INDEX_FINGER_DIP", "INDEX_FINGER_TIP",
    "MIDDLE_FINGER_MCP", "MIDDLE_FINGER_PIP", "MIDDLE_FINGER_DIP", "MIDDLE_FINGER_TIP",
    "RING_FINGER_MCP", "RING_FINGER_PIP", "RING_FINGER_DIP", "RING_FINGER_TIP",
    "PINKY_MCP", "PINKY_PIP", "PINKY_DIP", "PINKY_TIP",
]
 
HAND_CONNECTIONS = [
    (0, 1), (1, 2), (2, 3), (3, 4),           # Thumb
    (0, 5), (5, 6), (6, 7), (7, 8),           # Index
    (5, 9), (9, 10), (10, 11), (11, 12),      # Middle
    (9, 13), (13, 14), (14, 15), (15, 16),    # Ring
    (13, 17), (17, 18), (18, 19), (19, 20),   # Pinky
    (0, 17),                                  # Palm base
]
 
FINGER_TIPS = [4, 8, 12, 16, 20]
WRIST_IDX = 0
INDEX_TIP_IDX = 8
 
 
def normalize_hands(raw):
    """Parses raw payload structure safely into a unified list schema."""
    if not raw:
        return []
 
    hands = []
    for item in raw:
        if not isinstance(item, dict):
            continue
 
        gesture = item.get("gesture") or item.get("label")
        score = item.get("score", item.get("confidence"))
        handedness = item.get("handedness") or item.get("hand")
        raw_points = item.get("landmarks") or item.get("keypoints")
        
        landmarks = []
        if raw_points:
            for idx, kp in enumerate(raw_points):
                if isinstance(kp, dict):
                    x, y = kp.get("x"), kp.get("y")
                    vis = kp.get("visibility", kp.get("confidence", 1.0))
                else:
                    vals = list(kp)
                    x, y = vals[0], vals[1]
                    vis = vals[2] if len(vals) >= 3 else 1.0
 
                if x is None or y is None:
                    vis = 0.0
 
                name = HAND_LANDMARK_NAMES[idx] if idx < len(HAND_LANDMARK_NAMES) else str(idx)
                landmarks.append({"index": idx, "name": name, "x": x, "y": y, "visibility": vis})
 
        hands.append({
            "gesture": gesture,
            "score": score,
            "handedness": handedness,
            "landmarks": landmarks,
        })
    return hands
 
 
def _pointing_turn_direction(hand):
    """Calculates relative pointer heading dynamically from keypoints."""
    landmarks = hand.get("landmarks") or []
    if len(landmarks) <= max(WRIST_IDX, INDEX_TIP_IDX):
        return 0
 
    wrist = landmarks[WRIST_IDX]
    tip = landmarks[INDEX_TIP_IDX]
    if wrist["visibility"] < MIN_VISIBILITY or tip["visibility"] < MIN_VISIBILITY:
        return 0
 
    dx = tip["x"] - wrist["x"]
    if abs(dx) < 1e-3:
        return 0
    return 1 if dx > 0 else -1
 
 
def draw_hands(frame, hands):
    if not hands:
        return frame
    h_px, w_px = frame.shape[:2]
 
    def to_px(lm):
        x, y = lm["x"], lm["y"]
        if x is None or y is None:
            return None
        if 0 <= x <= 1 and 0 <= y <= 1:
            return int(x * w_px), int(y * h_px)
        return int(x), int(y)
 
    for hand in hands:
        landmarks = hand["landmarks"]
        for a_idx, b_idx in HAND_CONNECTIONS:
            if a_idx >= len(landmarks) or b_idx >= len(landmarks):
                continue
            a, b = landmarks[a_idx], landmarks[b_idx]
            if a["visibility"] < MIN_VISIBILITY or b["visibility"] < MIN_VISIBILITY:
                continue
            pa, pb = to_px(a), to_px(b)
            if pa is not None and pb is not None:
                cv2.line(frame, pa, pb, SKELETON_COLOR, 2)
 
        for lm in landmarks:
            if lm["visibility"] < MIN_VISIBILITY:
                continue
            p = to_px(lm)
            if p is not None:
                cv2.circle(frame, p, 3, SKELETON_COLOR, -1)
 
        for tip_idx in FINGER_TIPS:
            if tip_idx >= len(landmarks):
                continue
            lm = landmarks[tip_idx]
            if lm["visibility"] < MIN_VISIBILITY:
                continue
            p = to_px(lm)
            if p is not None:
                cv2.circle(frame, p, 6, TRACKED_COLOR, -1)
 
        if landmarks:
            label_pos = to_px(landmarks[0])
            if label_pos:
                label = f"{hand['handedness'] or '?'}: {hand['gesture'] or 'Unknown'}"
                cv2.putText(frame, label, (label_pos[0] - 20, label_pos[1] + 30),
                            cv2.FONT_HERSHEY_SIMPLEX, 0.6, TRACKED_COLOR, 2)
    return frame
 
 
def draw_hud(frame, hands, stable_gesture, fps, flash_frames_left):
    h_px, w_px = frame.shape[:2]
    panel_w, panel_h = 260, 84
    overlay = frame.copy()
    cv2.rectangle(overlay, (10, 10), (10 + panel_w, 10 + panel_h), (0, 0, 0), -1)
    frame = cv2.addWeighted(overlay, 0.55, frame, 0.45, 0)
 
    hand_status = f"Hands: {len(hands)}" if hands else "Hands: none"
    color = (0, 255, 0) if hands else (0, 0, 255)
    cv2.putText(frame, hand_status, (22, 36), cv2.FONT_HERSHEY_SIMPLEX, 0.62, color, 2)
    cv2.putText(frame, f"Gesture: {stable_gesture or '-'}", (22, 60),
                cv2.FONT_HERSHEY_SIMPLEX, 0.58, (0, 200, 255), 2)
    cv2.putText(frame, f"FPS: {fps:.1f}", (22, 82),
                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (180, 180, 180), 1)
 
    if flash_frames_left > 0:
        border_color = (0, 255, 0) if hands else (0, 0, 255)
        cv2.rectangle(frame, (2, 2), (w_px - 3, h_px - 3), border_color, 4)
    return frame
 
 
# --- Main Loop Execution ---
with BonicBot(host='[IP_ADDRESS]') as bot:
    print("📷 Initializing physical camera hardware and frame streams...")
    bot.system.start_camera()
    bot.start_camera()
    bot.camera.wait_for_image(timeout=5.0)
 
    print("🧠 Starting on-board edge gesture execution pipeline...")
    bot.enable_detection('gesture')
    print("✅ Live Stream Window Up. Press 'q' inside video window to exit safely.\n")
 
    prev_time = time.time()
    fps = 0.0
 
    last_frame_gesture = None
    same_gesture_count = 0
    stable_gesture = None
    flash_frames_left = 0
    lost_frames = 0
 
    GESTURE_LOST_GRACE = GESTURE_HOLD_FRAMES  # frames of no-hand tolerated before treating gesture as lost
 
    try:
        while True:
            frame = bot.camera.get_latest_image()
            if frame is not None:
                raw_gestures = bot.get_gesture_full()
                hands = normalize_hands(raw_gestures)
 
                now = time.time()
                dt = now - prev_time
                prev_time = now
                if dt > 0:
                    fps = fps * 0.9 + (1.0 / dt) * 0.1
 
                raw_gesture = hands[0]["gesture"] if hands else None
                if raw_gesture is not None:
                    lost_frames = 0
                    current_gesture = raw_gesture
                else:
                    lost_frames += 1
                    # hold onto the last confirmed gesture through brief dropouts
                    current_gesture = stable_gesture if lost_frames <= GESTURE_LOST_GRACE else None
 
                # --- Debounce Pipeline ---
                if current_gesture == last_frame_gesture:
                    same_gesture_count += 1
                else:
                    last_frame_gesture = current_gesture
                    same_gesture_count = 1
 
                if same_gesture_count == GESTURE_HOLD_FRAMES and current_gesture != stable_gesture:
                    stable_gesture = current_gesture
                    ts = time.strftime("%H:%M:%S")
                    if stable_gesture:
                        print(f"[{ts}] ✋ Confirmed Stable Gesture: {stable_gesture}")
                    else:
                        print(f"[{ts}] ⚪ Tracking Frame Lost")
                    flash_frames_left = FLASH_DURATION
 
                    # =========================================================================
                    # STUDENT ACTIVITY SPACE: MATCH GESTURES TO ROBOT RESPONSES
                    # =========================================================================
                    
                    if stable_gesture == "Open_Palm":
                        pass  # handled by the continuous per-frame send below
 
                    elif stable_gesture == "Thumb_Up":
                        # Command the robot to stop immediately
                        bot.stop()
 
                    elif stable_gesture == "Closed_Fist":
                        # TODO Student Activity: Add actions for "Closed_Fist"
                        pass
 
                    elif stable_gesture == "Victory":
                        # TODO Student Activity: Add actions for "Victory"
                        pass
 
                    elif stable_gesture == "Pointing_Up":
                        # TODO Student Activity: Add actions for directional tracking
                        pass
 
                    elif stable_gesture == "Thumb_Down":
                        # TODO Student Activity: Add actions for "Thumb_Down"
                        pass
 
                    elif stable_gesture == "ILoveYou":
                        # TODO Student Activity: Add actions for "ILoveYou"
                        pass
 
                    elif stable_gesture == "Forward":
                        # TODO Student Activity: Add alternative forward gesture response
                        pass
 
                    elif stable_gesture == "Backward":
                        # TODO Student Activity: Add alternative reverse gesture response
                        pass
 
                    else:
                        # CRITICAL FAIL-SAFE: If gesture tracking drops, immediately stop movement 
                        bot.stop()
                        bot.look_center()
                    # =========================================================================
 
                if flash_frames_left > 0:
                    flash_frames_left -= 1
 
                # Re-send drive command every frame — robot stops itself if cmd_vel isn't refreshed
                if stable_gesture == "Open_Palm":
                    bot.move(FORWARD_SPEED, 0.0, 0.0)
 
                display = draw_hands(frame.copy(), hands)
                display = draw_hud(display, hands, stable_gesture, fps, flash_frames_left)
                cv2.imshow("BonicBot Vision", display)
 
            if cv2.waitKey(1) & 0xFF == ord("q"):
                break
    finally:
        # Runtime teardown safety fallback handler
        print("\n🛑 Safety shutdown initiated. Killing engine vectors...")
        try:
            bot.stop()
        except BonicBotError:
            pass
        bot.disable_detection()
        cv2.destroyAllWindows()
        bot.stop_camera()
        bot.system.stop_camera()
        print("Done.")

Replace [IP_ADDRESS] with the IP address of your own BonicBot. Every BonicBot may have a different IP address depending on your network configuration.

📌 Example:


with BonicBot(host="172.20.10.2") as bot:

Replace 172.20.10.2 with the IP address assigned to your BonicBot.

Code Walkthrough

Line-by-line explanation

Imports — time and cv2 are used as in earlier lessons; BonicBot connects to the robot; NavigationError and BonicBotError are exception types the script can catch if a motion command fails, so a shutdown doesn’t crash with an unhandled error.
Tuning constants (MIN_VISIBILITY, GESTURE_HOLD_FRAMES, SKELETON_COLOR, TRACKED_COLOR, FLASH_DURATION, FORWARD_SPEED, TURN_ANGULAR) — Central knobs for how confident a landmark must be to draw, how many consecutive matching frames confirm a gesture, drawing colors, and how fast the robot should drive/turn.
HAND_LANDMARK_NAMES / HAND_CONNECTIONS / FINGER_TIPS — The standard 21-point MediaPipe Hands layout: readable names for each landmark, which pairs of landmarks should be connected to draw the hand skeleton, and which five indices are fingertips (for the highlighted dots).
normalize_hands(raw) — The bridge’s raw gesture payload can vary in shape (different key names, landmarks as dicts or plain lists), so this function converts whatever comes back into one consistent structure: a list of hands, each with a gesture, score, handedness, and a clean list of {index, name, x, y, visibility} landmarks. Writing a “normalize” function like this is a common pattern any time an external API’s response format isn’t perfectly predictable.
_pointing_turn_direction(hand) — A helper (not yet wired into the main gesture actions) that compares the wrist and index fingertip’s x-position to guess whether a pointing hand is angled left or right — a building block left for the “Pointing” student activity.
draw_hands(frame, hands) — For every detected hand, draws the skeleton lines and joint dots (skipping any landmark below MIN_VISIBILITY), highlights the five fingertip positions in a different color, and labels the hand with its handedness and recognized gesture.
draw_hud(...) — Same translucent stats-panel pattern as earlier lessons, but now showing hand count, the current stable gesture, and FPS, plus the familiar border-flash effect.
bot.enable_detection('gesture') — Switches the robot’s vision pipeline to the gesture-recognition model.
bot.get_gesture_full() — Returns the raw per-frame gesture/hand data, which is immediately cleaned up by normalize_hands().
Dropout tolerance (lost_frames, GESTURE_LOST_GRACE) — If no hand is seen for a frame, the script doesn’t instantly forget the gesture; it holds onto the last stable gesture for up to GESTURE_LOST_GRACE frames before treating it as truly lost, so brief flickers in detection don’t interrupt the robot’s behavior.
Debounce pipeline (same_gesture_count, GESTURE_HOLD_FRAMES) — Counts how many consecutive frames report the same raw gesture. Only once that count reaches GESTURE_HOLD_FRAMES does the script treat the gesture as “confirmed” (stable_gesture), which filters out momentary misreads.
Student Activity Space (if stable_gesture == "Open_Palm": ...) — A chain of elif blocks, one per recognized gesture name, where Thumb_Up is already wired to bot.stop() and most others are left as pass placeholders for you to fill in with your own robot actions.
Fail-safe else branch — If the confirmed gesture doesn’t match any known name (including None, meaning tracking was lost), the robot immediately stops and recenters its head via bot.look_center(), so it never keeps performing a stale action once gesture tracking drops.
Continuous drive command (if stable_gesture == "Open_Palm": bot.move(...)) — Runs every frame, independent of the debounce block above. Many robot motion APIs expect a fresh velocity command on a regular basis (it’s commonly tied to something called a “cmd_vel” heartbeat) — if you stop sending it, the robot stops on its own as a safety measure, so this line keeps re-issuing “keep moving forward” for as long as the Open Palm gesture stays confirmed.
finally: safety shutdown — Always attempts bot.stop() first (catching BonicBotError in case the robot is already disconnected or stopped), then disables detection, closes the window, and stops the camera — ensuring the robot never keeps moving after the script exits, even if it crashes.

Expected Output

Click to see expected output

Visual Output:

Terminal Output:


🧠 Starting on-board gesture detection...
✅ Vision system ready.

[14:32:18] ✋ Confirmed Stable Gesture: Open_Palm
[14:32:24] ✋ Confirmed Stable Gesture: Thumbs_Up

The BonicBot Vision Window will display:

A hand skeleton for every detected hand
Highlighted fingertips
The detected hand (Left or Right)
The recognized gesture
Number of detected hands
Current processing speed (FPS)
A border flash whenever gesture tracking changes

When an Open Palm is held steadily, BonicBot moves forward.

When a Thumbs Up is held steadily, BonicBot stops safely.

🔧 Under the Hood

How does BonicBot recognize hand gestures?

BonicBot’s built-in gesture detector first locates a hand and estimates 21 hand landmarks representing the wrist and finger joints.

These landmarks describe the shape of the hand. The gesture recognition system then analyzes the arrangement of these landmarks to determine which gesture is being shown.

Some examples include:

Open Palm
Fist
Peace
Pointing
Thumbs Up
Thumbs Down
I Love You

The program also uses a technique called debouncing.

Instead of reacting immediately to every prediction, BonicBot waits until the same gesture has been detected for several consecutive frames before accepting it. This reduces false detections and makes robot behavior much more stable.

Once a gesture has been confirmed, your Python program decides what the robot should do. This means you can easily customize BonicBot by assigning different actions to different gestures.

Student Challenge

Add your own robot actions for the remaining gestures.

For example:

✊ Fist → Turn left
✌️ Peace → Turn right
👇 Pointing → Make BonicBot look toward the pointing direction
👎 Thumbs Down → Move backward
🤟 I Love You → Make BonicBot speak a greeting

Try creating your own gesture-controlled robot!

Hint

Inside the Student Activity Space, you’ll find several empty elif blocks.

Replace each pass statement with the BonicBot movement or speech command you want to execute.

Reflection Question

Why do you think BonicBot waits until the same gesture has been detected for several consecutive frames before responding, instead of reacting immediately to every prediction?