Lesson 06: Teaching BonicBot to Detect Colors

Learning Objective

Teach BonicBot to detect a specific color, track its movement, and announce when the colored object has been found.

Introduction

So far in this Artificial Intelligence series, BonicBot has used trained AI models to recognize objects, faces, gestures, and people.

In this lesson, we explore another core discipline within Artificial Intelligence — Computer Vision — using a classical, rule-based technique called color detection, rather than a trained deep learning model.

Instead of asking BonicBot:

“What is this object?”

we ask:

“Where is the red object?”

“Can you follow the blue object?”

The robot does this by looking for a specific range of colors instead of recognizing the object’s shape or identity.

As BonicBot detects a colored object, it will:

isolate only the selected color
locate the largest object of that color
draw a tracking marker
announce when the object has been found
announce the direction in which the object is moving
wave its arm in the direction the object is moving

You can also click on any color in the Vision Window to instantly teach BonicBot a new target color.

Setup: Installing Packages

Before running the code, make sure your computer has the required Python packages installed. Open a terminal and run:


pip install bonicbot-bridge opencv-python numpy pyttsx3

What each package does

Package	Purpose
`bonicbot-bridge`	The official BonicBot SDK. Provides the `BonicBot` class used to connect to the robot, control its camera, and send arm movement commands via `bot.move_left_arm()` / `bot.move_right_arm()`. Color detection itself runs entirely in this script using OpenCV — it doesn’t rely on an onboard `bot.enable_detection()` model like earlier lessons.
`opencv-python`	Imported as `cv2`. Does the heavy lifting in this lesson: converting frames to HSV, building the color mask, finding the largest blob, drawing the tracking overlay and mask preview, handling mouse clicks, and displaying the video window.
`numpy`	Imported as `np`. Used to define the HSV color ranges in `COLOR_PRESETS` and for the array math behind color masking and click-to-calibrate sampling.
`pyttsx3`	Text-to-speech library used by `VoiceSpeaker` to announce detections and movement direction. Optional — if it isn’t installed, the script still detects and tracks colors normally, it just skips the spoken announcements.

If you already installed bonicbot-bridge, opencv-python, numpy, and pyttsx3 in Lesson 5, you don’t need to reinstall anything for this lesson.

If pip install fails, try pip3 install ... instead, or use a virtual environment:


python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy pyttsx3

How to run the program

Note the connection line. Unlike earlier lessons, this script’s with BonicBot(host='localhost') as bot: line already defaults to localhost rather than an [IP_ADDRESS] placeholder.
- If you’re running this against the ROS 2 simulation (see below), you can leave it as localhost and run it as-is.
- If you’re connecting to a real BonicBot, replace 'localhost' with your robot’s actual IP address, e.g. host="172.20.10.2" — find it via the robot’s on-device display, its companion app, or your router’s connected-devices list.
Save the code below into a file, e.g. lesson6_color.py.
Make sure your BonicBot (real or simulated) is powered on/running and has a colorful object available to test with — anything from the 8 presets (red, orange, yellow, green, cyan, blue, purple, pink) or any object you click on to calibrate.
(Optional) Connect a Bluetooth speaker to hear the spoken announcements. Detection and tracking still work without one.
Run the script:
```
python lesson6_color.py
```
A window titled “BonicBot Vision - Color” should open, showing the live feed, a yellow tracking circle/crosshair once a color is locked on, and a small color-mask preview in the corner.
Press keys 1–8 to switch between the eight preset colors, or click directly on any object in the video window to calibrate BonicBot to that object’s color on the fly.
Press q with the video window focused to stop the stream and exit cleanly.

Don’t have a physical BonicBot? Try it in simulation (optional)

This lesson’s code already defaults to host='localhost', so it’s already set up to run directly against the ROS 2 simulation environment without any code changes:

Launch the BonicBot simulation:
```
ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True
```
- use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion (including the arm-wave gesture) are still simulated. This is the easiest way to test detection, since you can just hold whatever colored object you want detected up to your laptop camera.
- use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).
With the simulation running, simply run the script as written — no [IP_ADDRESS] substitution is needed here.
Everything else in the code — color masking, blob tracking, voice announcements, and the arm wave — works the same way, since the simulation exposes the same interface as a real robot.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, switching host='localhost' to its actual IP address is the recommended way to go through this lesson on real hardware.

Code

Click to view the complete program


import time
import threading
import queue
import collections
import cv2
import numpy as np
from bonicbot_bridge import BonicBot
 
try:
    import pyttsx3
    _TTS_AVAILABLE = True
except ImportError:
    _TTS_AVAILABLE = False
 
 
class VoiceSpeaker:
    def __init__(self, rate=175, volume=0.85, min_gap_seconds=4.0):
        self._queue = queue.Queue()
        self._rate = rate
        self._volume = volume
        self._min_gap = min_gap_seconds
        self._last_spoken_ts = 0.0
        self._thread = None
        if _TTS_AVAILABLE:
            self._thread = threading.Thread(target=self._run, daemon=True)
            self._thread.start()
 
    def _run(self):
        while True:
            text = self._queue.get()
            if text is None:
                break
            try:
                engine = pyttsx3.init()
                engine.setProperty('rate', self._rate)
                engine.setProperty('volume', self._volume)
                engine.say(text)
                engine.runAndWait()
                engine.stop()
                del engine
            except Exception as e:
                print(f"⚠️ TTS error: {e}")
 
    def speak(self, text):
        if not _TTS_AVAILABLE:
            return
        now = time.time()
        if now - self._last_spoken_ts < self._min_gap:
            return
        self._last_spoken_ts = now
        while not self._queue.empty():
            try:
                self._queue.get_nowait()
            except queue.Empty:
                break
        self._queue.put(text)
 
    def stop(self):
        if _TTS_AVAILABLE:
            self._queue.put(None)
            self._thread.join(timeout=1)
 
 
COLOR_PRESETS = {
    ord("1"): ("red",    np.array([0,   180, 120]), np.array([9,   255, 255])),
    ord("2"): ("orange", np.array([11,  175, 130]), np.array([23,  255, 255])),
    ord("3"): ("yellow", np.array([25,  150, 140]), np.array([35,  255, 255])),
    ord("4"): ("green",  np.array([38,  110,  90]), np.array([83,  255, 255])),
    ord("5"): ("cyan",   np.array([85,  110, 110]), np.array([101, 255, 255])),
    ord("6"): ("blue",   np.array([104, 130,  80]), np.array([129, 255, 255])),
    ord("7"): ("purple", np.array([132, 110,  90]), np.array([151, 255, 255])),
    ord("8"): ("pink",   np.array([154, 100, 150]), np.array([166, 155, 255])),
}
current_color_name, HSV_LOWER_DEFAULT, HSV_UPPER_DEFAULT = COLOR_PRESETS[ord("1")]
 
 
def name_for_hue(hue):
    for ub, name in [(9,"red"),(24,"orange"),(36,"yellow"),(84,"green"),
                     (92,"cyan"),(129,"blue"),(152,"purple"),(167,"pink"),(179,"red")]:
        if hue <= ub:
            return name
    return "colour"
 
 
# ── Tunables ─────────────────────────────────────────────────────────────────
MIN_BLOB_AREA          = 3000      # in FULL-resolution pixel^2 (auto-scaled internally)
PROCESS_WIDTH           = 320       # detection runs on a frame downscaled to this width
                                     # (same trick the RPi's YOLO node uses: imgsz=320)
DETECTION_FRAME_SKIP    = 2         # process 1 of every N *new* frames (like the RPi's yolo_skip=5)
GAUSSIAN_KSIZE          = (7, 7)    # was (17,17) — big win, especially after downscaling
MORPH_KSIZE             = (5, 5)    # was (9,9)
MORPH_ITERS             = 1         # was 2
MASK_PREVIEW_H          = 120
DIRECTION_MIN_DELTA     = 20
DIRECTION_COOLDOWN      = 1.5
DIRECTION_HISTORY_LEN   = 8
MIN_SOLIDITY            = 0.4
STABLE_FRAME_THRESHOLD  = 10
MISS_TOLERANCE          = 20
LOCK_HOLD_TIMEOUT       = 1.5
REANNOUNCE_COOLDOWN     = 5.0
SMOOTHING_ALPHA         = 0.25
CLICK_TOLERANCE         = np.array([5, 55, 50])
latest_frame            = None
 
 
def apply_color_mask(frame, lower_hsv, upper_hsv, color_name):
    hsv = cv2.GaussianBlur(cv2.cvtColor(frame, cv2.COLOR_BGR2HSV), GAUSSIAN_KSIZE, 0)
    if color_name == "red":
        mask = cv2.bitwise_or(
            cv2.inRange(hsv, lower_hsv, upper_hsv),
            cv2.inRange(hsv, np.array([165, lower_hsv[1], lower_hsv[2]]),
                             np.array([179, upper_hsv[1], upper_hsv[2]]))
        )
    else:
        mask = cv2.inRange(hsv, lower_hsv, upper_hsv)
    k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, MORPH_KSIZE)
    mask = cv2.erode(mask, k, iterations=MORPH_ITERS)
    mask = cv2.dilate(mask, k, iterations=MORPH_ITERS)
    return cv2.medianBlur(mask, 3)
 
 
def find_largest_blob(mask, min_area):
    contours, _ = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    if not contours:
        return None
    biggest = max(contours, key=cv2.contourArea)
    area = cv2.contourArea(biggest)
    if area < min_area:
        return None
    M = cv2.moments(biggest)
    if M["m00"] == 0:
        return None
    (ex, ey), radius = cv2.minEnclosingCircle(biggest)
    if (area / (np.pi * radius * radius)) < MIN_SOLIDITY:
        return None
    return {"cx": int(M["m10"]/M["m00"]), "cy": int(M["m01"]/M["m00"]),
            "ex": int(ex), "ey": int(ey), "radius": float(radius)}
 
 
def draw_tracking(frame, blob, sx, sy):
    if blob is None or sx is None:
        return frame
    cv2.circle(frame, (blob["ex"], blob["ey"]), int(blob["radius"]), (0, 255, 255), 2)
    cx, cy = int(sx), int(sy)
    cv2.line(frame, (cx-14, cy), (cx+14, cy), (0, 255, 255), 2)
    cv2.line(frame, (cx, cy-14), (cx, cy+14), (0, 255, 255), 2)
    cv2.circle(frame, (cx, cy), 5, (0, 200, 255), -1)
    return frame
 
 
def blit_mask_preview(frame, thumb):
    """Blit an already-small thumbnail prepared by the worker thread — no resize here."""
    if thumb is None:
        return frame
    h, w = frame.shape[:2]
    th, tw = thumb.shape[:2]
    if h <= th + 20 or w <= tw + 20:
        return frame
    frame[h-th-10: h-10, w-tw-10: w-10] = thumb
    return frame
 
 
def draw_hud(frame, confirmed, color_name, lower_hsv, upper_hsv, detect_fps, display_fps):
    x0, y0, x1, y1 = 10, 10, 310, 112
    roi = frame[y0:y1, x0:x1]
    black = np.zeros_like(roi)
    cv2.addWeighted(black, 0.55, roi, 0.45, 0, dst=roi)  # blend only the HUD rectangle, not the full frame
    status, col = (f"LOCKED: {color_name.upper()}", (0,255,0)) if confirmed \
                  else (f"SCANNING FOR {color_name.upper()}", (0,0,255))
    cv2.putText(frame, status, (22, 36), cv2.FONT_HERSHEY_SIMPLEX, 0.62, col, 2)
    cv2.putText(frame, f"H {lower_hsv[0]}-{upper_hsv[0]}  S {lower_hsv[1]}-{upper_hsv[1]}  V {lower_hsv[2]}-{upper_hsv[2]}",
                (22, 65), cv2.FONT_HERSHEY_SIMPLEX, 0.46, (180,180,180), 1)
    cv2.putText(frame, f"detect: {detect_fps:.1f} fps   display: {display_fps:.1f} fps",
                (22, 90), cv2.FONT_HERSHEY_SIMPLEX, 0.46, (180,180,180), 1)
    return frame
 
 
def _push_color_change(q, name, lower, upper):
    while not q.empty():
        try:
            q.get_nowait()
        except queue.Empty:
            break
    q.put((name, lower, upper))
 
 
def on_mouse_click(event, x, y, flags, param):
    color_change_q = param
    if event != cv2.EVENT_LBUTTONDOWN or latest_frame is None:
        return
    h, w = latest_frame.shape[:2]
    patch = cv2.cvtColor(latest_frame[max(0,y-5):min(h,y+5), max(0,x-5):min(w,x+5)],
                         cv2.COLOR_BGR2HSV).reshape(-1, 3).mean(axis=0)
    lower = np.clip(patch - CLICK_TOLERANCE, [0,60,60], [179,255,255]).astype(int)
    upper = np.clip(patch + CLICK_TOLERANCE, [0,60,60], [179,255,255]).astype(int)
    name = name_for_hue(patch[0])
    _push_color_change(color_change_q, name, lower, upper)
    print(f"🎯 Colour overridden to: {name}")
 
 
def wave_arm(bot, direction):
    if direction == "left":
        bot.move_left_arm(shoulder=90, elbow=30, wait=True)
        time.sleep(1)
        bot.move_left_arm(shoulder=0,  elbow=0,  wait=True)
    else:
        bot.move_right_arm(shoulder=90, elbow=30, wait=True)
        time.sleep(1)
        bot.move_right_arm(shoulder=0,  elbow=0,  wait=True)
 
 
class SharedState:
    """Thread-safe holder for the latest detection result the GUI loop reads.
 
    Mirrors camera.py's own pattern: the worker builds a brand-new object each
    cycle and swaps the reference in, so readers holding an old snapshot never
    see it mutated out from under them — no locking needed on the *contents*,
    only on the swap itself.
    """
    def __init__(self, color_name, lower, upper):
        self._lock = threading.Lock()
        self.blob = None
        self.smooth_cx = None
        self.smooth_cy = None
        self.confirmed = False
        self.color_name = color_name
        self.lower = lower
        self.upper = upper
        self.mask_thumb = None
        self.detect_fps = 0.0
 
    def snapshot(self):
        with self._lock:
            return dict(blob=self.blob, smooth_cx=self.smooth_cx, smooth_cy=self.smooth_cy,
                        confirmed=self.confirmed, color_name=self.color_name,
                        lower=self.lower, upper=self.upper,
                        mask_thumb=self.mask_thumb, detect_fps=self.detect_fps)
 
    def update(self, **kwargs):
        with self._lock:
            for k, v in kwargs.items():
                setattr(self, k, v)
 
 
class DetectorWorker(threading.Thread):
    """Runs the (downscaled) color-detection pipeline off the GUI thread,
    exactly like the RPi runs YOLO off the display path."""
 
    def __init__(self, bot, state, color_change_q, speaker, color_name, lower, upper):
        super().__init__(daemon=True)
        self.bot = bot
        self.state = state
        self.color_change_q = color_change_q
        self.speaker = speaker
        self._stop_evt = threading.Event()
 
        self.color_name = color_name
        self.lower = lower
        self.upper = upper
 
        self.hit_streak = self.miss_streak = 0
        self.is_tracking_confirmed = False
        self.last_hit_time = None
        self.smooth_cx = self.smooth_cy = None
        self.cx_history = collections.deque(maxlen=DIRECTION_HISTORY_LEN)
        self.cy_history = collections.deque(maxlen=DIRECTION_HISTORY_LEN)
        self.last_direction = None
        self.last_direction_ts = 0.0
        self.last_announced_ts_by_color = {}
 
        self._last_frame = None
        self._frame_counter = 0
        self._fps_ema = 0.0
        self._prev_t = time.time()
 
    def stop(self):
        self._stop_evt.set()
 
    def _reset_tracking(self):
        self.hit_streak = self.miss_streak = 0
        self.is_tracking_confirmed = False
        self.last_hit_time = None
        self.smooth_cx = self.smooth_cy = None
        self.cx_history.clear(); self.cy_history.clear()
        self.last_direction = None
 
    def run(self):
        while not self._stop_evt.is_set():
            try:
                name, lower, upper = self.color_change_q.get_nowait()
                self.color_name, self.lower, self.upper = name, lower, upper
                self._reset_tracking()
                print(f"🎨 → {self.color_name}")
            except queue.Empty:
                pass
 
            frame = self.bot.get_image()
            if frame is None or frame.shape[0] <= MASK_PREVIEW_H + 20 or frame.shape[1] <= 20:
                time.sleep(0.005)
                continue
 
            # Skip work entirely if no new frame has arrived yet (nothing changed)
            if frame is self._last_frame:
                time.sleep(0.005)
                continue
            self._last_frame = frame
 
            self._frame_counter += 1
            if self._frame_counter % DETECTION_FRAME_SKIP != 0:
                continue
 
            now = time.time()
            dt = now - self._prev_t
            self._prev_t = now
            if dt > 0:
                self._fps_ema = self._fps_ema * 0.9 + (1.0 / dt) * 0.1
 
            h, w = frame.shape[:2]
            scale = min(1.0, PROCESS_WIDTH / w)
            small = cv2.resize(frame, None, fx=scale, fy=scale,
                                interpolation=cv2.INTER_AREA) if scale < 1.0 else frame
 
            mask = apply_color_mask(small, self.lower, self.upper, self.color_name)
            blob = find_largest_blob(mask, MIN_BLOB_AREA * (scale ** 2))
 
            if blob is not None:
                inv = 1.0 / scale
                blob = {"cx": int(blob["cx"] * inv), "cy": int(blob["cy"] * inv),
                        "ex": int(blob["ex"] * inv), "ey": int(blob["ey"] * inv),
                        "radius": blob["radius"] * inv}
                self.hit_streak += 1; self.miss_streak = 0; self.last_hit_time = now
            else:
                self.miss_streak += 1; self.hit_streak = 0
 
            if not self.is_tracking_confirmed and self.hit_streak >= STABLE_FRAME_THRESHOLD:
                self.is_tracking_confirmed = True
                if now - self.last_announced_ts_by_color.get(self.color_name, 0.0) >= REANNOUNCE_COOLDOWN:
                    phrase = f"I have detected {self.color_name}!"
                    print(f"[{time.strftime('%H:%M:%S')}] 🎯 {phrase}")
                    self.speaker.speak(phrase)
                    self.last_announced_ts_by_color[self.color_name] = now
                self.smooth_cx = self.smooth_cy = None
                self.cx_history.clear(); self.cy_history.clear()
                self.last_direction = None
 
            elif self.is_tracking_confirmed and (
                self.miss_streak >= MISS_TOLERANCE
                or (self.last_hit_time is not None and now - self.last_hit_time >= LOCK_HOLD_TIMEOUT)
            ):
                self.is_tracking_confirmed = False
                print(f"[{time.strftime('%H:%M:%S')}] ⚪ Dropped")
 
            if blob is not None:
                if self.smooth_cx is None:
                    self.smooth_cx, self.smooth_cy = float(blob["cx"]), float(blob["cy"])
                else:
                    self.smooth_cx += SMOOTHING_ALPHA * (blob["cx"] - self.smooth_cx)
                    self.smooth_cy += SMOOTHING_ALPHA * (blob["cy"] - self.smooth_cy)
 
            if self.is_tracking_confirmed and blob is not None:
                self.cx_history.append(blob["cx"])
                self.cy_history.append(blob["cy"])
                if len(self.cx_history) == DIRECTION_HISTORY_LEN:
                    dx = self.cx_history[-1] - self.cx_history[0]
                    if abs(dx) >= DIRECTION_MIN_DELTA:
                        direction = "right" if dx > 0 else "left"
                        if direction != self.last_direction and now - self.last_direction_ts >= DIRECTION_COOLDOWN:
                            print(f"   ↳ moving {direction}")
                            self.speaker.speak(f"Moving {direction}!")
                            threading.Thread(target=wave_arm, args=(self.bot, direction), daemon=True).start()
                            self.last_direction = direction
                            self.last_direction_ts = now
            else:
                self.cx_history.clear(); self.cy_history.clear()
                self.last_direction = None
 
            pw = int(w * MASK_PREVIEW_H / h)
            thumb = cv2.resize(cv2.cvtColor(mask, cv2.COLOR_GRAY2BGR), (pw, MASK_PREVIEW_H),
                                interpolation=cv2.INTER_NEAREST)
            cv2.rectangle(thumb, (0, 0), (pw-1, MASK_PREVIEW_H-1), (200, 200, 200), 1)
 
            self.state.update(blob=blob, smooth_cx=self.smooth_cx, smooth_cy=self.smooth_cy,
                               confirmed=self.is_tracking_confirmed, color_name=self.color_name,
                               lower=self.lower, upper=self.upper, mask_thumb=thumb,
                               detect_fps=self._fps_ema)
 
 
speaker = VoiceSpeaker(min_gap_seconds=1.2)
 
with BonicBot(host='192.168.0.193') as bot:
    bot.system.start_camera()
    bot.start_camera()
    bot.camera.wait_for_image(timeout=5.0)
 
    color_change_q = queue.Queue(maxsize=1)
 
    cv2.namedWindow("BonicBot Vision - Color")
    cv2.setMouseCallback("BonicBot Vision - Color", on_mouse_click, color_change_q)
    print("✅ Live. Keys: 1-8 colours, q quit")
 
    state = SharedState(current_color_name, HSV_LOWER_DEFAULT.copy(), HSV_UPPER_DEFAULT.copy())
    worker = DetectorWorker(bot, state, color_change_q, speaker,
                             current_color_name, HSV_LOWER_DEFAULT.copy(), HSV_UPPER_DEFAULT.copy())
    worker.start()
 
    prev_time = time.time()
    display_fps = 0.0
 
    try:
        while True:
            frame = bot.get_image()
            if frame is not None and frame.shape[0] > MASK_PREVIEW_H + 20 and frame.shape[1] > 20:
                latest_frame = frame
 
                now = time.time()
                dt = now - prev_time
                prev_time = now
                if dt > 0:
                    display_fps = display_fps * 0.9 + (1.0 / dt) * 0.1
 
                snap = state.snapshot()
                display = draw_tracking(frame.copy(), snap["blob"], snap["smooth_cx"], snap["smooth_cy"])
                display = blit_mask_preview(display, snap["mask_thumb"])
                display = draw_hud(display, snap["confirmed"], snap["color_name"],
                                    snap["lower"], snap["upper"], snap["detect_fps"], display_fps)
                cv2.imshow("BonicBot Vision - Color", display)
 
            key = cv2.waitKey(1) & 0xFF
            if key == ord("q"):
                break
            elif key in COLOR_PRESETS:
                name, lower, upper = COLOR_PRESETS[key]
                _push_color_change(color_change_q, name, lower.copy(), upper.copy())
 
    finally:
        worker.stop()
        worker.join(timeout=1.0)
        cv2.destroyAllWindows()
        bot.stop_camera()
        bot.system.stop_camera()
        speaker.stop()

Replace [IP_ADDRESS] with the IP address of your own BonicBot. Every BonicBot may have a different IP address depending on your network configuration.

📌 Example:


with BonicBot(host="172.20.10.2") as bot:

Replace 172.20.10.2 with the IP address assigned to your BonicBot.

🔊To hear BonicBot’s voice announcements, connect a Bluetooth speaker to your BonicBot before running this program. If no Bluetooth speaker is connected, the program will still detect colors and track motion normally, but you won’t hear the spoken announcements.

Code Walkthrough

Line-by-line explanation

Imports — time, threading, queue support background speech; collections provides the deque used for direction history; cv2 and numpy do the actual color-detection math; BonicBot connects to the robot; pyttsx3 is imported inside a try/except so the script still runs without it.
VoiceSpeaker — Similar to earlier lessons’ TTS worker, but adds a minimum gap between announcements (min_gap_seconds): if speak() is called again too soon after the last one, the call is simply ignored, and any already-queued-but-unspoken message is drained before queuing the new one — so the robot doesn’t build up a backlog of stale phrases while tracking is jittery.
COLOR_PRESETS — A dictionary mapping keyboard key codes (ord("1")–ord("8")) to a color name and an HSV lower/upper bound np.array. Pressing that number key switches the active detection range.
current_color_name / current_lower / current_upper — Module-level variables holding whichever color is currently active, initialized to preset "1" (red) and updated either by a key press or by clicking a color in the video window.
name_for_hue(hue) — Given a raw hue value (0–179 in OpenCV’s HSV), looks up which named color band it falls into, used to label a color the user picked by clicking rather than a preset.
Tuning constants (MIN_BLOB_AREA, STABLE_FRAME_THRESHOLD, MISS_TOLERANCE, LOCK_HOLD_TIMEOUT, SMOOTHING_ALPHA, etc.) — Control how large a colored region must be to count, how many consecutive frames confirm a “lock,” how much dropout is tolerated before losing the lock, and how much the tracked position is smoothed frame to frame.
apply_color_mask(frame, lower_hsv, upper_hsv) — Converts the frame to HSV and blurs it slightly, then builds a black-and-white mask of pixels inside the target range. Red gets special handling: because red wraps around both ends of the 0–179 hue scale in HSV, the function combines two cv2.inRange calls (one near 0, one near 179) with cv2.bitwise_or. It finishes with erosion/dilation (to remove small noise and fill small gaps) and a median blur.
find_largest_blob(mask) — Finds all contours in the mask, picks the largest by area, discards it if too small (MIN_BLOB_AREA) or too irregular in shape (MIN_SOLIDITY, comparing blob area to the area of its enclosing circle), and returns its centroid (cx, cy) plus enclosing-circle center/radius.
draw_tracking(...) — Draws the yellow tracking circle around the detected blob and a crosshair at the smoothed position.
embed_mask_preview(...) — Shrinks the black-and-white mask down and pastes it into the bottom-right corner of the display frame, so you can see exactly what the color filter is picking up.
draw_hud(...) — The familiar stats-panel pattern, showing whether the color is “LOCKED” or still “SCANNING,” the active HSV range, and FPS.
on_mouse_click(...) — Registered via cv2.setMouseCallback. On a left-click, it samples a small patch of pixels around the click point from latest_frame, converts it to HSV, averages it, and builds a new current_lower/current_upper range around that average (± CLICK_TOLERANCE) — this is the “click to calibrate” feature.
wave_arm(bot, direction) — Raises the matching arm (left or right) briefly using bot.move_left_arm() / bot.move_right_arm(), then lowers it again; run in its own thread so it doesn’t block the main detection loop.
Lock/streak logic in the main loop (hit_streak, miss_streak, is_tracking_confirmed) — A color is only announced as “detected” once hit_streak reaches STABLE_FRAME_THRESHOLD, and the lock is only dropped after MISS_TOLERANCE consecutive misses or LOCK_HOLD_TIMEOUT seconds without a hit — the same debounce philosophy as earlier lessons, applied to a classical CV pipeline instead of a trained model’s output.
Position smoothing (smooth_cx, smooth_cy, SMOOTHING_ALPHA) — An exponential moving average applied to the blob’s raw centroid, so the drawn crosshair doesn’t jitter frame to frame the way the raw detection might.
Direction detection (cx_history, DIRECTION_HISTORY_LEN, DIRECTION_MIN_DELTA, DIRECTION_COOLDOWN) — Keeps a short history of recent x-positions; once the history is full, it compares the oldest and newest x-position. If the object has moved far enough (DIRECTION_MIN_DELTA) and the direction is new (or the cooldown has passed), it announces the direction and kicks off wave_arm in a background thread.
Key handling (1–8, q) — Number keys swap current_color_name/current_lower/current_upper to a different preset and reset all tracking state; q breaks the main loop.
finally: cleanup — Closes the window, stops the camera, and calls speaker.stop(), same shutdown pattern as previous lessons.

Expected Output

Click to see expected output

Visual Output:

📌 Note: This visual demonstration is using the ROS 2 simulation with use_real_camera:=True.

Terminal Output:


✅ Live. Keys: 1-8 colours, q quit

🎨 → red

[14:20:45] 🎯 I have detected red!

   ↳ moving right
Moving right!

Hotkeys available while the program is running:

1 – Red
2 – Orange
3 – Yellow
4 – Green
5 – Cyan
6 – Blue
7 – Purple
8 – Pink
q – Quit

The BonicBot Vision Window will display:

The live vision feed.
A yellow tracking circle around the detected object.
A crosshair showing the tracked position.
A small preview of the color mask.
The currently selected target color and lock status.
Current processing speed (FPS).

You can also:

Press 1–8 to switch between preset colors.
Click on any object in the Vision Window to automatically calibrate BonicBot to that color.

When BonicBot confirms a stable color match, it also waves the arm on the side matching the direction the object last moved.

🔧 Under the Hood

How does BonicBot detect colors?

Unlike previous lessons, this program does not rely on a trained deep learning model. Instead, it uses a classical Computer Vision technique based on the HSV color space — a rule-based method that has long been part of the Artificial Intelligence toolkit, prized for being fast, lightweight, and predictable, even without a trained model behind it.

The process works like this:


Vision Frame
│
▼
Convert to HSV
│
▼
Select Target Color
│
▼
Create Color Mask
│
▼
Remove Noise
│
▼
Find the Largest Blob
│
▼
Track Its Position
│
▼
Announce Detection
│
▼
Wave Arm Toward Movement

1. HSV Color Space

Instead of working directly with Red, Green, and Blue (RGB), the program converts every image into the HSV color space.

HSV separates:

Hue → the color itself
Saturation → how rich the color is
Value → how bright the color is

This makes color detection much more reliable under different lighting conditions.

2. Color Mask

The program keeps only pixels that fall inside the selected HSV range.

Everything else is removed.


Original Image

↓

Color Mask

↓

Only the chosen color remains

3. Noise Removal

Real images contain small unwanted colored regions.

Morphological operations such as erosion, dilation, and median filtering remove these tiny noisy areas, leaving only the main object.

4. Blob Detection

The program searches for the largest connected colored region.

This region is called a blob.

The center of this blob becomes the object’s tracked position.

5. Stable Detection

Instead of announcing a color immediately, BonicBot waits until the object has been detected consistently for several consecutive frames.

This prevents false detections caused by temporary lighting changes or image noise.

The robot also waits before repeating the same announcement, making the interaction much more natural.

6. Gesture Feedback

Once BonicBot is confidently tracking an object, it doesn’t just speak — it also gestures. When the object moves clearly enough to the left or right, BonicBot raises the matching arm briefly, giving a physical acknowledgment alongside the voice announcement.

Student Challenge

Teach BonicBot to detect your own favorite object.

Instead of using one of the preset colors:

Hold a colorful object in front of BonicBot.
Click on the object in the Vision Window.
Observe how BonicBot automatically learns the new color.
Move the object around and watch BonicBot track it.

Try testing objects with different shades of the same color.

Hint

The mouse click automatically samples the color underneath your cursor and creates a new HSV range.

No changes to the Python code are required.

Simply click on the object you want BonicBot to track.

Reflection Question

This lesson uses color detection, a classical Computer Vision technique, instead of a trained AI object recognition model.

Can you think of situations where tracking a specific color might be simpler and faster than using a trained AI object detection model?