Lesson 5: Teaching BonicBot to Detect ArUco Markers

Learning Objective

Teach BonicBot to detect ArUco markers, identify each marker by its unique ID, and respond whenever a marker is reliably recognized.

Introduction

So far, BonicBot has learned to detect objects, faces, body joints, and hand gestures. In this lesson, we’ll introduce another powerful vision tool used extensively in robotics: ArUco markers.

An ArUco marker is a specially designed black-and-white square pattern that contains a unique identification number. Unlike ordinary images, these markers are designed so robots can recognize them quickly and reliably.

BonicBot uses its built-in ArUco detection system to continuously look for markers in its surroundings. Whenever a marker appears, the robot reports its ID and displays it in the Vision Window.

Since temporary detection errors can occur, the program waits until the same marker has been detected for several consecutive frames before announcing it. This prevents BonicBot from repeatedly announcing markers that briefly appear due to noise or partial visibility.

Once a marker has been confirmed, BonicBot announces its ID using speech while updating the Vision Window with the detected markers.

Setup: Installing Packages

Before running the code, make sure your computer has the required Python packages installed. Open a terminal and run:


pip install bonicbot-bridge opencv-python numpy pyttsx3

What each package does

Package	Purpose
`bonicbot-bridge`	The official BonicBot SDK. Provides the `BonicBot` class used to connect to the robot, control its camera, enable ArUco detection, and read marker IDs back via `bot.get_aruco_markers()`. Also provides `BonicBotError` used for safe error handling.
`opencv-python`	Imported as `cv2`. Used to draw the marker ID chips and stats panel, and to display the live video window.
`numpy`	Backs the image arrays that both `opencv-python` and `bonicbot-bridge` work with internally.
`pyttsx3`	A text-to-speech library used by the `VoiceSpeaker` class to make BonicBot announce each confirmed marker ID out loud. This is optional — if it isn’t installed, the script still detects markers and prints/displays them normally, it just skips the spoken announcement.

If you already installed bonicbot-bridge, opencv-python, and numpy in an earlier lesson, you only need to add pyttsx3 for this one.

If pip install fails, try pip3 install ... instead, or use a virtual environment:


python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy pyttsx3

How to run the program

Find your BonicBot’s IP address. Check the robot’s on-device display, its companion app, or your router’s connected-devices list.
Save the code below into a file, e.g. lesson5_aruco.py.
Replace [IP_ADDRESS] in the code with your robot’s actual IP address (see the callout box under the code block).
(Optional) Connect a Bluetooth speaker to your BonicBot if you want to hear the spoken marker announcements. Detection still works without one — you just won’t hear the voice output.
Have one or more printed ArUco markers ready to hold in front of the camera.
Run the script:
```
python lesson5_aruco.py
```
A window titled “BonicBot Vision” should open, showing colored ID chips for currently visible markers and a small stats panel in the top-left corner.
Watch the terminal — a new line is only printed once a marker ID has been confirmed as stable.
Press q with the video window focused to stop the stream and exit cleanly.

Don’t have a physical BonicBot? Try it in simulation (optional)

If you don’t have physical access to a BonicBot, you can still work through this lesson using the ROS 2 simulation environment instead of a real robot:

Launch the BonicBot simulation:
```
ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True
```
- use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion are still simulated. This is the easiest way to test detection, since you can just hold whatever you want detected up to your laptop camera — in this case, a printed ArUco marker.
- use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).
Once the simulation is running, use localhost as the host instead of a physical IP address:
```
with BonicBot(host="localhost", port=9090, timeout=10) as bot:
```
Everything else in the code — enabling ArUco detection, reading bot.get_aruco_markers(), and the voice announcements — works the same way, since the simulation exposes the same interface as a real robot.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, connecting to its actual IP address is the recommended way to go through this lesson.

Code

Click to view the complete program


import time
import cv2
import numpy as np
import threading
import queue
from bonicbot_bridge import BonicBot, BonicBotError
 
try:
    import pyttsx3
    _TTS_AVAILABLE = True
except ImportError:
    _TTS_AVAILABLE = False
 
 
class VoiceSpeaker:
    """Background TTS worker so marker announcements happen without
    blocking the camera loop. Re-inits pyttsx3 per utterance —
    reusing one engine across multiple say()/runAndWait() calls goes
    silent after the first call on Linux/espeak, so a fresh engine per
    message avoids that bug."""
 
    def __init__(self, rate=175, volume=0.3):
        self._queue = queue.Queue()
        self._rate = rate
        self._volume = volume
        self._thread = None
        if _TTS_AVAILABLE:
            self._thread = threading.Thread(target=self._run, daemon=True)
            self._thread.start()
        else:
            print("⚠️ pyttsx3 not installed, voice output disabled (pip install pyttsx3)")
 
    def _run(self):
        while True:
            text = self._queue.get()
            if text is None:
                break
            try:
                engine = pyttsx3.init()
                engine.setProperty('rate', self._rate)
                engine.setProperty('volume', self._volume)
                engine.say(text)
                engine.runAndWait()
                engine.stop()
                del engine
            except Exception as e:
                print(f"⚠️ TTS playback error: {e}")
 
    def speak(self, text):
        if _TTS_AVAILABLE:
            self._queue.put(text)
 
    def stop(self):
        if _TTS_AVAILABLE:
            self._queue.put(None)
            self._thread.join(timeout=1)
 
 
# --- ArUco detection now runs onboard the robot via bot.enable_detection('aruco') ---
# The onboard vision pipeline publishes detected marker IDs over rosbridge and
# bot.get_aruco_markers() just returns the latest list[int] of IDs — no corner
# points are provided by this API, so we can no longer draw per-marker outlines
# the way the old local-OpenCV version could. The marker dictionary/type is
# whatever the onboard pipeline is configured with, not something this script
# controls.
 
# How many CONSECUTIVE polls a given marker ID must be seen before it's
# treated as "stable" and announced — debounces flicker/false detections.
TARGET_HOLD_FRAMES = 8
 
MARKER_COLOR = (0, 255, 0)   # detected but not yet announced (or not currently tracked)
TARGET_COLOR = (0, 0, 255)   # marker(s) that have been announced this appearance
FLASH_DURATION = 12           # frames the border pulse stays visible after a new announcement
 
 
class MarkerTracker:
    """Tracks each marker ID's visibility streak and whether it has already
    been announced during its current appearance. Tolerates brief single/
    double-frame dropouts (common with noisy false-positive flicker or a
    momentary occlusion) without resetting — an ID is only fully reset
    after missing for MISS_TOLERANCE_FRAMES consecutive frames."""
 
    MISS_TOLERANCE_FRAMES = 5  # allow up to N missed frames before resetting
 
    def __init__(self, hold_frames):
        self._hold_frames = hold_frames
        self._state = {}  # marker_id -> {"streak": int, "announced": bool, "missed": int}
 
    def update(self, marker_ids):
        current_ids = set(marker_ids)
 
        for marker_id in list(self._state.keys()):
            if marker_id not in current_ids:
                entry = self._state[marker_id]
                entry["missed"] += 1
                if entry["missed"] > self.MISS_TOLERANCE_FRAMES:
                    del self._state[marker_id]  # truly gone -> full reset
 
        newly_stable = []
        for marker_id in current_ids:
            entry = self._state.get(marker_id)
            if entry is None:
                entry = {"streak": 1, "announced": False, "missed": 0}
                self._state[marker_id] = entry
            else:
                entry["missed"] = 0  # seen again, clear the miss counter
                entry["streak"] += 1
 
            if entry["streak"] == self._hold_frames and not entry["announced"]:
                entry["announced"] = True
                newly_stable.append(marker_id)
 
        return newly_stable
 
    def announced_ids(self):
        return {mid for mid, entry in self._state.items() if entry["announced"]}
 
 
def draw_marker_chips(frame, marker_ids, announced_ids, top_left=(22, 96)):
    """Draw a small colored chip per currently-detected marker ID since the
    onboard detector only gives us IDs, not corner points to draw outlines
    around. Green = seen but not yet announced, red = already announced
    this appearance."""
    x, y = top_left
    chip_w, chip_h, gap = 46, 26, 8
    for marker_id in marker_ids:
        color = TARGET_COLOR if marker_id in announced_ids else MARKER_COLOR
        cv2.rectangle(frame, (x, y), (x + chip_w, y + chip_h), color, -1)
        cv2.putText(frame, str(marker_id), (x + 6, y + 19),
                    cv2.FONT_HERSHEY_SIMPLEX, 0.55, (0, 0, 0), 2)
        x += chip_w + gap
    return frame
 
 
def draw_hud(frame, marker_ids, action_label, fps, flash_frames_left, announced_ids):
    """Translucent stats panel + marker ID chips + a brief border flash
    whenever a new marker ID gets announced."""
    h_px, w_px = frame.shape[:2]
    panel_w, panel_h = 320, 132 if marker_ids else 84
    overlay = frame.copy()
    cv2.rectangle(overlay, (10, 10), (10 + panel_w, 10 + panel_h), (0, 0, 0), -1)
    frame = cv2.addWeighted(overlay, 0.55, frame, 0.45, 0)
 
    markers_status = f"Markers: {marker_ids}" if marker_ids else "Markers: none"
    color = (0, 255, 0) if marker_ids else (0, 0, 255)
    cv2.putText(frame, markers_status, (22, 36), cv2.FONT_HERSHEY_SIMPLEX, 0.58, color, 2)
    cv2.putText(frame, f"Last: {action_label}", (22, 60),
                cv2.FONT_HERSHEY_SIMPLEX, 0.56, (0, 200, 255), 2)
    cv2.putText(frame, f"FPS: {fps:.1f}", (22, 82),
                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (180, 180, 180), 1)
 
    if marker_ids:
        frame = draw_marker_chips(frame, marker_ids, announced_ids)
 
    if flash_frames_left > 0:
        cv2.rectangle(frame, (2, 2), (w_px - 3, h_px - 3), (0, 0, 255), 4)
 
    return frame
 
 
speaker = VoiceSpeaker()
tracker = MarkerTracker(TARGET_HOLD_FRAMES)
 
with BonicBot(host='[IP_ADDRESS]', port=9090, timeout=10) as bot:
    print("📷 Starting camera and streaming...")
    bot.system.start_camera()
    bot.start_camera()
    bot.camera.wait_for_image(timeout=5.0)
 
    print("🔎 Enabling onboard ArUco detection...")
    try:
        bot.enable_detection('aruco')
    except BonicBotError as e:
        print(f"⚠️ Could not enable onboard ArUco detection: {e}")
 
    print("✅ Live Stream Active. Press 'q' to quit.\n")
 
    prev_time = time.time()
    fps = 0.0
    action_label = "-"
    flash_frames_left = 0
 
    try:
        while True:
            frame = bot.get_image()
            if frame is not None:
                marker_ids = bot.get_aruco_markers()
 
                now = time.time()
                dt = now - prev_time
                prev_time = now
                if dt > 0:
                    fps = fps * 0.9 + (1.0 / dt) * 0.1
 
                newly_stable = tracker.update(marker_ids)
 
                for marker_id in newly_stable:
                    ts = time.strftime("%H:%M:%S")
                    message = f"Marker {marker_id} detected"
                    action_label = message
                    speaker.speak(message)
                    flash_frames_left = FLASH_DURATION
                    print(f"[{ts}] 🟢 {message}")
 
                if flash_frames_left > 0:
                    flash_frames_left -= 1
 
                display = draw_hud(frame.copy(), marker_ids, action_label, fps,
                                    flash_frames_left, tracker.announced_ids())
                cv2.imshow("BonicBot Vision", display)
 
            if cv2.waitKey(1) & 0xFF == ord("q"):
                break
    finally:
        cv2.destroyAllWindows()
        try:
            bot.disable_detection()
        except BonicBotError as e:
            print(f"⚠️ Error disabling detection: {e}")
        bot.stop_camera()
        bot.system.stop_camera()
        speaker.stop()

Replace [IP_ADDRESS] with the IP address of your own BonicBot. Every BonicBot may have a different IP address depending on your network configuration.

📌 Example:


with BonicBot(host="172.20.10.2") as bot:

Replace 172.20.10.2 with the IP address assigned to your BonicBot.

🔊To hear BonicBot’s voice announcements, connect a Bluetooth speaker to your BonicBot before running this program. If no Bluetooth speaker is connected, the program will still detect ArUco markers normally, but you won’t hear the spoken announcements.

Code Walkthrough

Line-by-line explanation

Imports — time and cv2 are used as in earlier lessons; threading and queue support running text-to-speech in the background; BonicBot and BonicBotError connect to the robot and let the script catch robot-specific errors; pyttsx3 is imported inside a try/except so the whole script still runs (just without voice) if it isn’t installed.
_TTS_AVAILABLE — A flag set based on whether the pyttsx3 import succeeded, checked everywhere voice output is used so the rest of the script doesn’t need repeated try/except blocks.
VoiceSpeaker class — Runs text-to-speech on a background thread so speaking a message doesn’t pause the camera loop. Its _run() worker pulls messages off a queue one at a time and creates a brand-new pyttsx3 engine for every message — the comment explains this works around a real bug where reusing one engine across multiple calls goes silent after the first utterance on Linux/espeak. speak(text) just queues a message (no-op if TTS isn’t available), and stop() sends a None sentinel to cleanly end the background thread.
TARGET_HOLD_FRAMES — How many consecutive times a marker ID must be seen before it’s treated as “stable” and announced — the same debouncing idea used in earlier lessons.
MARKER_COLOR / TARGET_COLOR / FLASH_DURATION — Drawing colors for markers not yet announced vs. already announced, and how long the border-flash effect lasts.
MarkerTracker class — Keeps a small state dictionary per marker ID with a visibility streak, whether it’s already been announced, and how many frames it’s recently been missed. update(marker_ids) is called every frame: it increments the miss counter for any ID no longer visible (fully forgetting it only after MISS_TOLERANCE_FRAMES consecutive misses), grows the streak for IDs still visible, and returns any IDs that just crossed the hold_frames threshold for the first time. announced_ids() reports which markers are currently in the “already announced” state, used for chip coloring.
draw_marker_chips(...) — Draws one small colored rectangle (“chip”) per currently visible marker ID with its number on it, since the onboard detector only returns IDs (not corner points), so outlining the physical marker in the frame isn’t possible here.
draw_hud(...) — The same translucent stats-panel pattern as earlier lessons, now showing the list of visible marker IDs, the last announced message, and FPS, plus the marker chips and a border flash.
speaker = VoiceSpeaker() / tracker = MarkerTracker(TARGET_HOLD_FRAMES) — Created once, outside the with BonicBot(...) block, so they exist for the whole program and persist state across every frame of the loop.
BonicBot(host='[IP_ADDRESS]', port=9090, timeout=10) — Same connection pattern as earlier lessons, but this lesson explicitly sets the rosbridge port and a connection timeout, since ArUco detection runs onboard the robot and communicates over rosbridge.
bot.enable_detection('aruco') wrapped in try/except BonicBotError — Switches on the onboard ArUco pipeline, but catches the error if it fails (e.g. the robot doesn’t support it or is busy) so the script can report a warning instead of crashing outright.
bot.get_aruco_markers() — Returns the current list of visible marker IDs (plain integers, no corner geometry) for this frame.
Main loop — Grabs a frame, gets the current marker IDs, updates FPS, calls tracker.update(...) to find any markers that just became stable, and for each one: builds an announcement message, speaks it, sets the flash timer, and prints a log line. It then draws the HUD and marker chips and shows the window.
finally: cleanup — Closes the display window, disables detection (catching BonicBotError in case it’s already off), stops the camera, and calls speaker.stop() to shut down the background TTS thread cleanly.

Expected Output

Click to see expected output

Visual Output:

Terminal Output:


📷 Starting vision system...
🧠 Enabling onboard ArUco detection...
✅ Vision system ready.

[14:45:18] 🟢 Marker 23 detected
[14:45:27] 🟢 Marker 7 detected

Whenever BonicBot detects an ArUco marker, it will announce:

“Marker 23 detected.”

The BonicBot Vision Window will display:

The IDs of all currently visible markers
A live marker count
The most recently announced marker
Current processing speed (FPS)
A brief border flash whenever a new marker is confirmed

Each detected marker appears as a colored ID chip:

🟢 Green — Marker detected but not yet announced
🔴 Red — Marker has already been announced during its current appearance

🔧 Under the Hood

How does BonicBot detect ArUco markers?

Unlike object detection, which recognizes general categories such as people or bottles, ArUco detection identifies artificial visual markers that each contain a unique numerical ID.

Each marker is a square pattern designed so that its internal black-and-white arrangement represents a specific identifier.

In this lesson, BonicBot performs the marker detection onboard. Your Python program simply requests the latest list of detected marker IDs using:


bot.get_aruco_markers()

The robot continuously monitors which markers are currently visible.

To make detection more reliable, the program keeps track of how long each marker has remained visible. A marker is only considered stable after being detected for several consecutive frames.

This technique prevents repeated announcements caused by brief tracking errors or momentary occlusions.

Once a marker becomes stable, BonicBot:

announces the marker ID using speech
updates the Vision Window
remembers that the marker has already been announced until it disappears

This approach is commonly used in robotics to make perception systems more reliable and less sensitive to noisy detections.

Student Challenge

Modify the program so BonicBot performs a different action for different marker IDs.

For example:

Marker 1 → Say “Welcome!”
Marker 2 → Turn left
Marker 3 → Turn right
Marker 4 → Move forward

Try designing your own interactive marker-based activity using BonicBot.

Hint

Inside the loop that processes newly detected markers, check the marker ID before deciding what action BonicBot should perform.

For example:


if marker_id == 1:
    # Your action here
elif marker_id == 2:
    # Another action

Reflection Question

Why might a robotics application prefer using ArUco markers instead of relying only on object detection when it needs to identify specific locations or objects?