Lesson 10: Controlling BonicBot with QR Codes

If you already installed bonicbot-bridge, opencv-python, and pyttsx3 in earlier lessons, you don’t need to reinstall anything for this lesson.

If pip install fails, try pip3 install ... instead, or use a virtual environment:

python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python pyttsx3

1. Find your BonicBot’s IP address — via the robot’s on-device display, its companion app, or your router’s connected-devices list.
2. Generate your QR codes first. Create four Text QR codes containing exactly TURN LEFT, TURN RIGHT, LOOK LEFT, and LOOK RIGHT, and have them ready on your phone screen or printed out.
3. Save the code below into a file, e.g. lesson10_qr_control.py.
4. Replace HOST = "localhost" with your BonicBot’s actual IP address, e.g. HOST = "172.20.10.2".
5. Make sure your BonicBot is powered on and network-connected, and that it has clear open space around it to safely turn left/right without hitting anything.
6. (Optional) Connect a Bluetooth speaker to hear the spoken announcements. QR detection still works without one.
7. Run the script:

   python lesson10_qr_control.py

8. A window titled “BonicBot QR Command” should open, showing the live feed with a green box around any detected QR code and a HUD showing the robot’s state and FPS.
9. Press q with the video window focused to stop the stream and exit cleanly.

This lesson’s code uses HOST = "localhost" by default, so it can run directly against the ROS 2 simulation environment without any code changes as long as you leave HOST unmodified:

1. Launch the BonicBot simulation:

   ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True

• use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion (turning, looking) are still simulated. This is the easiest way to test detection, since you can just hold a printed or on-screen QR code up to your laptop camera.
• use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).

2. With the simulation running, simply run the script as written, leaving HOST = "localhost" unchanged.
3. Everything else in the code — QR detection, command confirmation, and the turn/look actions — works the same way, since the simulation exposes the same interface as a real robot.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, replacing HOST with its actual IP address is the recommended way to go through this lesson on real hardware.

import time, queue, threading
import cv2
from bonicbot_bridge import BonicBot, BonicBotError
 
try:
    import pyttsx3
    _TTS_AVAILABLE = True
except ImportError:
    _TTS_AVAILABLE = False
 
 
class VoiceSpeaker:
    """Background TTS worker so announcements don't block the camera loop."""
    def __init__(self, rate=175, volume=0.6):
        self._q = queue.Queue()
        self._rate, self._volume = rate, volume
        if _TTS_AVAILABLE:
            threading.Thread(target=self._run, daemon=True).start()
        else:
            print("⚠️ pyttsx3 not installed, voice output disabled (pip install pyttsx3)")
 
    def _run(self):
        while (text := self._q.get()) is not None:
            try:
                engine = pyttsx3.init()
                engine.setProperty('rate', self._rate)
                engine.setProperty('volume', self._volume)
                engine.say(text)
                engine.runAndWait()
                engine.stop()
            except Exception as e:
                print(f"⚠️ TTS error: {e}")
 
    def speak(self, text):
        if _TTS_AVAILABLE:
            self._q.put(text)
 
    def stop(self):
        if _TTS_AVAILABLE:
            self._q.put(None)
 
 
# --- Commands ---
# Print a QR code containing exactly one of these strings and show it to the robot.
TURN_SPEED, TURN_DURATION = 30, 3.0   # rad/s, seconds
POSE_SETTLE = 3                       # pause after each action for smooth motion
 
 
def do_turn_left(bot):
    bot.turn_left(speed=TURN_SPEED, duration=TURN_DURATION)
    time.sleep(TURN_DURATION + 0.2)
    bot.turn_right(speed=TURN_SPEED, duration=TURN_DURATION)  # return to initial pose
    time.sleep(TURN_DURATION + POSE_SETTLE)
 
 
def do_turn_right(bot):
    bot.turn_right(speed=TURN_SPEED, duration=TURN_DURATION)
    time.sleep(TURN_DURATION + 0.2)
    bot.turn_left(speed=TURN_SPEED, duration=TURN_DURATION)   # return to initial pose
    time.sleep(TURN_DURATION + POSE_SETTLE)
 
 
def do_look_left(bot):
    bot.look_left()
    time.sleep(POSE_SETTLE)
    bot.look_center()
    time.sleep(POSE_SETTLE)
 
 
def do_look_right(bot):
    bot.look_right()
    time.sleep(POSE_SETTLE)
    bot.look_center()
    time.sleep(POSE_SETTLE)
 
 
COMMANDS = {
    "TURN LEFT": do_turn_left,
    "TURN RIGHT": do_turn_right,
    "LOOK LEFT": do_look_left,
    "LOOK RIGHT": do_look_right,
}
 
 
def process_qr(frame, detector):
    """Detect & decode QR codes, draw boxes/text, return (annotated_frame, detections)."""
    display = frame.copy()
    detections = []
    ok, texts, points, _ = detector.detectAndDecodeMulti(frame)
    if ok and points is not None:
        for text, box in zip(texts, points):
            if not text:
                continue
            box = box.astype(int)
            cv2.polylines(display, [box], True, (0, 255, 0), 2)
            cv2.putText(display, text, tuple(box[0]), cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
            detections.append(text.strip().upper())
    return display, detections
 
 
def draw_hud(frame, detections, state, fps):
    overlay = frame.copy()
    cv2.rectangle(overlay, (10, 10), (310, 98), (0, 0, 0), -1)
    frame = cv2.addWeighted(overlay, 0.55, frame, 0.45, 0)
    cv2.putText(frame, f"State: {state}", (22, 36), cv2.FONT_HERSHEY_SIMPLEX, 0.58, (0, 255, 255), 2)
    cv2.putText(frame, f"QR seen: {detections[0] if detections else '-'}", (22, 60),
                cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 200, 255), 2)
    cv2.putText(frame, f"FPS: {fps:.1f}", (22, 80), cv2.FONT_HERSHEY_SIMPLEX, 0.48, (180, 180, 180), 1)
    return frame
 
 
# --- Setup ---
HOST = "localhost"          # <-- replace with your BonicBot's IP address
CONFIRM_FRAMES = 3          # consecutive consistent reads required before acting
COMMAND_COOLDOWN = 6.0      # seconds to ignore repeat triggers after a move ends
 
speaker = VoiceSpeaker()
print("🔎 Setting up QR detector (built into OpenCV, no model to load)...")
qr_detector = cv2.QRCodeDetector()
bot = None
 
# Background command execution — keeps the camera feed live and responsive
# while the robot is mid-turn/mid-look instead of freezing the display.
command_busy = threading.Event()
 
 
def run_command(bot, name):
    """Runs a command's blocking motion/sleep sequence on its own thread."""
    try:
        COMMANDS[name](bot)
        speaker.speak("Done!")
    except BonicBotError as e:
        print(f"⚠️ Movement failed: {e}")
        bot.stop()
    finally:
        command_busy.clear()
 
 
try:
    with BonicBot(host=HOST, port=9090, timeout=10) as bot:
        print("📷 Starting BonicBot camera and streaming...")
        bot.system.start_camera()
        bot.start_camera()
        bot.camera.wait_for_image(timeout=5.0)
        print("✅ Camera running. Press 'q' in the window to stop.\n")
 
        prev_time, fps = time.time(), 0.0
        state = "IDLE"
        confirm_counter, last_seen_text = 0, None
        last_command_text, last_command_time = None, 0.0
 
        while True:
            frame = bot.camera.get_latest_image()
            if frame is None:
                time.sleep(0.01)
                continue
 
            now = time.time()
            fps = fps * 0.9 + (1.0 / max(now - prev_time, 1e-6)) * 0.1
            prev_time = now
 
            display, detections = process_qr(frame, qr_detector)
            nearest = detections[0] if detections else None
 
            # A command is running in the background — keep displaying
            # frames, but don't evaluate new triggers until it's done.
            if command_busy.is_set():
                confirm_counter, last_seen_text = 0, None
                state = "MOVING"
 
            elif nearest and nearest in COMMANDS:
                confirm_counter = confirm_counter + 1 if nearest == last_seen_text else 1
                last_seen_text = nearest
 
                on_cooldown = nearest == last_command_text and now - last_command_time < COMMAND_COOLDOWN
 
                if confirm_counter >= CONFIRM_FRAMES and not on_cooldown:
                    state = "MOVING"
                    bot.move(0.0, 0.0, 0.0)
                    speaker.speak("Command detected!")
                    print(f"[{time.strftime('%H:%M:%S')}] 🤖 Executing command: {nearest}")
 
                    command_busy.set()
                    threading.Thread(target=run_command, args=(bot, nearest), daemon=True).start()
 
                    last_command_text, last_command_time = nearest, time.time()
                    confirm_counter, last_seen_text = 0, None
            else:
                confirm_counter, last_seen_text = 0, None
                if state == "IDLE":
                    bot.move(0.0, 0.0, 0.0)
 
            # Command finished on the background thread — return to IDLE
            if state == "MOVING" and not command_busy.is_set():
                state = "IDLE"
 
            display = draw_hud(display, detections, state, fps)
            cv2.imshow("BonicBot QR Command", display)
 
            if cv2.waitKey(1) & 0xFF == ord("q"):
                print("Stopping script...")
                break
 
except BonicBotError as e:
    print(f"Robot error: {e}")
finally:
    cv2.destroyAllWindows()
    if bot is not None:
        try:
            bot.move(0.0, 0.0, 0.0)
            bot.stop()
            bot.system.stop_camera()
        except Exception:
            pass
    speaker.stop()
    print("Camera shutdown and window closed.")

Replace HOST = "localhost" with the IP address of your own BonicBot.

📌 Example:

HOST = "172.20.10.2"

Replace 172.20.10.2 with the IP address assigned to your BonicBot.

🔊 To hear BonicBot’s voice announcements, connect a Bluetooth speaker to your BonicBot before running this program. If no Bluetooth speaker is connected, the program will still detect QR codes normally, but you won’t hear the spoken announcements.

• Imports — time handles timing and sleeps; queue and threading power the background TTS worker and the background command-execution thread; cv2 runs QR detection and drawing; BonicBot / BonicBotError manage the robot connection.
• pyttsx3 import block — Wrapped in try/except ImportError to set _TTS_AVAILABLE, the same optional-TTS pattern used in earlier lessons.
• VoiceSpeaker — The familiar background-thread TTS worker: speak() pushes text onto a queue.Queue, _run() consumes it on a daemon thread and speaks each item with pyttsx3, and stop() pushes None as a sentinel to end the loop. If _TTS_AVAILABLE is False, it prints a one-time warning instead of starting the thread.
• TURN_SPEED, TURN_DURATION, POSE_SETTLE — Tuning constants: how fast (rad/s) and how long BonicBot turns for TURN LEFT/TURN RIGHT commands, and how long it pauses after each action to let motion settle smoothly.
• do_turn_left(bot) / do_turn_right(bot) — Each turns the robot in one direction for TURN_DURATION, sleeps, then turns back the opposite direction for the same duration to return to the original pose, followed by a POSE_SETTLE pause.
• do_look_left(bot) / do_look_right(bot) — Each moves the robot’s head/camera to look in one direction with bot.look_left() / bot.look_right(), pauses for POSE_SETTLE, then calls bot.look_center() to return to center, pausing again.
• COMMANDS — A dictionary mapping each recognized QR text string ("TURN LEFT", "TURN RIGHT", "LOOK LEFT", "LOOK RIGHT") to its corresponding function, used to look up and call the right action once a command is confirmed.
• process_qr(frame, detector) — Runs detector.detectAndDecodeMulti(frame) (OpenCV’s built-in multi-QR detector) on the current frame. For each decoded QR code with non-empty text, it draws a green polygon around it (cv2.polylines) using the returned corner points, labels it with the decoded text (cv2.putText), and appends the uppercased, stripped text to a detections list. Returns the annotated frame and that list.
• draw_hud(frame, detections, state, fps) — Draws the familiar semi-transparent black panel and overlays the current state (“IDLE” or “MOVING”), the first detected QR text (or "-" if none), and the current fps.
• HOST, CONFIRM_FRAMES, COMMAND_COOLDOWN — Connection target plus two stability constants: how many consecutive frames must show the same QR text before a command is executed, and how long (in seconds) that same command is ignored again after it just finished running, to prevent immediate re-triggering while the QR code is still in view.
• qr_detector = cv2.QRCodeDetector() — Created once at startup; unlike the YOLO/ArcFace lessons, no model file is loaded — QR decoding is a built-in OpenCV algorithm.
• command_busy (threading.Event) — Signals whether a command is currently executing on a background thread, so the main loop knows not to evaluate or trigger new commands while one is still in progress.
• run_command(bot, name) — Runs on its own thread once a command is confirmed. Looks up and calls the matching function from COMMANDS, speaks “Done!” on success, catches BonicBotError to call bot.stop() and print a warning if the movement fails, and always clears command_busy in a finally block so the main loop can accept new commands again.
• Main loop — Inside with BonicBot(host=HOST, port=9090, timeout=10) as bot:, after starting the camera, each iteration:
  • Pulls the latest frame with bot.camera.get_latest_image(), skipping briefly if none is available yet.
  • Updates a smoothed fps value the same way as earlier lessons.
  • Calls process_qr() to get the annotated display frame and any detections, taking the first detected text as nearest.
  • If command_busy.is_set() (a command is still running), it resets the confirmation counter and forces state = "MOVING", ignoring any new QR reads.
  • Otherwise, if nearest is a recognized command, it increments confirm_counter only if the same text was seen last frame (resetting to 1 otherwise), and checks on_cooldown to see if this exact command just ran recently. Once confirm_counter >= CONFIRM_FRAMES and the command isn’t on cooldown, it stops the robot’s base motion (bot.move(0.0, 0.0, 0.0)), speaks “Command detected!”, sets command_busy, and launches run_command on a new daemon thread — keeping the video feed responsive while the action plays out.
  • If no recognized command is seen, it resets the confirmation counter and, if idle, keeps calling bot.move(0.0, 0.0, 0.0) to hold the robot still.
  • Once the background thread clears command_busy, the state resets from "MOVING" back to "IDLE".
  • Draws the HUD and shows the frame in the "BonicBot QR Command" window, checking cv2.waitKey(1) for the q key to break the loop.
• Cleanup (finally) — Closes the OpenCV window, and if bot was successfully created, wraps a final bot.move(0.0, 0.0, 0.0), bot.stop(), and bot.system.stop_camera() in a try/except Exception: pass so shutdown never crashes even if the robot connection was already lost, then calls speaker.stop() and prints a final shutdown message.

Visual Output:

Terminal Output:

🔎 Setting up QR detector (built into OpenCV, no model to load)...
📷 Starting BonicBot camera and streaming...
✅ Camera running. Press 'q' in the window to stop.

The BonicBot Vision Window will display:

• A live view from BonicBot’s camera.
• A green box around detected QR codes.
• The decoded QR command.
• The robot’s current state and FPS.

When BonicBot reads a valid command, it says “Command detected!”, performs the corresponding action, and says “Done!” when the action is complete.

QR codes contain distinctive square patterns that help a computer locate and orient them.

The program uses OpenCV’s QRCodeDetector to find the QR code and decode the text stored inside it.

For example:

QR Code
   │
   ▼
QRCodeDetector
   │
   ▼
"TURN LEFT"
   │
   ▼
do_turn_left()
   │
   ▼
BonicBot Turns Left

The decoded text is checked against the COMMANDS dictionary. If a matching command exists, BonicBot runs the corresponding function.

The program also waits until the same QR code has been detected for 3 consecutive frames before acting. This prevents a single unstable camera reading from accidentally triggering an action.

While BonicBot is performing an action, it enters the MOVING state and ignores new commands until the current action is finished.

Create a new function:

def do_spin(bot):
    bot.turn_left(speed=0.8, duration=7.85)

Then add it to the command dictionary:

COMMANDS["SPIN"] = do_spin