Lesson 15: Giving BonicBot Eyes That Understand

Learning Objective

Discover a different, more modern way for BonicBot to understand a scene: instead of stitching together a separate object detector and a separate language model, BonicBot sends the actual picture straight to a single AI model that can both see and talk — and compare how that changes what BonicBot notices.

Introduction

In the last lesson, BonicBot described what it saw using a two-step process: an object detector picked out a label like "bottle", and then a language model — which never actually saw the picture, only that one word — wrote a sentence about it. That works, but it has a real limitation: the language model is describing a word, not the scene. It has no idea how many bottles there are, what color they are, what’s next to them, or what’s actually happening in the room.

In this lesson, BonicBot uses a Vision-Language Model, or VLM, instead. A VLM is a single AI model trained to understand images and text together — you can literally hand it a photograph and it describes what’s in it based on the actual pixels, not a pre-computed label. We’ll use a small, efficient image-captioning VLM hosted on Hugging Face (ovi054/image-to-prompt), which BonicBot calls over the internet using the gradio_client library — the description that comes back is then spoken aloud using a local Piper text-to-speech voice, the same kind used in the voice-assistant lessons.

The workflow this time is simpler than Lesson 14, but the AI underneath is doing something more sophisticated:


camera -> press spacebar -> capture one frame
       -> send the actual image to a VLM hosted online
       -> VLM looks at the picture and writes a description
       -> speak the description out loud and print it to the console

Notice there’s no object detector step at all this time, and no debouncing — because BonicBot isn’t waiting to “confirm” a label across many frames, it’s just looking at a single snapshot the moment you ask it to, the same way you might glance at a photo and describe it.

The Iron Triangle: Speed, Quality, and Cost

Whenever you build something with AI, you’re almost always balancing three things against each other:

Speed — how fast BonicBot responds
Quality — how accurate, detailed, or thoughtful the response is
Cost — how much computing power, memory, or money it takes to run

This trade-off shows up so often in engineering that it has a name: the Iron Triangle. The rule of thumb is that you can usually only push hard on one or two corners of the triangle at a time — pick the third, and something has to give.

Lesson 14 and Lesson 15 are a perfect real example of the Iron Triangle in action, using the exact same camera and the exact same goal:

	Lesson 14 (detector + text LLM)	Lesson 15 (VLM)
Speed	Fast — runs every frame, live	Slow — a few seconds (sometimes longer) per snapshot, since the image has to travel over the internet to a hosted model and back
Quality	Limited — only knows a single word, no scene context	Rich — understands the whole picture at once
Cost	Cheap — small models, low memory	No big local download, but you’re depending on a shared, publicly hosted model that other people are using at the same time

Neither approach is “better” in every way — they sit at different points on the same triangle. Lesson 14 trades quality for speed and cost, so it can run continuously in real time. Lesson 15 trades speed for quality, so it only makes sense to run occasionally, on demand — which is exactly why this script waits for you to press spacebar instead of analyzing every frame.

You may notice real lag in this lesson. It’s normal for the hosted image-description model to take anywhere from a couple of seconds to nearly half a minute to respond after you press spacebar — the video will keep playing smoothly, but BonicBot’s answer will lag behind. This isn’t a bug; it’s the Iron Triangle showing up directly: you’re seeing the “Speed” corner get pulled on by network travel time and by sharing a free, publicly hosted model with everyone else using it at that moment. Paid, dedicated cloud VLMs feel much snappier partly because they run on hardware set aside just for you, instead of a shared public queue.

Setup: Installing Packages

Before running the code, make sure your computer has the required Python packages and system libraries installed. Open a terminal and run:


pip install bonicbot-bridge opencv-python numpy sounddevice piper-tts gradio_client

What each package / tool does

Package / Tool	Purpose
`bonicbot-bridge`	The official BonicBot SDK. Provides `BonicBot` for camera connection and frame retrieval (`bot.start_camera()`, `bot.get_image()`).
`gradio_client`	Python client for Hugging Face Spaces. Connects to `ovi054/image-to-prompt` VLM to send raw camera frame snapshots over the internet and receive image descriptions.
`piper-tts`	Fast, local neural Text-to-Speech (TTS) engine. Synthesizes WAV audio for VLM descriptions.
`sounddevice`	Imported as `sd`. Plays synthesized audio buffers live through system speakers.
`opencv-python`	Imported as `cv2`. Manages live video frame streaming, downscaling (`_resize`), and UI text rendering (`draw_ui`).
`numpy`	Imported as `np`. Handles raw image arrays and audio byte stream buffer reshaping.

If pip install fails, try pip3 install ... instead, or use a virtual environment:


python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy sounddevice piper-tts gradio_client

How to run the program

Install PortAudio (system dependency for audio playback):
- Linux: sudo apt install portaudio19-dev
- macOS: brew install portaudio
Update IP address: Find your BonicBot’s IP address and set HOST = '[IP_ADDRESS]' (or 'localhost' for simulation).
Ensure Internet Connection: This lesson requires internet access to query the remote Hugging Face VLM model.
Save the code into a file, e.g., lesson15_vlm_vision.py.
Run the script:
```
python lesson15_vlm_vision.py
```
A video window titled “BonicBot Vision VLM” will open.
Press SPACEBAR to capture the current camera frame, downscale it, send it to the remote VLM, and receive a rich scene description.
While BonicBot is querying the VLM, “Thinking…” will display on-screen. Once ready, BonicBot will print and speak the description aloud.
Press ‘q’ to exit.

Don’t have a physical BonicBot? Try it in simulation (optional)

If you don’t have physical access to a BonicBot, you can still work through this lesson using the ROS 2 simulation environment:

Launch the BonicBot simulation:


ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True

When HOST = 'localhost', BonicBot connects to the simulated camera feed and sends simulated snapshots to the online VLM when you press spacebar.

Code

Click to view the complete program


import os, sys, subprocess, tempfile, threading, queue, time, wave, io
import cv2, numpy as np, sounddevice as sd
from piper import PiperVoice
from bonicbot_bridge import BonicBot
 
try:
    import gradio_client
except ImportError:
    subprocess.run([sys.executable, "-m", "pip", "install", "gradio_client", "--break-system-packages"], check=True)
from gradio_client import Client, handle_file
 
HOST = 'localhost'
GRADIO_SPACE = "ovi054/image-to-prompt"
GRADIO_API_NAME = "/predict"
VLM_MAX_IMAGE_SIDE = 128
PIPER_VOICE = "en_US-lessac-medium"
PIPER_VOICE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "piper_voices")
 
 
def _resize(frame, max_side):
    h, w = frame.shape[:2]
    scale = max_side / float(max(h, w))
    if scale >= 1.0:
        return frame
    return cv2.resize(frame, (int(w * scale), int(h * scale)), interpolation=cv2.INTER_AREA)
 
 
def ensure_piper_voice(voice_name):
    os.makedirs(PIPER_VOICE_DIR, exist_ok=True)
    model_path = os.path.join(PIPER_VOICE_DIR, f"{voice_name}.onnx")
    if not (os.path.exists(model_path) and os.path.exists(model_path + ".json")):
        subprocess.run([sys.executable, "-m", "piper.download_voices", "--data-dir", PIPER_VOICE_DIR, voice_name], check=True)
    return model_path
 
 
class VLMWorker:
    def __init__(self, client, voice):
        self._client = client
        self._voice = voice
        self._queue = queue.Queue(maxsize=1)
        self._lock = threading.Lock()
        self._busy = False
        threading.Thread(target=self._run, daemon=True).start()
 
    def analyze_frame(self, frame):
        try:
            self._queue.put_nowait(frame.copy())
        except queue.Full:
            pass
 
    def is_busy(self):
        with self._lock:
            return self._busy
 
    def _run(self):
        while True:
            frame = self._queue.get()
            with self._lock:
                self._busy = True
            try:
                description = self._ask_vlm(frame)
            except Exception as e:
                description = f"(couldn't reach the remote model: {e})"
            with self._lock:
                self._busy = False
            print(f"🗣️ {description}")
            self._speak(description)
 
    def _ask_vlm(self, frame):
        frame = _resize(frame, VLM_MAX_IMAGE_SIDE)
        fd, tmp_path = tempfile.mkstemp(suffix=".jpg")
        os.close(fd)
        try:
            cv2.imwrite(tmp_path, frame)
            result = self._client.predict(image=handle_file(tmp_path), api_name=GRADIO_API_NAME)
        finally:
            os.remove(tmp_path)
        return str(result).strip()
 
    def _speak(self, text):
        with io.BytesIO() as buf:
            with wave.open(buf, "wb") as wf:
                self._voice.synthesize_wav(text, wf)
            buf.seek(0)
            with wave.open(buf, "rb") as wf:
                ch, rate, raw = wf.getnchannels(), wf.getframerate(), wf.readframes(wf.getnframes())
        audio = np.frombuffer(raw, dtype=np.int16)
        if ch > 1:
            audio = audio.reshape(-1, ch)
        sd.play(audio, samplerate=rate)
        sd.wait()
 
 
def draw_ui(frame, busy):
    cv2.putText(frame, "SPACE = analyze what I see   |   Q = quit", (10, 26), cv2.FONT_HERSHEY_SIMPLEX, 0.55, (0, 255, 255), 2)
    if busy:
        cv2.putText(frame, "Thinking...", (10, 54), cv2.FONT_HERSHEY_SIMPLEX, 0.55, (0, 200, 255), 2)
    return frame
 
 
def main():
    voice = PiperVoice.load(ensure_piper_voice(PIPER_VOICE))
    client = Client(GRADIO_SPACE)
    worker = VLMWorker(client, voice)
 
    with BonicBot(host=HOST) as bot:
        bot.system.start_camera()
        bot.start_camera()
        bot.camera.wait_for_image(timeout=5.0)
 
        while True:
            frame = bot.get_image()
            if frame is None:
                if cv2.waitKey(1) & 0xFF == ord("q"):
                    break
                continue
 
            cv2.imshow(f"BonicBot Vision VLM ({GRADIO_SPACE})", draw_ui(frame.copy(), worker.is_busy()))
 
            key = cv2.waitKey(1) & 0xFF
            if key == ord("q"):
                break
            elif key == 32:
                worker.analyze_frame(frame)
 
        cv2.destroyAllWindows()
        bot.stop_camera()
        bot.system.stop_camera()
 
 
if __name__ == "__main__":
    main()

Before running the program, replace [IP_ADDRESS] with the IP address of your own BonicBot.


HOST = '[IP_ADDRESS]'

This lesson needs a working internet connection, since the image-description model runs on a public Hugging Face Space rather than on your own computer.

Install the required Python packages:


pip install opencv-python numpy sounddevice piper-tts gradio_client --break-system-packages

⚠️ On the first run, the script automatically downloads the small Piper voice files it needs for text-to-speech. This is a quick, one-time download — nowhere near as big as the local vision-language models used in some setups, since the actual “seeing” happens on the hosted model, not on your computer.

Code Walkthrough

Line-by-line explanation

Imports & Dependencies (lines 66–82) — Imports cv2, sd, PiperVoice, BonicBot, and gradio_client. Checks for gradio_client installation automatically. Sets constants: GRADIO_SPACE = "ovi054/image-to-prompt", VLM_MAX_IMAGE_SIDE = 128.
Image Resizing Helper (lines 85–90) — _resize() scales down image frames to a maximum side of 128px before network transmission to minimize upload bandwidth and API latency.
Piper Voice Setup (lines 93–98) — ensure_piper_voice() automatically downloads the required ONNX voice file on first run.
Asynchronous VLM Worker (lines 101–157) — VLMWorker wraps a background thread and a 1-slot queue (queue.Queue(maxsize=1)). _ask_vlm() saves the frame as a temporary JPEG, uploads it via gradio_client.Client, and fetches the VLM description. _speak() synthesizes WAV audio with Piper and plays it via sounddevice.
UI Rendering (lines 159–163) — draw_ui() draws on-screen instructions (SPACE = analyze what I see) and displays a yellow “Thinking…” status indicator while the VLM request is processing.
Main Interaction Loop (lines 166–197) — Connects to BonicBot, streams live camera frames, listens for keyboard input (ord("q") to quit, 32 [Spacebar] to trigger frame analysis), and cleanly releases resources upon shutdown.

Expected Output

Click to see expected output

Visual Output:

📌 Note: This visual demonstration is using the ROS 2 simulation with use_real_camera:=True.

Terminal Output:

A live video window opens titled “BonicBot Vision VLM,” with on-screen instructions along the top:


SPACE = analyze what I see   |   Q = quit

When you press spacebar, “Thinking…” appears in the corner of the window while BonicBot waits on the hosted model, and once a description comes back the console prints something like:


🗣️ A person sitting at a desk with a laptop and a coffee mug.

A few seconds after pressing spacebar, BonicBot will speak that description out loud and print it to the console with a 🗣️ icon.

Unlike the previous lesson, BonicBot won’t say anything on its own — it only analyzes a frame when you press spacebar, since a VLM request takes real thinking time and isn’t meant to run on every single frame.

Since the description comes from a model out on the internet, don’t be alarmed if BonicBot goes quiet for several seconds (sometimes up to 20-30 seconds) after you press spacebar before it speaks — the video feed will keep updating normally the whole time, since the request runs on a background thread. See the Iron Triangle note above for why this happens.

🔧 Under the Hood

How is this actually different from Lesson 14?

The most important line in this whole script is here:


result = self._client.predict(image=handle_file(tmp_path), api_name=GRADIO_API_NAME)

In Lesson 14, the model only ever received a single word, like "bottle" — it never saw the picture at all. Here, the actual JPEG image bytes are uploaded to a hosted image-captioning model on Hugging Face, which has learned, from millions of real photos and captions, how pixels relate to the words used to describe them. That means it can notice things no object detector was ever told to look for — colors, arrangements, actions, or even objects that were never in its detection list to begin with.

This comes at a cost, though. A VLM request takes noticeably longer to “think” than either the plain object detector or the plain text-only LLM from earlier lessons, since the image has to travel to a remote server and back instead of being processed instantly on your own machine. That’s exactly why this script waits for you to press spacebar instead of trying to analyze every frame automatically — a VLM is powerful, but it’s not meant to run continuously the way a lightweight object detector can.

The background-thread pattern (VLMWorker) is the same idea you saw in Lesson 14’s DescriptionWorker — the request is handed off to another thread so the camera preview never freezes while BonicBot is “thinking” about the picture.

Student Challenge

Run Lesson 14 and Lesson 15 back-to-back, on the same objects, and put the Iron Triangle to the test yourself.

Time it. Use a stopwatch (or just count seconds out loud) from the moment an object appears in Lesson 14 to when BonicBot speaks, versus from the moment you press spacebar in Lesson 15 to when BonicBot speaks. How big is the gap?
Compare the descriptions. Point BonicBot at the same objects using Lesson 14’s script, then this lesson’s script. Do the descriptions differ in detail or accuracy?
Try a busy scene. Set up multiple objects together (like a cup next to a book). Does the object detector from Lesson 14 — which only describes one object at a time — miss something that the VLM in this lesson notices about how the objects relate to each other?
Try a different image size. Change VLM_MAX_IMAGE_SIDE to a smaller or larger value. Does the description quality change? Does the response come back faster or slower?

Afterward, decide for yourself: if you were building a real product for BonicBot, when would you reach for Lesson 14’s approach, and when would Lesson 15’s slower-but-richer approach be worth the wait?

Hint

Try shrinking the image further before it’s sent:


VLM_MAX_IMAGE_SIDE = 64

A smaller image means less data to upload, which can shave time off the round trip — but it also gives the model fewer pixels to work with, so watch whether the description gets vaguer.

Reflection Question

Thinking about the Iron Triangle — speed, quality, and cost — Lesson 14 and Lesson 15 each chose to sacrifice a different corner to gain the other two. If you were designing a real BonicBot feature meant to run all day, every day, on a classroom robot with only a CPU, which corner would you be least willing to give up, and why?