Lesson 7: Teaching BonicBot Image Segmentation

This lesson also relies on the ultralytics package to load and run the YOLO segmentation model (YOLO("yolo26n-seg.pt")). It’s installed separately in the Prerequisites box inside the Code section below, since it isn’t part of the core BonicBot/OpenCV/NumPy stack shared across lessons.

If you already installed bonicbot-bridge, opencv-python, and numpy in an earlier lesson, you don’t need to reinstall anything for this lesson — just make sure ultralytics is installed as well.

If pip install fails, try pip3 install ... instead, or use a virtual environment:

python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy ultralytics

1. Find your BonicBot’s IP address — via the robot’s on-device display, its companion app, or your router’s connected-devices list. (This step isn’t required if you plan to use the ROS 2 simulation instead — see below.)
2. Save the code below into a file, e.g. lesson7_segmentation.py.
3. Note the connection line. This script already sets HOST = "localhost" and connects with with BonicBot(host=HOST, port=9090, timeout=10) as bot:.
   • If you’re running this against the ROS 2 simulation, you can leave HOST as "localhost" and run it as-is.
   • If you’re connecting to a real BonicBot, change HOST = "localhost" to your robot’s actual IP address, e.g. HOST = "172.20.10.2".
4. Make sure your BonicBot (real or simulated) is powered on and network-connected. Since this lesson calculates how much of the frame an object covers, make sure there’s clear space in front of the camera and a segmentable object (like a person or a bottle) that you can move closer to and farther from the lens.
5. Run the script:

   python lesson7_segmentation.py

6. A window titled “BonicBot Image Segmentation” should open, showing the live feed with colored pixel masks and bounding boxes drawn over detected objects, plus a small HUD panel in the top-left corner.
7. Press q with the video window focused to stop the stream and exit cleanly.

This lesson’s code already defaults to HOST = "localhost", so it’s already set up to run directly against the ROS 2 simulation environment without any code changes:

1. Launch the BonicBot simulation:

   ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True

• use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion are still simulated. This is the easiest way to test detection, since you can just hold whatever you want detected up to your laptop camera.
• use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).

2. With the simulation running, simply run the script as written — since HOST is already "localhost", no code changes are needed.
3. Everything else in the code — segmentation, mask-area calculation, and the HUD overlay — works the same way, since the simulation exposes the same interface as a real robot.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, changing HOST to its actual IP address is still the recommended way to go through this lesson on real hardware.

• Imports — time tracks frame timing for the FPS counter; cv2 and numpy handle display, drawing, and mask math; YOLO from ultralytics loads and runs the segmentation model; BonicBot and BonicBotError from bonicbot_bridge handle the robot connection and error handling.
• process_segmentation(frame, results) — Takes the raw camera frame and the YOLO results object for that frame. Calls results.plot() to get an annotated frame with masks and boxes already drawn by Ultralytics. If results.masks is present, it loops over each (mask, box) pair: reads the detected class_id / class_name from results.names and the detection confidence from box.conf[0], converts the mask tensor to a NumPy array with mask.cpu().numpy(), counts pixels above the 0.5 threshold with np.sum(mask_np > 0.5), and divides by mask_np.size to compute area_pct — the percentage of the frame that object’s mask covers. Each detection is appended as a dictionary to detections, which is returned alongside annotated_frame.
• draw_hud(frame, detections, fps) — Draws a semi-transparent black panel in the top-left corner using cv2.rectangle + cv2.addWeighted, then overlays three lines of text: a “Segmented: N objects” status line (green if count > 0, red otherwise), a "Largest: <class> (<pct>% area)" line identifying the biggest detection by area_pct via max(detections, key=lambda x: x["area_pct"]), and the current fps value.
• HOST = "localhost" — The default connection target, set up to work out of the box with the ROS 2 simulation; swap this for a real BonicBot’s IP address to run on hardware.
• model = YOLO("yolo26n-seg.pt") — Loads the lightweight Nano segmentation model once, before the connection block, so it’s ready to run inference on every frame without reloading.
• try: with BonicBot(...) as bot: — Wraps the entire session in a try block so that a BonicBotError (e.g. a dropped connection) can be caught cleanly instead of crashing the script. Inside, bot.system.start_camera() and bot.start_camera() bring the camera online, and bot.camera.wait_for_image(timeout=5.0) blocks briefly until the first frame is ready.
• Main loop — Each iteration pulls the latest frame with bot.camera.get_latest_image(), skipping the rest of the loop (with a short time.sleep(0.01)) if no frame is available yet. It updates the smoothed fps value the same way as earlier lessons, then runs model.predict(source=frame, imgsz=320, conf=0.25, verbose=False)[0] to get segmentation results for that frame — imgsz=320 keeps inference fast, and conf=0.25 filters out low-confidence detections. The frame and results are passed to process_segmentation(), the HUD is drawn on top with draw_hud(), and the result is shown in the "BonicBot Image Segmentation" window via cv2.imshow.
• Quit condition — cv2.waitKey(1) & 0xFF == ord("q") checks each frame for the q key and breaks out of the loop when pressed.
• except BonicBotError as e: — Catches robot-specific connection or communication errors and prints a readable message instead of a raw traceback.
• finally: cleanup — Always runs, even if an error occurred: closes the OpenCV window with cv2.destroyAllWindows(), then attempts bot.system.stop_camera() inside its own try/except NameError, since bot may not exist yet if the connection itself failed before the with block ever bound it.

Visual Output:

Terminal Output:

🧠 Loading lightweight YOLOv26 segmentation model...
📷 Starting BonicBot camera and streaming...
✅ Camera running. Press 'q' in the window to stop.

[15:10:22] 📣 Alert! A large person is blocking my path!
[15:10:28] 📣 Alert! A large bottle is blocking my path!

The BonicBot Vision Window will display:

• The live camera feed from BonicBot.
• Precise, semi-transparent colored masks overlaying each detected object.
• Bounding boxes and confidence labels for each object.
• A translucent HUD showing:
  • Total number of segmented objects.
  • The name and pixel-percentage size of the largest detected object.
  • Processing speed (FPS).

Whenever a target object (like a person or bottle) gets too close to the camera (exceeding 12% of the frame’s total pixels), BonicBot will announce:

“Alert! A large [object] is blocking my path!”

1. Bounding Boxes vs. Segmentation Masks

Traditional object detection (like in Lesson 1) only outputs bounding boxes. A bounding box consists of four coordinates: [x_min, y_min, x_max, y_max].

Image Segmentation outputs a pixel-wise binary mask. If the input image is 320 × 320 pixels, the model produces a grid of the same dimensions where each pixel is labeled 1 (if it belongs to the object) or 0 (if it belongs to the background).

2. Lightweight Inference (Nano & Downscaling)

Running deep learning segmentation models is computationally expensive, especially on edge hardware. To achieve a high frame rate, we apply two optimization techniques:

• Nano Model Scale: We load yolov26n-seg.pt (Nano variant). It is trained on the 80-class COCO dataset but uses fewer layers and channels compared to larger models (like yolov26m-seg or yolov26x-seg), making it run fast even on CPU.
• Image Downscaling (imgsz=320): By default, YOLO models process images at 640 × 640 pixels. By setting imgsz=320, we reduce the total pixels the neural network has to compute by 75%, which dramatically increases inference speed (FPS).

3. Obstacle Size Calculation

Because we have a pixel mask, we can calculate how much space an object occupies in BonicBot’s camera view:

Percentage Area = (Pixels in Mask ÷ Total Pixels in Mask Grid) × 100

If this percentage exceeds a threshold (e.g., 12%), the object is close enough to be considered a physical obstacle in front of the robot.

Inside the Alert Decision Pipeline, check the object’s class and write conditional statements to call different BonicBot actions:

if det["class"] == "person" and det["area_pct"] > 15.0:
    bot.stop()
    bot.look_up()  # look up towards the person
    speaker.speak("Excuse me, could you step aside?")
elif det["class"] == "bottle" and det["area_pct"] > 10.0:
    bot.stop()
    bot.open_grippers()  # prepare to grab the bottle

Remember to import any exception handlers (like BonicBotError) to protect your code from runtime issues!