Lesson 9: Teaching BonicBot to Recognize Faces

This lesson also relies on face_database.pkl, the file produced by Lesson 8. Make sure that file exists in the same directory before running this script — the program will exit immediately if it isn’t found.

If you already installed bonicbot-bridge, opencv-python, numpy, pyttsx3, and deepface in earlier lessons, you don’t need to reinstall anything for this lesson.

If pip install fails, try pip3 install ... instead, or use a virtual environment:

python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy pyttsx3 deepface

1. Find your BonicBot’s IP address — via the robot’s on-device display, its companion app, or your router’s connected-devices list.
2. Make sure face_database.pkl from Lesson 8 is in the same folder as this script — the program loads it on startup and will exit with an error if it’s missing.
3. Save the code below into a file, e.g. lesson9_face_recognition.py.
4. Note the connection line. This script sets HOST = "localhost" and connects with with BonicBot(host=HOST, port=9090, timeout=10) as bot:.
   • If you’re running this against the ROS 2 simulation, you can leave HOST as "localhost" and run it as-is.
   • If you’re connecting to a real BonicBot, change HOST = "localhost" to your robot’s actual IP address, e.g. HOST = "172.20.10.2".
5. Make sure your BonicBot (real or simulated) is powered on and network-connected, and that at least one person whose face is in face_database.pkl is standing clearly in view of the camera.
6. (Optional) Connect a Bluetooth speaker to hear the spoken announcements. Recognition still works without one.
7. Run the script:

   python lesson9_face_recognition.py

8. A window titled “Face Recognition Feed” should open, showing the live feed with a green bounding box and name/confidence label drawn around each detected face.
9. Press q with the video window focused to stop the stream and exit cleanly.

This lesson’s code already defaults to HOST = "localhost", so it’s already set up to run directly against the ROS 2 simulation environment without any code changes:

1. Launch the BonicBot simulation:

   ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True

• use_real_camera:=True — the camera feed comes from your laptop’s webcam, while the robot’s body and motion are still simulated. This is the easiest way to test recognition, since you can just show your own face (if it’s in face_database.pkl) to your laptop camera.
• use_real_camera:=False — the camera feed comes from Gazebo instead (i.e. whatever the simulated camera sees inside the simulated world).

2. With the simulation running, simply run the script as written — since HOST is already "localhost", no code changes are needed.
3. Everything else in the code — face detection, embedding comparison, stability tracking, and voice announcements — works the same way, since the simulation exposes the same interface as a real robot.

This path is mainly useful for exploring the lesson without hardware on hand; if you have a real BonicBot, changing HOST to its actual IP address is still the recommended way to go through this lesson on real hardware.

• Imports — os, sys, subprocess, importlib support the auto-install logic; time handles timestamps and retry delays; pickle loads the face database; queue and threading power the background TTS worker and the threaded recognition pipeline; numpy (np) handles embedding math; BonicBot / BonicBotError manage the robot connection.
• pip_install(pkg) / ensure(module, pkg=None) — Small helper pair used to auto-install missing dependencies at runtime: ensure() tries importlib.import_module(module) first, and only calls pip_install() if the import fails, then imports again. cv2 is loaded this way (cv2 = ensure("cv2", "opencv-python")), as is deepface (ensure("deepface")).
• pyttsx3 import block — Wrapped in try/except ImportError to set _TTS_AVAILABLE, the same optional-TTS pattern used in earlier lessons.
• pick_detector_backend() — Checks for cv2.CascadeClassifier first and returns "opencv" if present. Otherwise it loops through mediapipe, mtcnn, and retinaface, using ensure() to install each one if needed and testing it with a throwaway DeepFace.represent() call on a blank image before returning the first backend that works. Exits with sys.exit(...) if none succeed.
• DETECTOR_BACKEND = pick_detector_backend() — Runs once at startup, and DeepFace.build_model("ArcFace") is called right after to pre-load the model so the first real recognition call isn’t slowed down by a cold start.
• VoiceSpeaker — Same background-thread TTS pattern as previous lessons: speak() pushes text onto a queue.Queue, _run() consumes it on a daemon thread and speaks each item with pyttsx3, and stop() pushes None as a sentinel to end the loop.
• HOLD_FRAMES, MISS_TOLERANCE — Tuning constants for the RecognitionTracker: how many consecutive “seen” updates are needed before a name is announced, and how many consecutive misses are tolerated before a person’s tracking state is forgotten.
• RecognitionTracker — Maintains a _state dictionary keyed by name, each holding a streak, announced flag, and missed counter. update(seen_names) increments missed for names no longer seen (deleting their state once missed exceeds MISS_TOLERANCE), and increments streak for names currently seen, resetting missed to 0. A name is added to the returned stable list — and announced is flipped to True — only the moment its streak first reaches HOLD_FRAMES, so each appearance is announced exactly once.
• Face database loading — Opens face_database.pkl with pickle.load(), pulling out known_names (the identities list) and stacking every stored embedding into a single 2D array with np.vstack(db["embeddings"]) for fast batch comparison. If the file doesn’t exist, sys.exit(...) stops the script immediately with a clear message.
• recognize_faces(frame) — Calls DeepFace.represent(img_path=frame, ...) to detect and embed every face in the given frame, using enforce_detection=False so a frame with no face doesn’t raise. For each detected face (fd), it skips low-confidence detections (face_confidence < 0.6), normalizes the new embedding the same way Lesson 8 did, then computes similarity against every known embedding at once via np.dot(known_matrix, vec) and picks the best match with np.argmax(sims). If that best similarity score is below 0.45, the person is labeled "Unknown" instead of a database name. Returns a list of (name, confidence, facial_area) tuples.
• recognition_worker(small_frame, scale) — Runs recognize_faces() on a background thread so the main video loop never blocks waiting on DeepFace. Because the frame passed in was downscaled for speed, each face’s bounding box (x, y, w, h) is divided back by scale to map it to full-frame coordinates before being stored in the shared latest_detections list (protected by lock). The busy event is cleared in a finally block so a new recognition pass can be scheduled once this one finishes.
• HOST, MAX_FAILS, RECOGNIZE_EVERY_N_FRAMES, DOWNSCALE_WIDTH — Connection target and performance tuning constants: how many consecutive failed frame grabs are tolerated before giving up, how often (in frames) a new recognition pass is kicked off, and how narrow frames are resized to before being sent to DeepFace.
• Shared state (speaker, tracker, lock, latest_detections, busy) — A single VoiceSpeaker and RecognitionTracker are created once; lock guards latest_detections, the most recent set of recognized faces available to the main loop; busy is a threading.Event used to prevent overlapping recognition passes from running at the same time.
• Main loop — Inside with BonicBot(host=HOST, port=9090, timeout=10) as bot:, the camera is started with bot.system.start_camera() / bot.start_camera() and bot.camera.wait_for_image(). Each iteration:
  • Pulls a frame with bot.camera.get_latest_image(), tracking consecutive failures in fails and breaking out after MAX_FAILS is reached.
  • Every RECOGNIZE_EVERY_N_FRAMES frames — and only if no recognition pass is already running (not busy.is_set()) — downsizes the frame via cv2.resize and launches recognition_worker on a new daemon thread, rather than running DeepFace on every single frame.
  • Reads the latest results out of latest_detections under lock, then calls tracker.update(...) with just the detected names; any name returned as newly “stable” triggers a printed log line and a speaker.speak(...) announcement — “Unknown person detected” for unmatched faces, "{name} recognized" otherwise.
  • Draws a green bounding box and a "{name} ({confidence})" label for every currently tracked detection using cv2.rectangle and cv2.putText, then shows the frame in the 'Face Recognition Feed' window.
  • Checks for the q key via cv2.waitKey(1) to break the loop.
• Cleanup — The inner finally block closes the OpenCV window and calls bot.system.stop_camera() once the loop ends. The outer try/except BonicBotError catches robot connection errors, and the outermost finally calls speaker.stop() and prints a final shutdown message regardless of how the script exited.

Visual Output:

Terminal Output:

Loading face database...
Loaded 18 profiles.

📷 Starting BonicBot camera and streaming...
✅ Camera running.

[14:52:08] 🟢 Alice recognized
[14:52:17] 🟢 Bob recognized
[14:52:29] 🟢 Unknown person detected

The BonicBot Vision Window will display:

• A live view from BonicBot’s vision system.
• Face recognition running continuously.
• Console messages whenever a person is successfully recognized.
• Voice announcements for recognized and unknown people.

BonicBot only announces a name once while the person remains in view. If they leave and return later, the robot will announce them again.

This lesson uses the ArcFace deep learning model for face recognition.

When BonicBot sees a face, it first generates a new face embedding.

For example:

Person Standing in Front of BonicBot

↓

ArcFace

↓

[0.14, -0.32, 0.87, ...]

The program then compares this new embedding with every embedding stored inside face_database.pkl.

To measure how similar two embeddings are, it uses Cosine Similarity.

New Face Embedding
│
▼
Compare with Database
│
▼
Highest Cosine Similarity
│
▼
Recognized Person

If the highest similarity score is greater than the chosen threshold, BonicBot considers the face to be a match.

Otherwise, the person is labeled as Unknown.

The program also uses a Recognition Tracker, which waits until the same person has been recognized for several consecutive processed frames before announcing their name. This makes the robot much more stable and prevents repeated announcements while someone remains standing in front of BonicBot.

Instead of saying:

speaker.speak(message)

Create a custom greeting.

For example:

speaker.speak(f"Hello {name}!")

You could also use an if statement to give different greetings to different people.