Lesson 18: BonicBot the Actor

Learning Objective

Connect LLM-generated movement scripts to the physical BonicBot. Learn how to dispatch neck and move actions directly to the SDK while translating abstract arm poses into precise joint angles using a preset lookup table.

Introduction

The LLM can write a script full of steps like arm_pose: "wave" — but this lesson is where BonicBot becomes the actor: the choreographer asks the LLM for a script, and every step is physically executed on the robot.

Two steps don’t need a translation table at all, because their schema values are the SDK method names:


def neck_gesture(bot, neck_pose, duration_seconds):
    getattr(bot, neck_pose)()          # neck_pose is literally "look_left" etc.
    time.sleep(duration_seconds)
    if neck_pose != "look_center":
        bot.look_center()

move steps are similar, but each SDK method takes slightly different keyword arguments (a different default speed for driving vs. turning), so move_gesture() uses a short if/elif instead of a single dictionary call.

arm steps are the one case that genuinely needs translation — "hug_self" isn’t a method name, it’s a preset that has to be looked up:


ARM_PRESETS = {
    "hug_self": (30.0, 50.0),   # (shoulder_deg, elbow_deg)
    ...
}
shoulder_deg, elbow_deg = ARM_PRESETS[arm_pose]

Because arm_pose is also locked to this exact list in the Pydantic schema, the LLM can only ever request a preset this dictionary actually knows how to translate — the schema and the translation table stay in sync.

Setup: Installing Packages

Before running the code, make sure your computer has the required Python packages and system libraries installed. Open a terminal and run:


pip install bonicbot-bridge opencv-python numpy ollama pydantic piper-tts pyaudio

What each package / tool does

Package / Tool	Purpose
`bonicbot-bridge`	The official BonicBot SDK. Connects to `BonicBot` to execute physical neck poses (`bot.look_left()`), arm movements (`bot.move_left_arm()`), and chassis maneuvers (`bot.move_forward()`).
`pydantic`	Data validation library. Validates LLM choreography JSON output against strict schemas before executing physical hardware actions.
`ollama`	Python client for local LLMs. Uses `qwen3.5:0.8b` (or `qwen3:0.6b`) to plan multi-step performance scripts.
`piper-tts`	Fast, local neural Text-to-Speech (TTS) engine. Synthesizes voice audio for `speak` choreography steps.
`pyaudio`	Audio I/O library. Streams synthesized PCM audio chunks directly to your computer speakers.

If pip install fails, try pip3 install ... instead, or use a virtual environment:


python3 -m venv bonicbot-env
source bonicbot-env/bin/activate   # On Windows: bonicbot-env\Scripts\activate
pip install bonicbot-bridge opencv-python numpy ollama pydantic piper-tts pyaudio

How to run the program

Install Ollama (one-time setup):
- Windows / Mac: Download and run the installer from ollama.com/download .
- Linux: Run curl -fsSL https://ollama.com/install.sh | sh in your terminal.
Install PortAudio (system dependency for pyaudio):
- Linux: sudo apt install portaudio19-dev
- macOS: brew install portaudio
Pull the LLM model:
```
ollama pull qwen3.5:0.8b
```
Update IP address: Find your BonicBot’s IP address and set HOST = '[IP_ADDRESS]' (or 'localhost' for simulation).
Ensure Choreography Helper: Make sure choreography.py from Lesson 17 is in the same folder.
Save the code into a file, e.g., lesson18_actor.py.
Run the script:
```
python lesson18_actor.py
```
Type instructions into the console (e.g. “act shy and say hello”, “turn around and walk away”).
BonicBot will generate the choreography, validate it with Pydantic, speak lines out loud via Piper TTS, and execute physical arm/neck/chassis movements!
Type “quit” or press Ctrl+C to exit.

Don’t have a physical BonicBot? Try it in simulation (optional)

If you don’t have physical access to a BonicBot, you can still work through this lesson using the ROS 2 simulation environment:

Launch the BonicBot simulation:


ros2 launch my_bot robot_system.launch.py use_sim_time:=true world:=obsworld.sdf use_real_camera:=True

When HOST = 'localhost', BonicBot connects to the simulated robot and executes arm poses, neck turns, and chassis movements on screen seamlessly.

Code

Click to view the complete program


 
import threading
import time
from pathlib import Path
 
import pyaudio
from piper import PiperVoice, SynthesisConfig
from piper.download_voices import download_voice
from bonicbot_bridge import BonicBot
from bonicbot_bridge.exceptions import BonicBotError
 
from choreography import (
    OLLAMA_MODEL,
    ask_choreographer,
    ensure_model_pulled,
    ensure_ollama_installed,
    ensure_ollama_running,
    warm_up_model,
)
 
 
HOST = 'localhost'
 
PIPER_VOICE_NAME = "en_US-lessac-medium"
PIPER_VOICES_DIR = Path(__file__).resolve().parent / "piper_voices"
 
ARM_PRESETS = {
    "rest": (0.0, 0.0),
    "wave": (90.0, 30.0),
    "reach_forward": (90.0, 0.0),
    "shrug": (45.0, 50.0),
    "droop": (-30.0, 15.0),
    "celebrate": (170.0, 10.0),
    "thinking": (60.0, 50.0),
    "hug_self": (30.0, 50.0),
}
SHOULDER_RANGE = (-45, 180)
ELBOW_RANGE = (0, 50)
 
DEFAULT_LINEAR_SPEED = 0.2
DEFAULT_TURN_SPEED = 15.0
 
MIN_MOVE_DURATION_SECONDS = 0.3
MAX_MOVE_DURATION_SECONDS = 6.0
 
 
def clamp(value, min_val, max_val):
    return max(min_val, min(max_val, value))
 
 
def timed_call(label, func, *args, **kwargs):
    arg_str = ", ".join([repr(a) for a in args] + [f"{k}={v!r}" for k, v in kwargs.items()])
    print(f"  🤖 {label}({arg_str})", end="", flush=True)
    t0 = time.time()
    result = func(*args, **kwargs)
    print(f" → {result!r}  ⏱ {time.time() - t0:.2f}s")
    return result
 
 
def neck_gesture(bot, neck_pose, duration_seconds):
    timed_call(neck_pose, getattr(bot, neck_pose))
    time.sleep(duration_seconds)
    if neck_pose != "look_center":
        timed_call("look_center", bot.look_center)
 
 
def arm_gesture(bot, arm_side, arm_pose, duration_seconds):
    shoulder_deg, elbow_deg = ARM_PRESETS[arm_pose]
    shoulder_deg = clamp(shoulder_deg, *SHOULDER_RANGE)
    elbow_deg = clamp(elbow_deg, *ELBOW_RANGE)
    method_name = "move_left_arm" if arm_side == "left" else "move_right_arm"
    move = getattr(bot, method_name)
 
    timed_call(method_name, move, shoulder_deg, elbow_deg, wait=True)
    time.sleep(duration_seconds)
    if arm_pose != "rest":
        timed_call(method_name, move, 0.0, 0.0, wait=True)
 
 
def move_gesture(bot, move_action, turn_degrees, move_distance_meters):
    if move_action in ("turn_left", "turn_right"):
        speed = DEFAULT_TURN_SPEED
        amount = turn_degrees
    else:
        speed = DEFAULT_LINEAR_SPEED
        amount = move_distance_meters
 
    raw_duration = amount / speed
    duration = clamp(raw_duration, MIN_MOVE_DURATION_SECONDS, MAX_MOVE_DURATION_SECONDS)
    if duration != raw_duration:
        print(f"  ⚠️ Requested move would take {raw_duration:.2f}s — capped to {duration:.2f}s for safety.")
 
    timed_call(move_action, getattr(bot, move_action), speed=speed, duration=duration)
 
 
def speak_step(piper_voice, pyaudio_instance, text):
    print(f"  🗣️  speak(text={text!r})", end="", flush=True)
    t0 = time.time()
    syn_config = SynthesisConfig(length_scale=1.0, volume=1.0)
    stream = None
    try:
        for chunk in piper_voice.synthesize(text, syn_config=syn_config):
            if stream is None:
                stream = pyaudio_instance.open(
                    format=pyaudio_instance.get_format_from_width(chunk.sample_width),
                    channels=chunk.sample_channels,
                    rate=chunk.sample_rate,
                    output=True,
                )
            stream.write(chunk.audio_int16_bytes)
    finally:
        if stream is not None:
            stream.stop_stream()
            stream.close()
    print(f" → done  ⏱ {time.time() - t0:.2f}s")
 
 
def ensure_piper_voice_ready(voice_name, voices_dir):
    voices_dir.mkdir(parents=True, exist_ok=True)
    model_path = voices_dir / f"{voice_name}.onnx"
    config_path = voices_dir / f"{voice_name}.onnx.json"
    if model_path.exists() and config_path.exists():
        return model_path, config_path
    print(f"Downloading voice '{voice_name}'...")
    download_voice(voice_name, voices_dir)
    return model_path, config_path
 
 
def run_step(bot, piper_voice, pyaudio_instance, step, step_num, total):
    print(f"Step {step_num}/{total} — {step.action}")
    t0 = time.time()
    try:
        if step.action == "speak":
            speak_step(piper_voice, pyaudio_instance, step.speech_text)
        elif step.action == "neck":
            neck_gesture(bot, step.neck_pose, step.duration_seconds)
        elif step.action == "arm":
            arm_gesture(bot, step.arm_side, step.arm_pose, step.duration_seconds)
        elif step.action == "move":
            move_gesture(bot, step.move_action, step.turn_degrees, step.move_distance_meters)
    except BonicBotError as e:
        print(f"  ⚠️ Couldn't perform this step: {e}")
    print(f"  ⏱  step total: {time.time() - t0:.2f}s\n")
 
 
def perform_choreography(bot, piper_voice, pyaudio_instance, choreography):
    print(f"\n🎬 Performing: {choreography.title}\n")
    steps = choreography.steps
    total = len(steps)
    i = 0
    while i < total:
        step = steps[i]
        if step.run_with_next and i + 1 < total:
            partner = steps[i + 1]
            print(f"Steps {i + 1}+{i + 2}/{total} — running together: {step.action} + {partner.action}")
            thread_a = threading.Thread(target=run_step, args=(bot, piper_voice, pyaudio_instance, step, i + 1, total))
            thread_b = threading.Thread(target=run_step, args=(bot, piper_voice, pyaudio_instance, partner, i + 2, total))
            thread_a.start()
            thread_b.start()
            thread_a.join()
            thread_b.join()
            i += 2
        else:
            run_step(bot, piper_voice, pyaudio_instance, step, i + 1, total)
            i += 1
    print("✅ Performance complete.\n")
 
 
def main():
    ensure_ollama_installed()
    ensure_ollama_running()
    ensure_model_pulled(OLLAMA_MODEL)
 
    model_path, config_path = ensure_piper_voice_ready(PIPER_VOICE_NAME, PIPER_VOICES_DIR)
    piper_voice = PiperVoice.load(model_path, config_path=config_path)
    pyaudio_instance = pyaudio.PyAudio()
 
    warm_up_model(OLLAMA_MODEL)
 
    try:
        with BonicBot(host=HOST, port=9090, timeout=10) as bot:
            print("✅ Connected. Type an instruction, or 'quit' to stop.")
            print('   Example: "act shy and say hello"\n')
 
            while True:
                instruction = input("Instruction> ").strip()
                if not instruction:
                    continue
                if instruction.lower() in {"quit", "exit"}:
                    break
 
                choreography = ask_choreographer(instruction)
                if choreography is None:
                    continue
 
                print(f"\n📋 {choreography.title}")
                print(choreography.model_dump_json(indent=2))
 
                perform_choreography(bot, piper_voice, pyaudio_instance, choreography)
 
    except BonicBotError as e:
        print(f"⚠️ Robot error: {e}")
    finally:
        pyaudio_instance.terminate()
 
 
if __name__ == "__main__":
    main()

Replace HOST = 'localhost' near the top of the CONFIG section with your BonicBot’s actual IP address before running.

Code Walkthrough

Line-by-line explanation

Arm Presets & Speed Constants (lines 75–93) — Defines ARM_PRESETS mapping 8 emotion poses (rest, wave, reach_forward, shrug, droop, celebrate, thinking, hug_self) to (shoulder_deg, elbow_deg) angles. Sets joint limits (SHOULDER_RANGE, ELBOW_RANGE) and default speeds.
Dynamic Helper Functions (lines 95–142) — clamp() caps values to safe hardware limits. timed_call() logs SDK method executions with timing. neck_gesture(), arm_gesture(), and move_gesture() dispatch actions to BonicBot methods.
Speech Synthesis (lines 144–164) — speak_step() synthesizes WAV audio with Piper TTS and plays speech streams via PyAudio.
Parallel Step Execution (lines 177–215) — perform_choreography() iterates through plan steps. When run_with_next=True is set on a step, it uses threading.Thread to execute that step and the next step simultaneously (e.g., speaking while waving).
Main Hardware Loop (lines 217–256) — Connects to BonicBot, warms up Ollama model, prompts for user instructions, fetches validated choreography plans via ask_choreographer(), and executes physical movements on the robot.

Expected Output

Click to see expected output

Visual Output:

📌 Note: This visual demonstration is using the ROS 2 simulation with use_real_camera:=True.

Terminal Output:


✅ Connected. Type an instruction, or 'quit' to stop.
   Example: "act shy and say hello"

Instruction> act shy and say hello

📋 Shy Hello
{
  "title": "Shy Hello",
  "steps": [
    { "action": "neck", "speech_text": null, "neck_pose": "look_right", "arm_side": null, "arm_pose": null, "move_action": null, "duration_seconds": 1.0 },
    { "action": "arm", "speech_text": null, "neck_pose": null, "arm_side": "left", "arm_pose": "hug_self", "move_action": null, "duration_seconds": 1.5 },
    { "action": "speak", "speech_text": "Oh... um, hello there.", "neck_pose": null, "arm_side": null, "arm_pose": null, "move_action": null, "duration_seconds": 1.5 }
  ]
}

🎬 Performing: Shy Hello

  🤖 bot.look_right()
  🤖 bot.move_left_arm(...)  # hug_self
  🗣️  BonicBot says: "Oh... um, hello there."

✅ Performance complete.

BonicBot’s neck and arm physically move this time, in the order the LLM planned — and each 🤖 line shows the exact Bridge SDK call actually being made.

🔧 Under the Hood

Why does “arm” need a lookup table, but “neck” and “move” don’t?

neck_pose values (look_left, look_center, look_right) and move_action values (move_forward, move_backward, turn_left, turn_right) were deliberately named to match the SDK’s real method names exactly. That means getattr(bot, neck_pose)() calls the right method with zero translation — the schema value is the method name.

arm_pose couldn’t work that way. There’s no bot.hug_self() method — a physical arm pose is just a (shoulder, elbow) angle pair, and “hug_self” is a human-friendly label the choreographer needs so it can reason about emotion rather than raw degrees. ARM_PRESETS is the piece that bridges that gap:


shoulder_deg, elbow_deg = ARM_PRESETS[arm_pose]

Because arm_pose is locked to this exact dictionary’s keys in the Pydantic schema, the LLM can only ever request a preset this table actually knows how to translate — the schema and the translation table stay in sync.

What happens if one gesture fails partway through a performance?

Each step in perform_choreography() is wrapped in its own try/except BonicBotError:


try:
    ...
except BonicBotError as e:
    print(f"  ⚠️ Couldn't perform this step: {e}")

A single failed arm move or a momentary connection hiccup only skips that one step — the loop moves on to the next one instead of crashing the whole performance. That matters here specifically because this lesson is the first one actually touching hardware: a network blip, a servo at its limit, or a robot that briefly lost connection are all real possibilities now, in a way they weren’t when every step was just being printed.

Student Challenge

Add a "bow" preset to ArmPose and ARM_PRESETS — pick (shoulder_deg, elbow_deg) values that stay within the documented joint limits (shoulder −45°–180°, elbow 0°–50°) — and add a one-line description of it to SYSTEM_PROMPT so the choreographer knows it exists. Since a bow usually wants both arms moving together and each arm step only controls one side, you’ll want the LLM to plan two consecutive arm steps (one left, one right) to make it read as a real bow.

Reflection Question

perform_choreography() catches BonicBotError around each individual step instead of letting one failure crash the whole performance. Why does that matter more here than it did in Lesson 17, where nothing physical was connected?