A Glass-Box Robot: Tell It What to Do, Watch It Think

Maker Faire Bay Area 2026

September 25th-27th

Walk up to my robot and tell it what to do: "follow that person," "push the ball to the wall" and then watch it think. Next to the robot, a screen shows everything happening inside its head in real time: what the camera sees, what the LiDAR detects, how your words got turned into a plan, and each step executing live. When it can't find the ball, you see that too. Kids watch the robot move; adults get pulled into the screen.

This started because my daughter got fascinated with self-driving cars and kept asking if we could have a robot at home. So I built one on a TurtleBot 4 in our living room and our house is the test course: couch, chairs, her toys everywhere. These days it earns its keep shuttling small laundry bags down the hallway on a 3D-printed hook. Under the hood it runs ROS 2 with YOLO for object detection, LiDAR-based path planning, and a small task language I wrote called ToyScript that turns commands into robot behaviors. The part I'm most proud of is the "glass box" panel: most robot demos show you what the robot is doing, and this one shows you why. When you give it a command in plain English, the system parses it, picks a plan, and streams its reasoning to the screen before the robot moves.

Everything runs self-contained — the robot doesn't need venue WiFi or any infrastructure to do its thing. At the booth, visitors can give the robot their own commands — "find the green ball," "push the black one to the wall" and watch the reasoning panel work out what they meant. They can place objects in the arena for the robot to find, and it can also follow a person around the arena and carry small items.