SparkEye — poseDetect

Real-time identification and pose tracking of multiple robots from a single overhead camera. Each robot carries a ring of 16 RGB LEDs coloured to form a cyclic de Bruijn codeword, so any sufficiently long arc of consecutive LEDs uniquely determines both the robot's identity and its orientation — even under partial occlusion. The tracker runs on a Raspberry Pi 5 and publishes (id, x, y, theta) pose tuples over MQTT at frame rate.

For the algorithmic details (codebook construction, decoding theory, evaluation), see the SparkEye paper: "SparkEye: Occlusion-Robust Identification and Pose Tracking of Multiple Robots Using Active LED Rings" (paper/main.tex in the parent research folder).

How it works

Every frame goes through five stages (Section "Method" of the paper):

1. Ring detection — a Hough-circle detector finds each robot's LED ring inside a small crop centred on its last known position. If nothing is found, the accumulator threshold is relaxed step by step, so the centre is recovered even from a partial arc.
2. LED sampling — the 16 LED sites are sampled on an annulus around the detected centre and each is classified into one of 4 colours (yellow / green / cyan / magenta) by median hue.
3. Decoding — the observed colour string is matched against the codebook (database.py, 123 codewords) over all cyclic rotations. The best match gives the robot's identity; the winning shift gives its front LED and therefore its heading.
4. Filtering — identity is stabilised by a running majority vote, heading by a trimmed circular mean, and position optionally by a constant-velocity Kalman filter.
5. Output — the pose list is published over MQTT (and optionally logged to CSV).

Repository layout

File	Role
robot_traj.py	Main tracker (runs on the Pi). Detection, decoding, filtering, MQTT publishing.
listener.py	Operator UI (runs on a laptop/server). Setup confirmation, live overlay of the pose stream.
const.py	All configuration: camera, detection, LED ring, smoothing, toggles.
camera.py	Frame source: Picamera2 (Pi), USB webcam, or video file.
database.py	The codebook — 123 cyclic codewords of length 16 over 4 colours.
mqtt_send.py	MQTT client used by the tracker (broker address is set here).
KF.py	2D constant-velocity Kalman filter for position smoothing.
logger.py	Optional per-frame CSV logging.
util.py	Angle math, cyclic-string scoring helpers.

System architecture

 Raspberry Pi 5 + Camera Module 3          laptop / central server
 ┌───────────────────────────┐             ┌──────────────────────┐
 │  robot_traj.py (tracker)  │ ── MQTT ──▶ │ listener.py (UI)     │
 │  detection + decoding     │ ◀── MQTT ── │ setup clicks, keys   │
 └───────────────────────────┘             └──────────────────────┘
                 ▲  both connect to an MQTT broker (mosquitto)

MQTT topics:

Topic	Direction	Content
arena/Loc	Pi → UI	Setup requests (setup:{...}) and the live pose stream [(id, x, y, theta_deg), ...]
arena/image	Pi → UI	JPEG frames (setup frame, then optional debug stream)
arena/imgIO	UI → Pi	Operator reply: confirmed centres, or -1 to reject
arena/imgMode	UI → Pi	Toggle the debug image stream on/off

Requirements

On both machines:

pip install opencv-python numpy paho-mqtt

On the Pi additionally (Picamera2 is installed via apt, not pip):

sudo apt install python3-picamera2

An MQTT broker must be reachable by both machines, e.g. mosquitto:

sudo apt install mosquitto        # then: sudo systemctl start mosquitto

Set the broker address before running — it is hardcoded in two places: BROKER in listener.py and the connect(...) call in mqtt_send.py (default 192.168.11.100:1883).

Running

1. Start the broker on the central server (or anywhere both machines can reach).

2. Start the operator UI on the laptop:

   python listener.py
   

3. Start the tracker on the Pi:

   python3 robot_traj.py
   

4. Setup handshake (in the listener window):

   • (Optional — only if EXPOSURE_REF_ID >= 0 in const.py) Exposure calibration: the Pi first sends a frame labelled "Click ONE reference circle (exposure)". Click the ring of the robot whose ID equals EXPOSURE_REF_ID, press Enter, and the Pi steps the exposure down from 20 ms until that ring decodes reliably, then locks it for the session. Auto-exposure usually overexposes LEDs and washes their hues to white, which is why this step exists.
   • Robot confirmation: the Pi sends a frame labelled "Confirm all robots" with its auto-detected ring centres pre-loaded. Edit them with the mouse (see controls below) and press Enter. Tracking starts on the confirmed set — this pins the number of tracked targets so spurious detections cannot enter the tracker.

5. The listener now prints and draws the live pose stream. Positions are in image pixels; theta is the heading in degrees (image coordinates, 0° = +x, clockwise-positive).

Listener controls

Input	Action
Left-click	Add a circle (empty space) or move an existing one (on a circle)
Right-click	Remove a circle
Enter	Confirm the current centres → tracking starts
n	Reject this frame; the Pi grabs another one
d	Toggle the Pi's debug image stream (off saves bandwidth/CPU)
Esc	Quit

Offline testing (no Pi, no robots)

The whole pipeline runs on a laptop against a recorded video or a USB webcam. In const.py set:

CAM_SOURCE = "path/to/recording.mp4"   # or an integer webcam index

Everything else works the same — you can run the tracker and listener on the same machine with a local mosquitto broker (BROKER = "localhost").

Key configuration (const.py)

Setting	Meaning
R	Ring radius in pixels — must match your camera resolution / mounting height (~100 px at 4608×2592, ~60 px at 1080p). Most detection geometry derives from it.
CAM_SOURCE, CAM_W, CAM_H	Frame source and resolution.
CAM_AUTO_EXPOSURE, CAM_EXPOSURE_US, CAM_GAIN	Auto vs. locked manual exposure. A short locked exposure gives crisper LED colours.
EXPOSURE_REF_ID	-1 disables the click-to-calibrate exposure step; >= 0 enables it and names the reference robot's ID.
LIGHT	False = dark-scene detection (grayscale threshold); True = bright-scene mode (saturation channel).
MQTT	True for the normal Pi + listener deployment.
LOG_CSV	Write per-frame results to robot_frames_log.csv (columns defined in logger.py).
USE_KALMAN_FILTER	Kalman-smooth positions (off by default; poses are still majority/circular-mean filtered).
POSE_WIN, POSE_KEEP_FR	Heading smoothing window (20 frames) and trim fraction (keep the 60% closest to the newest).
HUE_BANDS	Empirical hue intervals for the four LED colours — recalibrate if you change LEDs or camera.

Adding robots

Robot identities are rows of SEQUENCES in database.py (0 = cyan, 1 = magenta, 2 = yellow, 3 = green, read clockwise). Flash a robot's LED ring with an unused codeword and it is immediately trackable — the codebook supports 123 distinct identities with no synchronisation or blinking required.