Input to the system includes cameras, GameController and NatNet.
Cameras
The Booster K1 has a stereo head-mounted RGB and depth camera, but we only use one of the two available RGB ones. The camera is owned by the Booster Robotics SDK, which publishes frames as ROS 2 sensor_msgs/Image messages over DDS for us to subscribe to.
Because the Booster Robotics SDK drives the sensor, exposure, gain, white balance and frame rate are not ours to configure. What we do control is which topics we listen to, how frames are converted into our own Image message, and which kinematics each frame is paired with.
The Camera module
The Camera module subscribes to the camera topics using the Booster Robotics SDK and emits an Image for every frame that arrives.
Each entry under cameras in Camera.yaml sets up one subscription.
| Parameter | Type | Description |
|---|---|---|
topic | string | The DDS topic carrying the frames. This is the ROS 2 topic path with an rt/ prefix, so /boostercamera/head/rgb becomes rt/boostercamera/head/rgb. |
name | string | A human-readable name for the camera, carried through on the Image message and used to tell outputs apart. |
id | integer | Numeric camera ID, also carried on the Image message. |
Topics are subscribed to once, at startup. Editing topic while the binary is running logs a warning and does nothing else, so restart to pick up a change.
Frames are read unreliably with a queue depth of one and a latest-only overflow policy. If we fall behind, the stale frame is dropped rather than queued, since a frame we can no longer match to current kinematics is not worth processing.
Pixel formats
The encoding field on the incoming message is mapped to the FOURCC code the rest of the vision pipeline expects.
| ROS encoding | FOURCC |
|---|---|
rgb8 | RGB3 |
bgr8 | BGR3 |
mono8 | GREY |
bayer_bggr8 | BGGR |
bayer_grbg8 | GRBG |
bayer_rggb8 | RGGB |
bayer_gbrg8 | GBRG |
nv12 | NV12 |
Frames with an encoding that isn't in this list are dropped with a warning.
NV12 is the exception to the pass-through. The K1 publishes NV12 by default, which nothing downstream understands, so Camera converts it to RGB3 with OpenCV's cvtColor. The destination buffer is sized inside the Image message first and a cv::Mat is pointed at it, so the converted pixels are written straight into the message with no intermediate copy.
Lens parameters
Lens parameters come from the camera rather than from config. Alongside each image topic, Camera reliably subscribes to <topic>/camera_info, which carries a sensor_msgs/CameraInfo with the intrinsic matrix k and distortion coefficients d. CameraInfo is low rate and we want every update, so this one is taken reliably rather than latest-only.
Those values are converted into the fields our Image::Lens uses. Note that both the focal length and the centre offset are normalised by image width, which keeps them valid if the resolution changes.
| Field | Derived from |
|---|---|
projection | Always RECTILINEAR. The K1's lens is not a fisheye. |
focal_length | fx / width |
fov | 2 * atan(width / (2 * fx)) |
centre | ((cx - width / 2) / width, (cy - height / 2) / width) |
k | The first two distortion coefficients from d, or zero if they aren't published. |
CameraInfo with a zero focal length or zero dimensions is ignored. Until the first valid message arrives, frames go out with a default-constructed lens, so if vision looks geometrically wrong on startup it is worth checking that the camera_info topic is actually being published.
The projection tool, based on panotools' fisheye projection calculations, uses lens.projection to map between rays in camera space and pixels in the image. It also supports equidistant and equisolid projections, but only the rectilinear path is exercised here.
Physical mounting offsets are handled separately, by the Hpc transform in Sensors. Camera mounting varies between robots, so Hpc is calibrated individually for each one.
Timestamps and kinematics
Every image needs to be paired with where the camera was when the shutter fired, not when the frame reached us. The Camera module uses the capture time from the message header for this, and falls back to its own receive time when the header stamp is zero or more than a second away from our clock. A skewed stamp is usually a sign that the Booster Robotics SDK and NUbots aren't sharing a clock, so the fallback logs a warning the first time it happens.
The camera-to-world transform is pulled from a buffer rather than read live. Camera keeps the last 500 ms of Hcw transforms from incoming Sensors messages and picks whichever buffered sample is closest in time to the frame's capture timestamp. Since Sensors and the camera run on independent DDS readers at different rates, this is what keeps the two lined up.
Summary
The Camera module consumes:
message::input::Sensors, to buffer recentHcwtransformssensor_msgs/Imageandsensor_msgs/CameraInfoover DDS
and emits:
It needs a running the Booster Robotics SDK publishing the configured topics. The DDS ChannelFactory is a process-wide singleton shared with the other modules that talk to the robot, so the Camera module initialises it through ensure_channel_factory rather than calling Init itself. NUClear doesn't order on<Startup> reactions between modules, so whichever module gets there first wins and the rest are no-ops.
