Connect models to a camera stream
A ROS image subscriber stores the latest camera frame. A timer invokes the processing pipeline, while CvBridge handles conversion between ROS images and the model inputs.
PROJECT 02 / ROBOT PERCEPTION
Semantic Perception Workspace.
Give robot images a semantic layer.
A ROS integration that turns camera images and text prompts into object masks and labels using Grounding DINO and Segment Anything.
- ROS
- Grounding DINO
- SAM
- PyTorch
IMAGE + LANGUAGE → OBJECT REGIONSSCHEMATIC
01 / OVERVIEW
The idea.
This workspace connects language-guided vision models to a robot’s camera stream. Grounding DINO finds objects named in a text prompt; Segment Anything turns the detected boxes into masks. The ROS wrapper publishes the resulting mask image and object labels for other nodes to consume.
- Text-prompted object detection
- Instance masks and semantic labels
- Outputs published as ROS messages
02 / IMPLEMENTATION
How it works.
- Image + text prompt
A camera frame and the objects to look for.
- Grounding DINO
Text-conditioned detection produces boxes and labels.
- Segment Anything
Detected boxes guide instance-mask prediction.
- ROS outputs
A colored mask image and timestamped labels are published.
The diagram follows the image-to-mask ROS wrapper in the public workspace.
Use language to specify the objects
The launch configuration provides the text prompt, model checkpoints, and detection thresholds. Grounding DINO supplies candidate boxes and labels, and the SAM predictor generates a mask for each box.
Publish outputs with image context
The wrapper produces a colored mask image and an array of object labels. Both outputs carry the input image header, so downstream nodes retain the camera frame’s timestamp and frame information.
PROJECT CONTEXT
An integration workspace built around existing Grounding DINO and Segment Anything models. The project focuses on the ROS interface, configuration, and image-to-mask processing flow.
03 / CODE
Explore the code.
A few useful places to start in the public repository.