On-Camera AI with the Raspberry Pi AI Camera: the short answer
The Raspberry Pi AI Camera uses the Sony IMX500, a sensor with its own small neural network processor built in. The model runs on the camera itself, so the Pi only receives the image plus the detection results.
- Level
- Advanced
- Time
- About 6 min
- Topic
- Raspberry Pi
Run object detection on the Raspberry Pi AI Camera and use the results in your own Python code.
Before this: Raspberry Pi Camera with Picamera2
The Raspberry Pi AI Camera uses the Sony IMX500, a sensor with its own small neural network processor built in. The model runs on the camera itself, so the Pi only receives the image plus the detection results. That leaves the Pi's CPU almost free, and it works on any Pi with a camera connector, including the Pi Zero 2 W and Pi 4.
Install the support package with sudo apt install imx500-all, then reboot. The package includes ready-made models in /usr/share/imx500-models and demo settings in /usr/share/rpi-camera-assets. rpicam-hello -t 0s --post-process-file /usr/share/rpi-camera-assets/imx500_mobilenet_ssd.json shows live boxes around common objects. The first time a model loads, its firmware is copied to the camera, which can take a minute or more; after that it starts quickly.
For your own projects, the Picamera2 examples include IMX500 scripts that read the output tensors from each frame's metadata and turn them into boxes, labels and scores. Treat the results like any sensor: smooth them over a few frames, ignore low scores (start with a threshold of 0.5), and test in the real lighting where the camera will be used. Custom models must be converted with Sony's IMX500 tools and fit in the sensor's small memory, so start with the provided models.
The model runs on the camera
The Raspberry Pi AI Camera has a sensor with its own processor that runs a neural network on each frame, so the Pi receives the image together with the results as metadata. That leaves the Pi's own processor free and works on a small, low-power board. The network is loaded into the camera when your program starts, which takes a short time.
The accelerator in the camera has limited memory and supports a set of model types, so choose a model packaged for it, or convert your own with the tools provided.
Use the metadata
For a detector, the metadata gives each object's class, score and box. Read it in the same loop as the frame, and apply a confidence threshold. You can draw boxes on a preview, count objects crossing a line or trigger an action, without running the model on the Pi itself.
Check that your lighting and viewing angle suit the model, as with any camera model, and test the frame rate and the delay.
When to choose it
It is a good fit when you need modest detection at low power and low cost. For large models, many cameras or heavy processing, a Jetson is the better tool. Consider privacy: because inference happens on the camera, you can send only the results and not the video.
- Expecting any model to run on the camera without conversion.
- Not allowing for the start-up time of loading the network.
- Sending video elsewhere when only the results are needed.
Practise this lesson with Pro
The explanation above is free for everyone. The practice for Advanced lessons in every domain are part of the Pro plan at ₹999 a month.
With Pro you also get: a worked example, a hands-on task, a 3-question check.
