Learn / Raspberry Pi / Cameras and vision

On-Camera AI with the Raspberry Pi AI Camera

ReviewedAdvanced ProAbout 6 min
Pi 4
In short

On-Camera AI with the Raspberry Pi AI Camera: the short answer

The Raspberry Pi AI Camera uses the Sony IMX500, a sensor with its own small neural network processor built in. The model runs on the camera itself, so the Pi only receives the image plus the detection results.

Level
Advanced
Time
About 6 min
Topic
Raspberry Pi
You will learn to

Run object detection on the Raspberry Pi AI Camera and use the results in your own Python code.

Before this: Raspberry Pi Camera with Picamera2

The Raspberry Pi AI Camera uses the Sony IMX500, a sensor with its own small neural network processor built in. The model runs on the camera itself, so the Pi only receives the image plus the detection results. That leaves the Pi's CPU almost free, and it works on any Pi with a camera connector, including the Pi Zero 2 W and Pi 4.

Install the support package with sudo apt install imx500-all, then reboot. The package includes ready-made models in /usr/share/imx500-models and demo settings in /usr/share/rpi-camera-assets. rpicam-hello -t 0s --post-process-file /usr/share/rpi-camera-assets/imx500_mobilenet_ssd.json shows live boxes around common objects. The first time a model loads, its firmware is copied to the camera, which can take a minute or more; after that it starts quickly.

For your own projects, the Picamera2 examples include IMX500 scripts that read the output tensors from each frame's metadata and turn them into boxes, labels and scores. Treat the results like any sensor: smooth them over a few frames, ignore low scores (start with a threshold of 0.5), and test in the real lighting where the camera will be used. Custom models must be converted with Sony's IMX500 tools and fit in the sensor's small memory, so start with the provided models.

The model runs on the camera

The Raspberry Pi AI Camera has a sensor with its own processor that runs a neural network on each frame, so the Pi receives the image together with the results as metadata. That leaves the Pi's own processor free and works on a small, low-power board. The network is loaded into the camera when your program starts, which takes a short time.

The accelerator in the camera has limited memory and supports a set of model types, so choose a model packaged for it, or convert your own with the tools provided.

Use the metadata

For a detector, the metadata gives each object's class, score and box. Read it in the same loop as the frame, and apply a confidence threshold. You can draw boxes on a preview, count objects crossing a line or trigger an action, without running the model on the Pi itself.

Check that your lighting and viewing angle suit the model, as with any camera model, and test the frame rate and the delay.

When to choose it

It is a good fit when you need modest detection at low power and low cost. For large models, many cameras or heavy processing, a Jetson is the better tool. Consider privacy: because inference happens on the camera, you can send only the results and not the video.

Common mistakes
  • Expecting any model to run on the camera without conversion.
  • Not allowing for the start-up time of loading the network.
  • Sending video elsewhere when only the results are needed.
Advanced lesson · Pro plan

Practise this lesson with Pro

The explanation above is free for everyone. The practice for Advanced lessons in every domain are part of the Pro plan at ₹999 a month.

With Pro you also get: a worked example, a hands-on task, a 3-question check.

Common questions

What will I learn in "On-Camera AI with the Raspberry Pi AI Camera"?

Run object detection on the Raspberry Pi AI Camera and use the results in your own Python code.