Optimising Models with TensorRT: the short answer
A model trained in PyTorch runs on a Jetson, but slowly compared with a TensorRT engine. TensorRT takes the network, fuses layers, picks the fastest GPU kernels for your exact device and can lower the precision of the maths.
- Level
- Advanced
- Time
- About 6 min
- Topic
- NVIDIA Jetson
Convert a trained model to a TensorRT engine, choose FP16 or INT8, and measure the speed honestly.
Before this: Object Detection with Pretrained Models on Jetson
A model trained in PyTorch runs on a Jetson, but slowly compared with a TensorRT engine. TensorRT takes the network, fuses layers, picks the fastest GPU kernels for your exact device and can lower the precision of the maths. The usual route is: export the model to ONNX from your training code (for example torch.onnx.export, or the export command of tools such as Ultralytics YOLO), then build the engine on the Jetson with /usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.engine --fp16.
Precision is the main speed lever. FP32 is the training precision. FP16 halves the memory and is usually close to twice as fast with almost no change in accuracy, so it is the default choice. INT8 is faster again but needs calibration: a few hundred representative images from your real scene so TensorRT can choose scale factors. Always check accuracy on your own test images after changing precision, not just speed.
An engine is tied to the GPU, the TensorRT version and the input size it was built for. Build it on the device that will run it, rebuild after a JetPack upgrade, and keep the ONNX file as the portable original. Measure with trtexec, which reports latency and throughput, and then measure the full pipeline (camera, pre-processing, inference, post-processing), because inference is often less than half of the total time.
What TensorRT does
TensorRT takes a trained model, usually exported as ONNX, and builds an engine tuned for the exact GPU. It merges layers, picks fast kernels and can use lower precision. The engine file is specific to the device type and the TensorRT version, so build it on the Jetson, or on the same type, and rebuild after a software update.
The trtexec tool builds an engine and reports the latency and throughput, which makes a quick first test.
Precision and accuracy
FP16 usually gives a large speed-up with almost no loss in accuracy. INT8 is faster again and smaller, but needs calibration images that represent your real data, and it can cost accuracy, so check the result on a test set. Compare the accuracy of the original and the optimised model, not only the speed.
Fix the input size and the batch size when you can. A dynamic shape is more flexible but may be slower.
Measure properly
Warm up the model before timing, run many iterations, and report the average and the slowest. Time the whole pipeline, since pre-processing and post-processing, such as resizing and non-maximum suppression, can take as long as the model itself.
- Copying an engine file to a different device type or software version.
- Using INT8 with calibration images that do not match the real scene.
- Timing only the model and missing the pre- and post-processing.
Practise this lesson with Pro
The explanation above is free for everyone. The practice for Advanced lessons in every domain are part of the Pro plan at ₹999 a month.
With Pro you also get: a worked example, a hands-on task, a 3-question check.
