Projects / Ideas / Local language model assistant

Local language model assistant

AdvancedEdge AIIdeaReviewed
In short

How do you build a local language model assistant?

Run a small language model on a Jetson Orin so a voice assistant answers without the cloud. You need NVIDIA Jetson Orin Nano Super Developer Kit, NVMe SSD 256 GB (M.2 2230/2242), MAX4466 microphone module and Speaker 8 Ω 0.5 W.

Level
Advanced
Time
About 480 min
Parts
4
An idea, not a guided build yet. Goal, parts and steps; no wiring, code or tests. Copy it to My Garage to plan it.
Goal

Deploy a language model on edge hardware.

How to build it

  1. Pick a model that fits in memory
  2. Quantise it
  3. Run it with a local server
  4. Connect speech in and out
  5. Measure tokens per second

Related projects

Common questions

What parts do I need for the local language model assistant?

NVIDIA Jetson Orin Nano Super Developer Kit, NVMe SSD 256 GB (M.2 2230/2242), MAX4466 microphone module and Speaker 8 Ω 0.5 W.

How long does the local language model assistant take to build?

About 480 minutes for a advanced maker.