Running a 3B vision-language model on an 8 GB board
What actually breaks when you put a MoonViT + Qwen2-3B model on a Jetson Orin Nano — and the threading trick that makes half a frame per second feel live.
Half a frame per second sounds unusable. It isn't — as long as you stop making the camera wait for the model.
The naive loop
while True:
frame = camera.read()
boxes = model(frame, prompt) # ~2 seconds
draw(frame, boxes)This gives you a 0.5 fps video feed. It looks broken, even though the model is doing exactly what you asked.
Splitting the loops
Run two threads. The inference thread always grabs the newest frame, never a queued one. The render thread draws every frame at camera rate and carries the last known boxes forward with an optical-flow tracker.
The result is a 25 fps picture with boxes that are up to one inference stale. The honest move is to say so on the overlay — boxes are 1.8s old — rather than let the viewer assume they're current.