LocateAnything-3B on the Jetson Orin Nano

I got NVIDIA's LocateAnything-3B — a 3-billion-parameter open-vocabulary vision-language model — running fully on-device on a Jetson Orin Nano. No cloud, no internet. Everything local.

And honestly? It just works.

You type what you're looking for — "person, cup, laptop", or even a phrase like "the lamp that's turned ON" — and it finds it live on the camera feed. No training, no fixed class list. It even picks up on state, not just object names. Change the words and the next frame adapts.

Somewhere in the middle of testing, it clicked how genuinely useful this could be day to day — point it at your doorstep and get a ping when a package arrives (or goes missing), keep an eye on a specific shelf or item, or get an alert when something's state changes. All running privately on a small local device, with no cloud watching your space.

Live open-vocabulary detection on the Orin Nano

A couple of things that stuck with me

The model. LocateAnything-3B is really good at open-vocabulary grounding — natural language, relational phrases, even object state, with clean boxes.
The board. The Orin Nano quietly holds a 3B VLM in memory at 4-bit and keeps the GPU busy, all within an 8 GB budget. For something this small, that's impressive.

The models

Part	What it is
MoonViT	Native-resolution vision encoder — tiles the image at its real aspect ratio instead of forcing a square, which matters for box precision. Runs in fp16, not quantized. ~310 ms at 640 px.
Qwen2-3B	Causal LM decoder with a custom box head. This is the part quantized to 4-bit, and where ~68% of the runtime goes.
Parallel Box Decoding	Emits a whole box per step instead of one token at a time. It's why a 3B model is viable here at all — and why the attention backend matters, since the block mask needs a kernel that supports it.

It hands coordinates back as integers in [0, 1000], normalised regardless of image size, so I scale them to pixels with x_px = x_int / 1000 * width.

A few things it actually took to make it run

4-bit (NF4) quantization to fit a ~7.7 GB model into the ~6.5 GB you really get on an 8 GB board
Swapped the default Hopper-only attention kernel for SDPA + FlashAttention-2, which the Orin's Ampere GPU actually supports
Pre-quantized the weights and freed system memory (page cache + desktop) to get past load-time OOMs
MAXN_SUPER power mode — ~2.7× faster than the default profile on this board
Decoupled inference from rendering plus MOSSE optical-flow tracking, to hold ~25 fps video — the thing that makes it feel smooth end to end

Quantization

python

from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    # bf16, not fp16. On sm_87 the fp16 compute path measured 1.8x SLOWER
    # for these 4-bit kernels: 95 ms/token against 54 ms/token.
    # This is the opposite of the usual Ampere advice.
    bnb_4bit_compute_dtype=torch.bfloat16,
)

Quantizing at load time holds a 4.7 GB bf16 shard while the quantized weights accumulate — a peak above 5.5 GB that only fits on an otherwise idle board. Quantizing once and saving the ~2.2 GB NF4 checkpoint cuts load from 24 s to 10 s at a far lower peak.

Two things bit me here that look completely harmless. Passing dtype= when loading an already-quantized checkpoint triggers a conversion pass that materialises bf16 copies of every weight — instant OOM. And calling .to(device) on a quantized model raises, because bitsandbytes has already placed the weights during from_pretrained via device_map.

Attention backend

python

# config.json bakes in "magi" (MagiAttention) at the top level, which is
# Hopper/Blackwell only, and leaves text_config._attn_implementation unset.
# Relying on from_pretrained(attn_implementation=...) to propagate is fragile,
# so set it explicitly on all three configs.
config = AutoConfig.from_pretrained(MODEL_DIR, trust_remote_code=True)

config._attn_implementation = "sdpa"
config.text_config._attn_implementation = "sdpa"

# Counter-intuitively, forcing the vision tower to sdpa reproducibly OOMs at
# load -- its mask path allocates a large buffer.
config.vision_config._attn_implementation = "flash_attention_2"

I went and read the model code before settling on this, because it felt like a downgrade: sdpa is a first-class path for the parallel-box-decoding block mask, not a fallback, and fast decoding still works under it.

The visual-token budget

python

# Default is 25600 pre-merge patches, so a large photo produces a visual-token
# sequence whose KV cache OOM-kills generation. 4096 lands at ~1024 visual
# tokens after the 2x2 merge -- about 1 MP, plenty for detection.
#
# Do NOT treat this as a speed knob. At 1024 the model is starved of detail,
# degenerates into a repetition loop, and gets 5x SLOWER (3.11 s -> 15.95 s).
processor.image_processor.in_token_limit = 4096

Getting past load-time OOM

bash

sudo systemctl stop gdm                                   # ~1 GB back
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'      # page cache holds ~3 GB
sudo nvpmodel -m 2                                        # MAXN_SUPER, see below

# Do NOT set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
# It saves ~1.1 GB for small models, but makes this 4-bit load fail outright:
# Tegra's unified memory does not get along with the allocator's virtual
# memory segments at this size. Without it, the load takes ~9.8 s.

When the load fails it reports NVML_SUCCESS == r INTERNAL ASSERT FAILED. That is not a torch bug — it is PyTorch's OOM message-building path calling NVML, which Tegra doesn't support, masking the real "out of memory". If you see that assert anywhere on a Jetson, you are out of memory.

Loading is also non-deterministic: three attempts at an identical 6.53 GB free gave OK, OK, FAILED. A failed attempt leaks ~2.6 GB that gc.collect() cannot reclaim, so retrying in-process starts with less memory each time. Retry as a fresh process instead.

Decoupling inference from rendering

The biggest win I got here wasn't in the model at all. My first loop encoded one frame per inference, so the picture froze for seconds at a time and the whole thing felt broken even though the model was doing exactly what I'd asked.

python

# Render thread: draws whatever the model last produced, at camera rate.
def render_loop():
    while running:
        frame = camera.latest()           # newest frame, never a queued one
        boxes = tracker.advance(frame)    # carried forward between inferences
        publish(draw(frame, boxes))       # ~25 fps MJPEG

# Inference thread: independent, ~0.3-0.6 Hz.
def inference_loop():
    while running:
        frame = camera.latest()
        boxes = parse_boxes(infer(frame, prompt))
        tracker.reset(frame, boxes)

Between inferences a sparse Lucas–Kanade optical flow pass plus a per-object MOSSE correlation filter carries each box forward, so it actually follows its object instead of sitting frozen. Optical flow follows the scene, and a detection box always contains background that drags it — the correlation filter locks onto the object's own appearance instead.

I benchmarked the OpenCV trackers on the board with 3 boxes, and only MOSSE fits a 25 fps budget:

Tracker	3 boxes, this board
MOSSE	12.5 ms
CSRT	88.9 ms
KCF	508 ms

The boxes lag the picture by up to one inference, and the overlay says so — boxes are X.Xs old — rather than letting you assume they're current.

Benchmarks

Measured on-device at 640 px, 4096 patches, MAXN_SUPER.

Component	Cost	Share of a ~3.1 s query
CPU preprocessing	14 ms	negligible
Vision (MoonViT)	~310 ms	~10%
Prefill	~660 ms	~21%
Decode	~54 ms/token	~68%

So I pay about 1.0 s of fixed cost, then ~54 ms per generated token. Output length turned out to be the thing that actually drives the total — roughly 0.33 s per extra box — so fewer categories per query and tighter prompts buy me speed directly.

Things that were supposed to help and didn't

Idea	Expected	Measured
Cut visual tokens 4096 → 1024	prefill scales with tokens, so faster	5× slower. Starved of detail it loops, emitting ~42 junk boxes until the token cap. 3.11 s → 15.95 s.
fp16 compute for the 4-bit kernels	the classic Ampere fast path	1.8× slower than bf16 on sm_87.
TensorRT the vision tower	it's a whole ViT, must be worth it	Vision is only ~10% end to end. An optimistic 2.5× saves ~6%. Not worth it.
Smaller input, 640 → 384 px	fewer pixels, faster	Only 1.13×, and object counts get noisy. Not the lever.
`nvpmodel -m 0`	mode 0 is surely the fast one	On the Orin Nano Super, mode 0 is the 15 W profile with the GPU capped at 612 MHz. Mode 2 is MAXN_SUPER at 1020 MHz: 4.72 s → 1.73 s.

Honest limits

About 3–5 fps is the honest ceiling I could reach for a 3B VLM on this board. 30 fps isn't happening at this memory bandwidth and I'd rather say so than imply otherwise. The Orin Nano has no DLA to offload to, no FP8 (that needs Hopper or Ada), and roughly 6.5 GB of usable unified memory once the OS has taken its share. Every decision above came out of that one number.

What I haven't claimed yet is better W4 kernels — AWQ or GPTQ instead of bitsandbytes should be worth ~1.5× on decode, and CUDA graphs would go after the ~40% of per-token time that isn't bandwidth. Both need aarch64 + sm_87 kernels and an architecture mapping for this custom model, so it's real work rather than a flag I can flip. Haven't got to it.

LocateAnything-3B is NVIDIA's model, for non-commercial and research use. This is a personal project exploring what's possible on the edge.