Pixy — a desk assistant that runs entirely on one Jetson

I had an old portable black-and-white CRT sitting in a drawer, and I couldn't think of a single sensible thing to do with it.

So I gave it a face.

Meet Pixy — a desk buddy running entirely on one NVIDIA Jetson Orin Nano (8 GB). No cloud, no network at run time. She sees, listens, talks back, and mostly just keeps me company while I work.

What she does

• Looks at you — a person detector drives her gaze, so her eyes actually follow you around the desk
• Answers questions about what's in front of her, not just what's in her weights
• Handles the boring stuff out loud — tasks, reminders, a focus timer
• Keeps a journal of the day and compresses it into memory while nobody's around
• Sometimes speaks first

The brain — three tiers sharing one model slot

• A rules router that resolves commands in 0 ms, with no model at all
• NVIDIA Cosmos-Reason2-2B, resident, for everything visual
• Qwen3-4B-Instruct, swapped in on demand for knowledge and tool calls

That swap is the whole trick. A vision model and a good text model don't fit together in 8 GB, so only one is ever loaded — and the reload happens after she finishes speaking, so nobody ever waits for it.

The router matters for a second reason. Deterministic rules pick the route before any model is consulted, which means nothing can be invented at that step. When I let a model decide intent instead, it routinely turned small talk into actions and created tasks nobody asked for. Rules decide first; a model is the fallback.

Everything on the board

Everything is 4-bit. Whisper small stays on disk as a fallback for very short commands, where Parakeet is less reliable.

Why these, and not the others

I tried a 4B vision model first. It matched Cosmos on scene questions but left too little memory behind it — the stack failed while loading speech recognition. The 2B is 1.1 GB lighter for equivalent answers, so it won on the only axis that was actually scarce.

Qwen3-4B-Instruct was the only text model I tested that could call tools at all, while matching the others on knowledge, loading faster, and refusing to invent answers about things that don't exist. Two that didn't make it:

Getting vision fast enough took real effort

The first working version took about 60 seconds to answer a question about what it could see. It now takes 2.1, with 764 MB still free. Four things got it there.

Building llama.cpp for CUDA on the Orin

The stock path had the vision encoder falling back to the CPU, which is where most of that 60 seconds went. Building it myself against CUDA for the Orin's SM 8.7 fixed that:

# nvcc is not on the default PATH on JetPack, and without this the configure
# step fails outright at enable_language(CUDA).
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=87 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc

cmake --build build -j4 \
  --target llama-mtmd-cli llama-server llama-bench

That takes about 35 minutes at -j4.

DirectIO, or it doesn't load at all

The first four attempts died identically, on cudaMalloc failed: out of memory with NvMapMemAllocInternalTagged: error 12 behind it. Going through the page cache, the weights get charged twice — once to the cache, once to the destination buffer — and on unified memory that is the difference between loading and not.

~/llama.cpp/build/bin/llama-server \
  -m  models/cosmos-reason2-2b-Q4_K_M.gguf \
  --mmproj models/mmproj-cosmos-reason2-2b-Q8_0.gguf \
  -c 2048 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  -lm dio

-lm dio reads the GGUF with O_DIRECT straight into the destination buffers and never touches page cache, so the weights are charged exactly once. The context is 2048 rather than something generous because two camera frames only cost about 940 tokens — sizing it to the work rather than to a round number is worth hundreds of megabytes here. The q8_0 KV cache and flash attention are the same kind of trade.

Requantising the vision projector

The projector shipped as F16 at 819 MB. The standard quantiser refuses projector files outright, so I wrote a small GGUF tool to do it: F16 → Q8_0, 819 MB → 536 MB, with answers unchanged.

Making the encoder do less work for the same tokens

This is the one I'm happiest with. I cut the encoder input from 896 to 672 px and moved the pooling factor from 4 to 3, resampling the position embeddings to match. The arithmetic works out so the language model still receives an identical 256 tokens — it cannot tell the difference — while the encoder does 44% of the work it used to.

Where the 8 GB goes

6,247 MB usable, once the OS has taken its share.

Time to answer, by route

A turn goes: wake word detected on-device, capture ends after 1.25 s of silence, speech to text in 0.5 s, then the router. She's capped at two sentences, with Piper doing the voice and the mouth lip-syncing to the output buffer.

Where she looks

Person detection runs on the CPU at 121 ms a frame, re-checked three times a second, with open-vocabulary targets — "my mug", "the blue book" — every 4 seconds.

The offset from frame centre becomes a gaze vector, amplified 2.2× so small movements read across the room, then smoothed with a dead-band so she doesn't jitter. Tracking gives up after 90 seconds, or 12 seconds out of sight — I didn't want her staring at nothing. The same presence signal greets you when you sit down and puts her to sleep when the room empties.

What she keeps

Tasks, reminders and a focus timer all live on disk and are served from Tier 0, which is why they cost nothing. Spoken due times are parsed in code, not by a model — "half an hour", "quarter past six", "next Friday". A background thread watches the clock and announces each reminder at its minute, showing a card on her face as she speaks. The focus timer is a 25/5 pomodoro over four rounds, counting down on screen, and she calls out every phase change unasked.

What she does while nobody's around

The ambient loop only runs while someone is present — about a 7% duty cycle when active. An empty room is never described.

With the room empty she stops looking, shows a resting face, and uses the quiet to compress the day: 46 raw notes down to 27 episodes, with named things kept verbatim.

When she speaks first, the content comes from the real task store and journal — never generated. She can nudge me about a due task but she cannot invent one; the model only picks the wording. That constraint is deliberate, and it's the same instinct as the rules router: the parts that can make things up are kept away from the parts that decide what's true.

Every figure here was measured on the board.