An Invitation That Arrived Faster Than My Slide Deck

Not long ago a wonderfully persistent Advisor named Angie Carel, who helps run a local AI empowerment and learning group in her community, talked me into presenting to her group. Angie spends her time helping regular people, not engineers, not researchers, just curious neighbors, get their hands on practical AI outcomes to improve their everyday lives. It is exactly the kind of grassroots technical literacy work that this industry needs more of, and I said yes before I had a talk to give, a slide in a deck, or, as of this writing, an actual scheduled date.

So consider this post the long form version of my own prep notes, written out loud while I still have time to fix whatever it exposes as broken.

The subject was a fairly easy pick the moment I mentioned I work with as well as write about security and AI, because a fair number of the regulars had already read my earlier piece on AI powered swim safety monitoring. The one where an ESP32S3 with a cheap OV2640 camera streams video to a Raspberry Pi 5 running Frigate, a YOLOv8n-pose model does the actual keypoint extraction, a SORT tracker keeps identities straight across frames, and the genuinely clever part, the system treats a sustained loss of pose keypoints as a proxy for someone going under the water and starts a countdown timer before it fires an alert over MQTT to email, SMS, or a physical siren. It is a neat trick. Turning a model’s known weakness, that pose estimators fall apart on submerged bodies, into the actual safety signal.

To be clear up front, since it comes up every time I mention both projects in the same sentence, the swim safety monitor and the project this post is actually about are separate builds, not one distilled from the other. They both grew out of the same nagging curiosity though: what can a camera watching a person’s body position actually be trusted to do, safely and reliably, once you strip away everything else. That curiosity is what led me back to a much smaller, much less life critical version of the same question. Could a person’s pose, on its own, control something in the real world, entirely offline, on one board small enough to sit on a shelf.

The Pose Controller Project

That question is what turned into Pose Controller. It watches your arms and turns specific poses and motions into media controls. Raise your right arm out to the side, skip a track. Sweep an arm overhead, also skip a track, because apparently I designed two ways to do the same thing and never noticed until writing this sentence. Raise both arms, play or pause. Swipe a wrist up or down in front of your body, volume up or down. All of it runs completely offline on a Qualcomm powered single board computer, no network connection required for the vision or gesture pipeline itself. Spotify in this example, still needs the internet to actually stream music, but the thing watching you and deciding what to do about it never leaves the board. But it could also be completely offline by leveraging Linux native MPRIS (Media Player Remote Interfacing Specification) via commands like playerctl.

What I did not fully appreciate when I picked this as the talk topic is that getting the gesture thresholds right would cost me an entire weekend standing in front of a webcam, arms out like an airplane, complaining out loud to Claude Code about camera backends while it patiently suggested I try a different calibrations for three dimensional vectors used to register gestures. It was not, in the sense that a computer vision pipeline was watching a grown man interpretive dance in his living room to generate calibration data. It absolutely was, in the sense that the pipeline eventually worked.

The Hardware, Because Specs Sheets Lie

Everything here runs on Qualcomm powered boards, mostly from Radxa’s Dragon line plus one very new Arduino that arrived later in this story. They were chosen because they expose a real neural processing unit through Qualcomm’s own AI Hub and QNN toolchain rather than making you fight a vendor specific SDK with no documentation.

The primary target is the Radxa Dragon Q6A, built around a Qualcomm QCS6490 with a Hexagon based sixth generation AI engine rated at roughly 12 TOPS, priced anywhere from about 105 to 170 US dollars depending on RAM configuration. The second target, added specifically because someone asked “does this even need this particular chip,” is the Radxa Dragon Q8B, a considerably beefier board built on the Qualcomm SC8280XP (the same silicon family Qualcomm markets as Snapdragon 8cx Gen 3 in Windows on ARM laptops) with a claimed 29 plus TOPS AI engine, priced from roughly 209 to 569 dollars depending on memory.

TOPS numbers on a spec sheet are marketing copy until you run your actual workload. The cheaper Q6A beat the more expensive Q8B on raw per model latency for our original two stage pipeline. The Q8B pulled ahead once we swapped in a newer, heavier single model architecture, and it also ran the entire live application at less than half the CPU load of the Q6A for a nearly identical frame rate. Neither of those results was predictable from the TOPS number alone.

Board	Chipset	Vendor AI Claim	Approximate Price
Radxa Dragon Q6A	Qualcomm QCS6490	12 TOPS	105 to 170 USD
Radxa Dragon Q8B	Qualcomm SC8280XP	29 plus TOPS	209 to 569 USD
Arduino VENTUNO Q	Qualcomm Dragonwing QCS8275	up to 40 TOPS	about 299 USD introductory

Generation One: The Boring, Reliable Two Stage Pipeline

The first working pipeline is a straightforward three model relay, and understanding it matters because it is still the default in the repository today. Nothing here is exotic. It is exactly the kind of composable, swappable design you want when you are not sure yet which model is actually going to earn its keep.

YOLOv8n-det does exactly one job, find the humans. This step exists because MediaPipe’s own bundled detector runs at a fixed 128 by 128 pixels internally, and at ordinary room distance a person simply becomes too small for it to notice, regardless of how well lit or sharp the source image is. Swapping in a proper 640 by 640 YOLOv8 nano detector, quantized to 8 bit weights and compiled for the Hexagon NPU through Qualcomm’s AI Hub, fixed detection outright and gave every downstream stage a much tighter, more reliable bounding box to work from.

BlazePose, distributed through Google’s MediaPipe Pose framework, takes that cropped region and produces the actual skeleton, a set of body landmarks with x, y, and a rough depth value, plus a single confidence score for the whole pose. That single score turned out to be a genuine design flaw. If overall confidence dips even slightly, the model discards every landmark at once, including a perfectly visible arm that had nothing wrong with it. It is efficient, and it is also the reason gesture recognition sometimes felt like it only worked while looking directly at the camera.

OSNet, specifically the tiny x0.25 variant from the Omni-Scale Feature Learning for Person Re-Identification research, is the quiet hero of the tracking stack. It generates a compact appearance embedding for each detected person, and that embedding is what lets someone walk fully out of frame and back in a few seconds later without being assigned a brand new identity. Kalman filtering and IoU overlap alone cannot survive a full occlusion or exit and reentry. Appearance matching can. OSNet runs in plain single precision floating point on the CPU rather than the NPU, deliberately, because it only needs to run occasionally rather than every single frame, and because its official quantization calibration dataset sits behind a gated download that was not worth chasing for a model this cheap to run as is.

On the Radxa Dragon Q6A, with a persistent onnxruntime session, the detector runs in roughly 2.2 milliseconds and the landmark model in roughly 1.1 milliseconds, both measured directly on device. The full loop, including tracking and gesture logic, sustains somewhere around 22 frames per second on real recorded footage. That is genuinely fast. What it is not, is especially reliable at holding onto a single identity, which is exactly the itch that led to generation two.

Pose Into Action, At a High Level

Once a skeleton exists, turning it into a media control command is mostly geometry and patience. Every pose first gets normalized, scaled by shoulder width and centered on the torso, so a person standing close to the camera and a person standing far away produce the same numbers. Left and right get corrected for the camera’s mirrored view so left genuinely means the viewer’s left arm, not screen left.

From there a small state machine watches each tracked person’s arm angles frame by frame. An arm held out to the side past a threshold for a debounced number of consecutive frames confirms as a static pose. A wrist trajectory that sweeps overhead within a rolling time window, or one that moves steadily up or down in front of the body without straying sideways, confirms as a motion based gesture. Every confirmed transition emits exactly one abstract action, never a raw pose stream, which keeps a arm held in the air from firing the same command sixty times a second.

That abstract action then reaches a small MediaController interface that maps it onto real commands sent over Linux D-Bus using playerctl, the same mechanism operating system media keys use. The trigger path from camera to command never touches a network socket. Spotify’s own client may still need internet access to stream audio, but the part making the decision to skip a track lives entirely on device.

This path has now been tested against a real, playing track rather than a mocked one. Spotify ships no Linux desktop client for ARM64 at all, so the board runs spotifyd, a small open source daemon that shows up as a Spotify Connect speaker and exposes the same D-Bus interface. Since the control code only ever talks to that interface, it needed zero changes. Over roughly half an hour of live waving, play and pause, skip, and next gestures all changed what was actually playing, with the dashboard’s now playing panel tracking the title as it changed.

Generation Two: YOLO26 and an Honest Look at Marketing Claims

Ultralytics released YOLO26 in January 2026 as a genuinely new architecture, fully end to end and free of the non maximum suppression post processing step every prior YOLO generation needed, with Ultralytics publicly claiming up to 43 percent faster CPU inference. A single end to end pose model also solves both of generation one’s real weaknesses at once, no separate crop step to mistune, and independent per keypoint confidence instead of one all or nothing gate score for the whole skeleton. It looked like a clean upgrade on paper.

Getting it actually running took more patience than it should have. Qualcomm’s own AI Hub toolchain, the same one that compiled every earlier model in this pipeline without complaint, would not cooperate with YOLO26 no matter how it was coaxed. However, Ultralytics ships its own separate, fully local export path that worked once pointed at the right target, though it took a second pass to get that targeting exactly right on both boards. The full blow by blow, including the specific settings that mattered, is written up in the repository for anyone who enjoys toolchain archaeology. What actually matters for this post is what the model does once it is running, and how it performs.

Once it actually ran, the results against the same recorded test footage used to validate generation one were the best evidence yet that the redesign was worth the trouble. Frames with a successfully detected pose jumped from 49 percent to 96 percent, and a person who had previously fragmented into two separate tracked identities on the same clip held one single continuous identity for the entire recording.

Metric on identical footage	Generation One (YOLOv8 plus BlazePose)	Generation Two (YOLO26)
Frames with a detected pose	451 of 923, 49 percent	890 of 923, 96 percent
Tracked identities for one person	2, fragmented	1, continuous
Gestures correctly triggered	7	14
Approximate frame rate	22.1 fps	12.6 fps

Notice that last row. The newer, architecturally superior, marketed as faster model runs measurably slower per frame on this specific edge NPU, because it now performs one full frame end to end forward pass instead of a small cropped landmark inference. Ultralytics is not lying about their CPU benchmark. It simply is not the workload or the silicon we are running on, which is precisely why an actual live test on the target hardware beats trusting any vendor’s number.

We then took the whole live application, camera capture, re identification, an on device web dashboard, and the media control path, and ran it interactively on both boards for real, not from a script.

Live application metric	Radxa Dragon Q6A	Radxa Dragon Q8B
Frame rate	approximately 9.6 fps average	approximately 9.9 fps average
CPU utilization	approximately 552 percent of 800 percent available	approximately 245 percent of 800 percent available
Resident memory	approximately 251 MB	approximately 279 MB

Almost identical throughput, and the pricier board did it using less than half the CPU headroom. Whether that gap traces back to the newer chip’s NPU driver stack, its memory subsystem, or something else entirely is still an open question.

Here is the actual dispatch table that turns a gesture into a command, trimmed down from the real source for a blog post.

_ACTION_TO_METHOD_NAME = {
    ControlAction.NEXT: "next",
    ControlAction.SKIP: "next",
    ControlAction.PREVIOUS: "previous",
    ControlAction.PLAY_PAUSE: "play_pause",
    ControlAction.VOLUME_UP: "volume_up",
    ControlAction.VOLUME_DOWN: "volume_down",
}

And here is genuinely the entire difference between running the old pipeline and the new one on the same board, a one line change in a YAML file.

inference:
  backend: yolo26

That is the part of the architecture I am proudest of. Neither pipeline is hardcoded into the application. Both live behind the same interface, selected entirely by config, which is what actually let us run a fair side by side comparison.

What a Full Body Skeleton Actually Costs You

One of the more useful questions I got while running an early demo pass for myself, before this ever got anywhere near Angie’s group, was almost embarrassingly simple once someone else finally asked it out loud. Why does the on screen overlay look so much sparser than the usual YOLO pose demo videos with a full skeleton and bounding box drawn over each person. The honest answer was that the original design only ever needed shoulders, elbows, wrists, and hips to run a gesture recognizer, so nothing else got wired up to the display, even though the underlying model had already computed every point.

Both YOLO26 and the intermediate HRNet based pipeline compute a full seventeen point COCO style skeleton, eyes, ears, knees, and ankles included, in that exact same single inference call already discussed above. The NPU itself does not do one extra unit of work to make that data available. What does change is the small amount of Python side decoding and drawing needed to actually surface all seventeen points plus a bounding box instead of nine points and nothing, and that is genuinely not free, just cheap.

It is also, in fairness, a real variable I have to own up to. The full skeleton and bounding box change landed in between the two live board tests referenced in the performance table above, which means the Q6A numbers were captured on the narrower nine point overlay and the Q8B numbers were captured after the fuller seventeen point version was already running. The frame rate and CPU gap between the two boards is real and worth investigating, but it is not yet a clean, controlled comparison, and that same sharp question about the skeleton is exactly what surfaced the discrepancy. A proper re run with identical code on both boards is now sitting near the top of the project backlog, and I would rather say that plainly than let a good story get ahead of the data.

Figure: a synthetic full body pose rendered through the updated overlay code, bounding box included, used to confirm the skeleton and box drawing logic before it got tested against an actual human. Image attached alongside this post.

A Third Board, and Lack of Camera Support

The newest member of the lab is the Arduino VENTUNO Q, an Arduino branded board built around Qualcomm’s Dragonwing QCS8275, with 16 GB of memory, eight efficiency class Cortex-A55 cores, and an NPU Arduino rates at up to 40 TOPS. It also carries a newer Hexagon generation than the Radxa boards, which turned out to matter. I figured it would be a quick port. I plugged in my ArduCam, copied the models over, and prepared to feel clever. Both halves of that plan failed, for opposite reasons.

The camera. Nothing detected it. I reseated the ribbon cable. I flipped the ribbon cable over. The cable was innocent. Once the camera stack was actually asking questions, its log showed it repeatedly probing for a Sony IMX577 sensor at that sensor’s own bus address, and my ArduCam is built around an IMX219, the sensor from the Raspberry Pi Camera Module 2. The board’s vendor camera libraries ship tuning for the IMX577 and a handful of siblings and never for the IMX219. A sensor that works on every Raspberry Pi does not automatically work on every board with a camera connector. That one is parked until Arduino ships broader camera support, and a USB webcam took over.

The NPU. Qualcomm’s AI Hub device farm hosts this exact board, so I compiled YOLO26 there and it reported about 4.7 milliseconds per inference, entirely on the NPU. On the actual board in front of me, the same model refused to start. Every variation I tried failed at the same early step, including a model rebuilt specifically for this chip. The model was fine. Something underneath it was not.

The answer came from somebody else’s work. Samuel Alexander published a depth estimation tool for this board on the Arduino Project Hub, and his repository’s board notes spell out what I had been missing. The board’s own package repository ships Qualcomm’s current AI runtime as two ordinary packages, and a program built on LiteRT reaches the NPU through Qualcomm’s delegate. The ONNX runtime I had been using appears to carry an older private copy of the same machinery, and it could not create a device on this newer chip. I had spent days debugging the model layer when the problem lived one layer down. Two apt packages and a different loader later, I had a lesson in why reading other people’s project notes beats solo toolchain archaeology.

Same recorded footage	Frames with a pose	Tracked identities	Processing speed
Dragon Q6A, YOLO26 on the NPU	96 percent	1, continuous	about 12.6 fps
VENTUNO Q, MediaPipe on the CPU only	97 percent	3, fragmented	about 12.4 fps
VENTUNO Q, YOLO26 on the NPU	97 percent	1, continuous	about 32.3 fps

The NPU model itself measured 8.7 milliseconds per inference on the real board, close to AI Hub’s own 8.4 millisecond estimate, and the same pipeline runs at nearly three times the frame rate the Q6A manages. The CPU only row is worth a second look. Detection rate alone barely changed, but the tracking fell apart into three identities, which is exactly the failure that sent this project to YOLO26 in the first place.

Some honest caveats, because this post keeps promising them. Those are processing speeds on recorded footage, not the full live application, which I have not measured on this board yet. This model is a floating point export while the Q6A runs a quantized one, and on this footage it triggered 10 gestures where the Q6A triggered 14, a gap I have not explained. And there is a loose thread that may apply to the other two boards: the same notes found that simply setting the NPU’s performance mode took one model from 58 milliseconds to under 13. My Radxa test sessions never set it, so those earlier numbers may have been measured at low clocks.

Where This Leaves the Talk, and Anyone Else Following Along

I still have not nailed down a date with Angie, but my working bet is that this lands with her group in a way the swim safety pitch never quite could on its own. Everyone can picture themselves waving an arm at a webcam to skip a song. Fully offline, genuinely private, real time computer vision running comfortably on a board that costs about as much as a nice dinner out is the kind of story that lands with a general audience in a way some advanced cybersecurity use cases never would.

The project remains actively in progress on the dev branch of the Pose Controller repository, including the full architecture writeup, the pipeline and performance reference this post pulled its numbers from, the running project journal that reads like a diary of every mistake made along the way, and a growing backlog of exactly what remains unverified, including that Q6A versus Q8B re test and the NPU performance mode experiment, because pretending a personal project is finished helps nobody who wants to build on it honestly.

If you build something from it, or you find a bug I haven’t, open an issue. And if you happen to catch a grown adult sweeping their arm overhead in front of a webcam this weekend, in the name of collecting a few more gesture samples before a talk that does not have a date yet, please do not knock to check if everything is alright. It is, mostly. I hear it builds character. It has definitely built a dataset.