Everyone with a Twitter account is watching the frontier model race. Meanwhile the last twelve months of hardware quietly rewrote what a product's AI stack should look like.
Snapdragon 8 Elite Gen 2 in flagship Android phones. Snapdragon X Elite / X2 Elite in Copilot+ PCs. Apple's M4 and M5 with the beefed-up Neural Engine. Intel Meteor Lake and Lunar Lake with an NPU that finally counts. MediaTek Dimensity 9500. Every one of them ships with an accelerator sized for LLMs — not for the vision models of 2019, for actual language models running now.
You can run a good 8B-parameter model on a mid-range laptop. You can run a 3B model on a phone with sub-second first-token latency. Neither of those sentences was true two years ago.
Why this matters for the products we build
Three things collapse when inference goes on-device:
- The bill. Zero API cost. That's not "cheap." That's zero. It changes the pricing math of every product that has a per-request AI feature.
- The privacy story. The prompt and the response never leave the device. For regulated data — health, legal, financial, HR — that is not a nice-to-have; it is what makes the feature legal to ship.
- The latency floor. No network round-trip. First-token latency measured in a hundred milliseconds. Cloud can't touch that, ever.
Where on-device still loses
It's not free of tradeoffs. Cloud still wins on:
- Anything that needs a frontier model. GPT-6, Claude Opus 5, Gemini 3 Ultra — those aren't running on a phone.
- Agentic tool use with dozens of external integrations.
- Freshness. On-device models are frozen; retrieval helps but the model itself can't learn about last week.
- Multi-user aggregation. If the whole point is comparing signals across a user base, the compute lives centrally.
The pattern we now default to
When we Embed for a client, the first architectural question is: which of your AI features can run on the device? Not "should any." Which ones. The answer used to be "none." It is now, for most consumer and prosumer products, "several."
Practically that looks like: on-device Gemini Nano / Apple Foundation Models for keyboard suggestions, on-device summarization, on-device search over local files, on-device voice input classification. Cloud for anything that needs the flagship, plus fallback for older devices.
The AI stack is bifurcating. Clever cloud for the hard problems, cheap silicon for the boring eighty percent. The product teams that notice ship faster and cheaper than the ones still routing every autocomplete through a Bedrock endpoint.
What to actually try this week
Pick one feature that currently makes an LLM call. Prototype it against an on-device model — Gemini Nano AICore on Android, Apple Foundation Models on iOS 26+, ONNX Runtime with Phi-4 or Llama 3.2 on desktop. Measure token/s and quality on your real inputs. Then look at the API bill you just deleted.
The savings pay for a lot of engineering.