What is on-device AI? The plain-English explainer

    The short answer

    On-device AI means the model runs on your own hardware instead of a cloud server: your prompt, the context and the reply never leave the machine. In a browser this works through WebGPU — your GPU executes a quantized model (roughly 600MB-2.6GB) inside the tab. ShadowTalk AI is a free workspace built around it: on-device chat with zero network egress, offline queueing and automatic sync, plus cloud models when a task needs them.

    "On-device" is the phrase behind private AI, offline AI and edge AI — and it's often used loosely for products that still phone home. This page defines it precisely, shows how a browser can run a model, and lays out what on-device AI is genuinely good for and where cloud models still win.

    On-device vs cloud AI

    The two architectures differ in where the model physically runs — everything else follows from that:

    AspectOn-device AICloud AI
    Where inference runsYour CPU/GPU — in a browser via WebGPUThe provider's servers
    What leaves your deviceNothing (zero network egress)Your prompt, context and identity
    Works offlineYes, after a one-time model downloadNo
    Model capabilityPractical ceiling ~3B-class in a browser tabFrontier models, hundreds of billions of parameters
    Cost of inferenceYour electricity — nothing elseMetered or subscription
    LatencySpeed of your hardware, not your signalNetwork round-trips

    How a browser can run a model

    Three pieces make it work. WebGPU exposes your GPU's compute shaders to JavaScript. An inference engine (MLC's WebLLM class) executes the transformer in a background Web Worker so the interface stays responsive. And model weights stream down as shards cached in the browser — which makes downloads resumable and repeat visits instant. ShadowTalk wraps that capability with a device-tiered model registry, quota pre-checks and watchdogged workers.

    The privacy math

    On-device chat has zero network egress: no prompt, no context, no metadata leaves the tab, so there is nothing to intercept or store server-side. That is the tier where 'nothing sees my prompt' is literally true. Stored history is a separate question — ShadowTalk's Private Mode encrypts chats end-to-end (AES-256-GCM) with keys derived on your device, and default chats are stored server-side when Private Mode is off. Both facts are documented, because privacy claims die in the fine print.

    What on-device AI is good for (and not)

    Good for: everyday chat, drafting, summarizing, code help at small scale, anything sensitive, and any situation with bad or no connectivity. Not (yet) for: frontier-level reasoning, very long context, or the newest model releases — those need cloud models, which is why ShadowTalk routes heavy tasks upward automatically instead of pretending a 3B model is a frontier model.

    Frequently asked questions

    Free to use — no account, no card.

    Works offline. Syncs when you're back. Your keys, your models, your chats.

    Try on-device AI — free