Run an LLM in your browser — no install, no server

    The short answer

    With WebGPU, a modern browser can run quantized LLMs — Llama 3.2, Qwen 2.5, SmolLM2 — directly in a tab, on your own GPU. ShadowTalk AI turns that capability into a product: a device-tiered model manager, resumable per-shard downloads, a watchdogged inference worker, and automatic escalation to cloud models when a task is too heavy — free.

    Browser LLMs spent years as demos. This page explains how they actually work, which models fit which machines, and what separates a packaged product from a proof-of-concept — with the specific engineering ShadowTalk ships for each gap.

    How a browser runs a language model

    Three pieces make it work. First, WebGPU: the browser exposes your GPU's compute shaders to JavaScript, which is what makes billion-parameter models interactive on consumer hardware. Second, an inference engine — ShadowTalk uses MLC's WebLLM — running in a background Web Worker so the interface stays responsive while the model thinks. Third, storage: model weights stream down as shards and are cached per-shard in the browser's Cache API, which is what makes downloads resumable and repeat visits instant.

    • WebGPU compute shaders execute the transformer on your hardware — prompts never leave the device
    • The engine runs in a Web Worker: the UI keeps 60fps while tokens stream
    • Per-shard caching means a dropped connection pauses the download; it resumes, never restarts
    • GPUs without the shader-f16 feature automatically get q4f32 quantized model twins — the swap happens at activation time

    Models you can run, and what they need

    ShadowTalk's model registry pre-checks your GPU features and storage quota, then offers the tiers that fit. All sizes below are approximate on-device footprints:

    ModelStrengthSize (q4f16 / q4f32)
    Qwen2.5-0.5B-InstructNano tier — modest devices600MB / 700MB
    Llama-3.2-1B-InstructInstant default — fast, capable880MB / 1.1GB
    SmolLM2-1.7B-InstructBalanced everyday chat1.68GB / 2.0GB
    Qwen2.5-Coder-1.5B / 3BCode generation on-device1.5GB / 2.3GB
    Llama-3.2-3B-InstructPro tier — needs 8GB+ devices2.1GB / 2.4GB
    Qwen2.5-3B-InstructPro tier — strongest local chat2.3GB / 2.6GB

    From demo to product: the gaps most browser LLMs leave open

    Running a model in a tab is easy; shipping it is not. What ShadowTalk adds around the raw engine:

    • Tiered, consent-gated background install with storage-quota pre-checks and auto-retry with backoff — offline pauses and resumes cleanly
    • Watchdogged engine lifecycle: 45s init watchdog, generation watchdogs, and a recovery ladder that caps worst-case wait at about three minutes
    • A tested hybrid router (30+ unit-tested cases): small talk and quick replies stay on-device, heavy tasks escalate to your chosen cloud model — Apple-Intelligence-style routing, in a browser
    • A supply chain you can audit: zero third-party runtime calls, self-hosted fonts, and model weights plus engine WASMs mirrored on the project's own git branch — Hugging Face is not on the critical path

    Developer notes

    The stack is Vite + React + TypeScript with @mlc-ai/web-llm 0.2.x, vite-plugin-pwa for the precached shell (10MB-per-file cap, weights excluded by design), and a puppeteer-verified offline smoke test in CI that asserts zero third-party hits under a network cut. The mirrored 0.5B model twins and their model_lib WASMs live on the repo's orphan weights branch, byte-identical to upstream, with an env override for self-hosters.

    Frequently asked questions

    Free to use — no account, no card.

    Works offline. Syncs when you're back. Your keys, your models, your chats.

    Run your first browser LLM — free