ollwrite on-device — the local-first vision.

01The vision in one line

An on-device, local-first AI writing studio — the model, your memory, and fast semantic search all run on your machine. Nothing leaves it.

ollwrite proves the pattern first. Once it works, the same three-part local stack — inference + a local knowledgebase + on-device search — becomes an oll-am on-device platform, delivered as an installable MVP. The debut is the device install, not a webapp (Sam, 2026-07-03): the Vault promise — pull the network and keep working, everything on your own disk — is something a browser tab structurally cannot make good on. Writing quality is validated separately by the existing cloud write.oll.am; this POC proves offline-native.

This is the deepest expression of the house line — "any AI model, without a subscription — pay once." "Pay once" is only fully honest when there is no per-call bill downstream, and that is exactly what on-device inference gives you: after the model is on the device, editing is free forever.

02Progress review — how much is already built

The single most de-risking fact: the local path is not a green-field build. The provider switch that makes "go local" a config flag is already in the codebase, verified by reading it.

Here is the real boot-gate — the reason the local path needs no cloud secret:

# services/oll-model/config.py  (verbatim, trimmed)
LLM_PROVIDER = os.environ.get("LLM_PROVIDER", "groq").lower()

# REQUIRED_ENV is PROVIDER-AWARE: only the chosen provider's key/URL
# is demanded at boot (the local "pay once" provider needs no cloud key).
if   LLM_PROVIDER == "groq":   _REQUIRED.append("GROQ_API_KEY")
elif LLM_PROVIDER == "claude": _REQUIRED.append("ANTHROPIC_API_KEY")
elif LLM_PROVIDER == "ollama": _REQUIRED.append("OLLAMA_BASE_URL")

OLLAMA_BASE_URL = os.environ.get("OLLAMA_BASE_URL", "http://localhost:11434")

03Research verdict — honest, and it corrects the dream

Numbers below are directional (benchmarks, market sizes). The conclusions are strong enough to design against; the caveats are real enough to respect.

① Model size is the dividing line — 7–8B is the floor, 1–3B fails

The naive dream — "a few tiny models that fit the hardware" — does not survive contact with the benchmarks. Sub-3B models (Gemma 2B, TinyLlama) hallucinate materially (~13.8% vs ~0% for frontier in BLESS human eval) and, damningly for a clarity tool, they lengthen text instead of tightening it — the opposite of the product's whole purpose. Gemma 2B also fell ~3× behind on grammar. The sweet spot is Llama 3.1 8B / Gemma 2 9B / Qwen 8B / Mistral 7B, 4-bit quantized. So "small enough to fit" only works if the smallest is ~7–8B — a genuine constraint (needs ~8GB+ of GPU/unified memory and a multi-GB download), not free.

② WebGPU is a demo, not the product — the debut is a device install

The research liked WebGPU for onboarding: WebLLM runs Llama 3.1 8B (4-bit) at ~41 tok/s on an M3 Max in a browser tab (~71–80% of native), and WebGPU now ships by default in all four major browsers (~83–84% coverage). Zero install, widest reach. But that finding is overruled here — for a decisive reason:

③ The killer feature is the per-rewrite tier switch — not "local" alone

The differentiator is choosing your privacy/quality level per rewrite:

This turns the honest 8B quality gap into a feature: "no, it's not Claude-grade — but it's good enough for most edits, and when it isn't, one click escalates." Local by default; escalate on demand. Note each tier maps one-to-one onto the provider field the gateway already accepts — the tier switch is literally the provider override, surfaced in the UI.

④ A fine-tuned small clarity model beats a generic large one

Grammarly's own CoEdIT is the proof: a 3B fine-tune beat 175B GPT-3-Edit, preferred 64% vs 10% in human eval. The path is therefore prompt-engineer / RAG first, fine-tune a clarity model later — do not ship vanilla Llama and call it done. Raw open models hallucinate too much; the CoEdIT precedent says a dedicated small model closes the gap.

⑤ Economics: on-device cost-to-serve ≈ $0 after delivery

This inverts the competitor model. Grammarly sustains ~80% margins but carries continuous inference cost that grows with usage; every Vault user, by contrast, is pure margin after the sale. Precedent for pay-once sustainability: iA Writer, Scrivener, Hemingway. Price the Vault tier at ~$39–49 one-time inside oll.am (anchored against Hemingway $19.99 and OpenWriter $39.99).

⑥ Competitor & moat

OpenWriter ($39.99 one-time, local llama.cpp) is the only direct competitor — and it's still "Coming Soon." It validates the category without locking it. The moat is structural: incumbents like Grammarly ($700M ARR) can't follow — unlimited on-device rewriting cannibalises their subscription inference revenue (the disruptor's-dilemma wedge). Privacy is an ownable trust angle, with one hard rule from Sam's memory:

04On-device building blocks — Sam's parts, made concrete

• ① Inference. A bundled Ollama sidecar shipping one 7–8B model on local disk — the whole point is that it runs with the network unplugged. It speaks the same oll-model provider contract the cloud tiers use, so product code is identical whether the model is local or remote.
• ② Local vector DB / memory / knowledgebase. sqlite-vec or LanceDB — embedded, no server, fast on-device approximate-nearest-neighbour search over the user's resources + past writings + recall store. Local embeddings through Ollama or ONNX. (The existing local-first-sqlite-store skill already captures this pattern.)
• ③ Personalization / style learning. RAG-based style transfer first — retrieve the user's own passages as few-shot anchors plus a distilled "style card"; fine-tune later. This reuses the humaniz prompt discipline (XML tags, 3–5 few-shot examples, a ban-list of AI-tells, mandated burstiness) documented in the humaniz research (Track 1).
• ④ Citation / verification. The same vector store grounds claims: retrieve the source passages the user added and cite/verify against them — no separate service.
• ⑤ Transcription. whisper.cpp / faster-whisper, local — a later phase, fully on-brand for on-device ("listen to a chat, transcribe, store").

05The architecture insight — local is a SWAP, not a rewrite

Because product logic (write-service) already calls the oll-model gateway over a fixed contract, and the gateway already has the ollama provider, "going local" reduces to two additive moves: swap the model backend (flip the provider) and add a local vector store the UI queries for memory + RAG. The Core contract (auth / billing) is untouched.

The vision is cheap relative to its ambition — because the seams were built in before we needed them.

This is the Clean-Code / dependency-inversion payoff made real: write-service depends on an abstraction (the gateway's provider-agnostic contract), not on a concrete vendor. Swapping Groq→Ollama changes a value, not a shape. That's why an idea this ambitious costs a config flag plus one embedded database, not a re-platform.

06Phased roadmap — respects the ship-vs-build guardrail

1. Phase 0 — NOW, UNCHANGED: the first stranger franc. do this Close the humaniz money path. Nothing below starts until a stranger pays.
2. Phase 1 — the device-install POC (Tauri shell + bundled Ollama sidecar + one 7–8B model + local sqlite-vec store). after franc Wraps the shipped ollwrite editor and runs a local oll-model gateway with provider = ollama — reusing existing code. Proves the one thing the web can't: offline-native writing with local memory.
3. Phase 2 — memory / RAG / style learning on the local store. after franc Add a resource → embed → retrieve; style card + few-shot from the user's own corpus; citation/verify against retrieved passages.
4. Phase 3 — the sellable installable later = offline license (pay-once, validated locally), code-signing / notarization, auto-update = the oll-am on-device platform.
5. Phase 4 — transcription / listen-and-store later (whisper.cpp local).

07Open decisions for Sam

• Sequencing. Confirm on-device is Phase-1+ after the humaniz first dollar (recommended) — not now.
• Debut surface — RESOLVED (Sam, 2026-07-03): device-install POC, skip WebGPU. The browser can't deliver offline-complete + data-sovereignty, which is the Vault promise. See §03②.
• Desktop shell: Tauri (recommended) — tiny binaries, first-class sidecar model for shipping Ollama, reuses the Angular frontend — vs Electron (heavier, but more mature tooling).
• POC model: which single 7–8B to bundle — Llama 3.1 8B vs Qwen 8B vs Mistral 7B (pick one for the POC; benchmark for clarity later).
• Style learning v1: RAG-based (recommended, ships now) vs a fine-tuned clarity model (later, per the CoEdIT precedent).

08Risks & honesty caveats

• Install + download friction — the debut costs a multi-GB app+model download before any value. Accepted deliberately (§03②): it's the price of an offline-native promise the web can't make.
• Hardware floor — a local 8B needs ~8GB+ of RAM/unified memory; below that, the local tier is slow or won't load. Weak devices fall back to the (network-requiring, non-Vault) Balanced tier or are simply not the target.
• Raw open-model hallucination (~13.8% vs ~0%) — mitigated by prompt-engineering now, a fine-tune later (CoEdIT precedent).
• Desktop distribution — code-signing / Apple notarization, model licensing, and auto-update are real Phase-3 work (the POC can be unsigned/self-run).
• The privacy say-do gap — privacy alone rarely sells; pair it with the concrete benefit (unlimited / pay-once / offline).
• Directional numbers — market-size and benchmark figures are indicative, not audited.