Security design · hardening plan · 2026-07-01 · plan only — nothing changed

How we secure the oll.am deployment

Today's setup is honest-but-provisional: stage-on-live-URLs, every container sharing Coolify's default coolify network. This is the audit of what's already right and the prioritized, actionable plan for what a properly hardened production looks like — so prod bakes in every learning from stage instead of inheriting its shortcuts.

The frame — stage discipline, promoted to prod

A read-only audit of the real repo. No infra was touched, no secret read, no VPS accessed. Sources are cited by file.

The dark-factory model already separates merge from deploy: work integrates on stage behind CI gates and only promotes to main deliberately, one service at a time. This page extends that discipline downward into the runtime: the same "only intentional, verified things reach production" principle applied to secrets, network reachability, and public surface.

The honest headline: Core is already in decent shape. The gaps are mostly operational — secrets that must be rotated, a shared network that gives zero isolation, and internal services that leaked public URLs. None of it requires a rewrite; it requires a rotation runbook, a network topology, and a boot-gate flag. Every item below is risk → what to do → why, and the two lists at the end keep this honest: a prioritized checklist, and a plain statement of what's already good.

Two secrets to treat as compromised right now. The live Stripe key was flagged burned earlier, and OLL_MODEL_SERVICE_TOKEN was pasted into a chat today. Exposed = burned: both must be rotated before real prod, regardless of everything else on this page.

The target topology — public edge, private core

Today: everything on the shared coolify network, some internal services with accidental public URLs. Target: a dedicated per-environment network where only the public edge is reachable from the internet.
🌐 Public edge
core.oll.am · product edges (write.oll.am, foto) · the landing. TLS at Traefik. These are the ONLY internet-reachable apps.
→
🔒 Private core
oll-model gateway · ollama runtime. No public domain. Reached only over the internal network by Docker DNS alias.
→
🗄️ Data (off-box)
Neon Postgres, one branch/DB per env. Reached over TLS with per-service credentials. Never another service's DB.

Everything that follows is in service of this picture: a small public surface, a private interior, and credentials scoped so one leak can't open everything.

The plan — ten concrete items

Each carries a priority pill (P1 = do before real prod · P2 = soon after · P3 = nice-to-have). The consolidated checklist is at the bottom.
01

Rotate every secret; store as Coolify secrets; per-service tokens

P1
Risk
The live Stripe key is burned and OLL_MODEL_SERVICE_TOKEN was pasted into a chat today — treat as compromised. Worse, that token is a single shared secret across every product (services/oll-model/routes.py checks one OLL_MODEL_SERVICE_TOKEN for all callers): one leak = every product can impersonate every other, and the whole LLM-cost surface is open.
Do
Rotate all security-critical secrets (Stripe live key, STRIPE_WEBHOOK_SECRET, AUTH_JWT_SECRET, the model token). Store them only as Coolify secrets — never in code, never in a committed .env. Give each product its own model token (OLL_MODEL_TOKEN_WRITE, _FOTO, …) so blast-radius is one product, not all. Write a short rotation runbook (which key, where it lives, how to roll it, who to notify) and keep it in site/.
Why
A shared secret makes the blast radius the whole platform; per-service tokens make a compromise containable and individually revocable. Secrets in Coolify (not the repo) means a git leak can't expose them, and rotation becomes a config change instead of a redeploy of source.
02

Isolate the network — a dedicated net per environment

P1
Risk
Today every app — oll-core, oll-model, ollama, humaniz, specview, springular — shares Coolify's default coolify network. That is zero isolation: any container can open a socket to any other. A compromise of one unrelated app (say an old POC) can reach Core's port or the model gateway directly.
Do
Define a dedicated network per environment — a ollam (prod) net distinct from a ollam-stage net, the pattern the compose files already use (docker-compose.local.yml → external ollam, docker-compose.dev.yml → ollam-dev). Attach only the services that must talk to each other, and reach them by the stable Docker-DNS alias (oll-core, oll-model) rather than a container name. Take the legacy POCs off the shared net.
Why
Network reachability is the cheapest lateral-movement control there is. If only Core, the product edges and the model gateway share a net, an unrelated container simply has no route to them — defence in depth beneath the service-token check, not instead of it.
03

Internal services get no public domain

P1
Risk
ollama was handed an accidental public sslip URL, and oll-model, while service-token-gated, is publicly resolvable. An unauthenticated Ollama endpoint on the internet is an open, uncapped LLM — free compute for anyone who finds it and a direct cost-blowup vector.
Do
Give internal-only services no public domain in Coolify — ollama always, and ideally oll-model too (products call it over the internal net by alias, so it never needs a public hop). Publicly exposed apps are only: Core (core.oll.am), the product edges (write.oll.am, the foto edge), and the landing. Published host ports stay dev/test-only — the 5055:5000 / 5113:5003 mappings in the compose files are for local smoke tests and must not exist in prod.
Why
The smallest possible public surface is the biggest single win. A service with no domain and no published port cannot be attacked from the internet at all — the token gate becomes a second line, not the only one.
04

Harden the service-token gate

P2
Risk
The gateway trusts any caller presenting the shared token (_check_service_token in services/oll-model/routes.py). The check itself is sound — it uses hmac.compare_digest (constant-time) and returns empty (not a default) when unset — but a single shared secret means "authenticated" says nothing about which product is calling.
Do
Land the per-service tokens from item 01 so the gateway can attribute + revoke by caller. Confirm the "no default bypass" property holds after refactor (an unset token must fail closed, never allow-all). Treat mTLS or a network-policy allow-list as a P3 follow-on once the token model is per-service.
Why
Identity you can attribute and revoke per caller is the difference between "someone has the key" and "the write-service key leaked — roll it, nothing else affected." Fail-closed on an unset token prevents a misconfigured deploy from silently disabling auth.
05

Prod fails fast on default secrets — APP_ENV flips the gate

P1 · mostly done
Risk
Sam wants every env var to ship a test default for dev velocity — good for stage, dangerous in prod. A known AUTH_JWT_SECRET in production means anyone can forge a valid 72h token for any user. Test defaults for Stripe keys or the service token in prod are silent footguns.
Do
Core already implements this: core/create_app.py reads APP_ENV and raises RuntimeError if AUTH_JWT_SECRET or STRIPE_WEBHOOK_SECRET is missing when APP_ENV=production, and only warns otherwise. Extend the same pattern to the product services + the model gateway (per-service tokens, provider keys), and make setting APP_ENV=production in Coolify a mandatory step in every prod app's config — so test defaults are impossible to ship live.
Why
This reconciles dev velocity with prod safety cleanly: defaults are fine in stage/dev, but a security-critical secret that's still the default in prod refuses to boot rather than serving with a forgeable secret. Fail loud beats fail silent for anything that signs tokens or moves money.
06

Auth / JWT — strong, unique-per-env, validated everywhere

P2
Risk
Core issues a 72h HS256 JWT (core/modules/auth/service.py: sub+email+iat+exp) and every product validates it with the same AUTH_JWT_SECRET. That shared symmetric secret is the entire SSO trust root — a weak or cross-env-reused secret forges the whole platform's identity.
Do
Generate AUTH_JWT_SECRET as a high-entropy random value, unique per environment (prod ≠ stage ≠ dev), stored as a Coolify secret. Confirm every consumer validates signature + expiry (not just decodes). Keep the 72h identity-only token as the recorded chosen state; a shorter TTL or revocable refresh is a separate, off-critical-path nicety.
Why
HS256 is symmetric — the secret both signs and verifies, so it is the crown jewel. Unique-per-env means a stage leak can't mint prod tokens; validating expiry means an old token can't live forever.
07

Rate-limit the public and cost-bearing endpoints

P2
Risk
Magic-link (email spend + user enumeration), checkout (Stripe abuse), and above all the model gateway (uncapped LLM cost) are abuse-and-blowup surfaces. Core's magic-link already has a custom in-memory limiter (core/modules/auth/rate_limit.py, 5/IP/hour), but it's per-process — it resets on redeploy and doesn't share state across replicas — and the gateway has none.
Do
Add rate limiting to checkout and the model gateway too. For anything that runs more than one replica, back the limiter with a shared store (Flask-Limiter + Redis) so the cap is real across instances; the in-process limiter is fine only for single-replica services. Cap by IP for anonymous routes and by token/user for the gateway.
Why
The gateway is the one endpoint where an unmetered request turns directly into money out the door (LLM spend). A shared, per-caller cap is the difference between a bounded bill and a runaway one.
08

Containers — non-root, minimal, healthchecked, no build secrets

P3 · largely done
Risk
A container running as root, on a fat base image, or baking a secret into a build layer widens the blast radius of any RCE and can leak a key into the image history.
Do
Keep the pattern Core already sets and apply it to every service: core/Dockerfile runs as a non-root appuser, on python:3.11-slim, with a HEALTHCHECK hitting /api/health over $PORT, and only a non-secret ARG APP_RELEASE at build time. Audit services/*/Dockerfile for the same three properties; forbid build-time ARG/ENV secrets in review.
Why
Non-root + minimal base shrinks what an attacker can do after a breakout; a real healthcheck lets Coolify pull an unhealthy container before it serves; no build-time secrets keeps keys out of image layers that outlive a rotation.
09

Stripe webhook — signature, idempotency, sole plan-writer

P2 · verify
Risk
The webhook flips users to paid, so a spoofed or replayed event is direct entitlement fraud. If any other code path could also write the plan, the money model loses its single source of truth.
Do
Core already does this correctly (core/modules/billing/): it verifies the HMAC with stripe.Webhook.construct_event, de-dupes by event_id, and _set_user_plan is the only writer of User.plan. The prod task is to keep it that way: create the parallel live webhook at core.oll.am/api/billing/webhook, set its rotated STRIPE_WEBHOOK_SECRET, and guard "sole plan-writer" with a review rule so no future route hand-rolls an entitlement flip.
Why
HMAC verification stops forged events; idempotency by event.id stops a retry storm double-granting; a single writer means the plan state has exactly one, auditable, authenticated source.
10

Backups & data isolation — Neon branch per env, PITR

P2
Risk
A bad migration, an accidental delete, or a compromised credential can lose paying-customer + billing data. Cross-service DB access would also let one breached service read another's data.
Do
Keep the database-per-service on Neon model (Core→oll_core, each product its own DB; services integrate over HTTP only, never another service's DB — ADR-008). Use a Neon branch per environment so prod and stage never share data, confirm point-in-time recovery is enabled on the prod branch, and store each service's DATABASE_URL as a per-service Coolify secret.
Why
DB-per-service is the data-layer twin of the network isolation in item 02 — a breach is bounded to one dataset. PITR turns "we lost the billing table" from a catastrophe into a rewind.

The prioritized checklist

P1 = do before real prod · P2 = soon after go-live · P3 = nice-to-have hardening. This is the doable list, in order.
P1 · before real prod
☐
Rotate the burned secrets — live Stripe key + OLL_MODEL_SERVICE_TOKEN (pasted in chat today), then AUTH_JWT_SECRET + STRIPE_WEBHOOK_SECRET. Store as Coolify secrets only. 01
☐
Per-service model tokens — replace the one shared token; each product gets its own, revocable independently. 01·04
☐
Dedicated network per env — ollam (prod) vs ollam-stage; take legacy POCs off the shared coolify net; reach services by DNS alias. 02
☐
No public domain for internal services — remove ollama's accidental URL; keep oll-model private; strip dev-only published ports. 03
☐
Set APP_ENV=production on every prod Coolify app so the boot-gate requires real secrets (already enforced in Core). 05
☐
Unique, high-entropy AUTH_JWT_SECRET per env — prod ≠ stage ≠ dev. 06
☐
Live Stripe webhook wired at core.oll.am/api/billing/webhook with the rotated secret. 09
P2 · soon after go-live
☐
Rate-limit checkout + the model gateway; move to a shared (Redis) limiter for any multi-replica service. 07
☐
Extend the APP_ENV boot-gate to product services + the gateway (fail-closed on missing token/keys). 05
☐
Confirm every JWT consumer validates signature + expiry, not just decodes. 06
☐
Neon branch per env + PITR on prod; per-service DATABASE_URL as Coolify secrets. 10
☐
Audit every services/*/Dockerfile for non-root + healthcheck + no build secrets. 08
☐
Write the rotation runbook — which secret, where it lives, how to roll it — into site/. 01
P3 · nice-to-have
☐
mTLS or a network-policy allow-list in front of the gateway, once per-service tokens are in. 04
☐
Shorter JWT TTL / revocable refresh — a session-longevity nicety, off the critical path. 06
☐
Structured audit logging of every side-effecting call (webhook, email, checkout) correlated by request_id. —

What stays as-is — already good

This isn't fear-mongering. A lot of the hard security was done during the Core lift, verified in the repo. Naming these keeps the plan honest — they don't need work, they need to be preserved.
✓ Boot-time secret gate exists. core/create_app.py reads APP_ENV and raises in production if AUTH_JWT_SECRET/STRIPE_WEBHOOK_SECRET is missing; _jwt_secret() in auth/service.py also raises rather than defaulting. The prod flag we need is already built — it just has to be applied platform-wide.
✓ Service-token check is constant-time and fail-closed. services/oll-model/routes.py uses hmac.compare_digest and returns empty (no allow-all default) when the token is unset. The gap is that it's shared, not that the check is weak.
✓ Stripe webhook is done right. HMAC-verified via construct_event, idempotent by event_id, and _set_user_plan is the sole writer of the plan. This is the money path's core control and it holds.
✓ Containers are already hardened. core/Dockerfile (and services/foto/Dockerfile) run non-root on python:3.11-slim, with a healthcheck and no build-time secrets.
✓ Magic-link is rate-limited. core/modules/auth/rate_limit.py caps 5/IP/hour with an OOM-safe eviction — the only gap is making it shared across replicas and extending it to more endpoints.
✓ Data isolation is the design. Database-per-service on Neon (ADR-008); services integrate over HTTP, never another service's DB. The plan preserves this and adds branch-per-env + PITR.
✓ Auth is passwordless. Magic-link only — no password store to breach, no credential-stuffing surface.

The setup isn't insecure — it's provisional. The work is to promote stage's shortcuts into prod's guarantees: rotate what leaked, isolate the network, hide the interior, and make prod refuse to boot on a default secret.