oll.am — Release & Deploy Runbook: ship the queued work, reliably

TL;DR

Read this box, then act. The rest is detail for each step.

Goal: ship-mode, not build-mode. Release the queued work, verify it, set env vars, make deploys reliable + repeatable — zero new code.
Reliability right now is GREEN. All four live endpoints return 200 (checked this session — see the strip below).
The single money gate is unchanged, Sam-console only, ~3 min: create the LIVE Stripe webhook at core.oll.am/api/billing/webhook + set STRIPE_WEBHOOK_SECRET on Core. Without it a payment succeeds but the plan never flips to pro. Everything else on this page is secondary to that.

core.oll.am/api/health

● 200 OK

neon ok · stripe ok

model.oll.am/api/text/health

● 200 OK

default=groq · groq ok · claude/ollama skipped

humaniz.me/api/health

● 200 OK

up (see note)

oll.am

● 200 OK

landing live

Stale-status correction: state.json's "humaniz DOWN 503 / 27 restarts" is STALE — humaniz is UP (humaniz.me/api/health returned 200 this session).

Money gate · do first

🔴 Create the LIVE Stripe webhook — this is the whole ballgame

Sam console, ~3 min: add a live Stripe webhook endpoint at core.oll.am/api/billing/webhook and set STRIPE_WEBHOOK_SECRET on the Core Coolify app. Without it, a real payment goes through but the account never flips to pro — the customer pays and gets nothing. Price is already set; this is the last wire. Nothing else on this page unlocks a paying customer if this isn't done.

What changed — the new release

Five commits queued on stage, not yet on main / not deployed. This is the "new release" available to promote.

PR	Service	What	Deploy risk
#48	foto	Real LoRA pipeline lifted verbatim from wardrobai; provider-switched (mock default = zero network/spend; replicate provider triple-guarded, pinned model ids left as a documented TODO).	none foto not deployed yet; P2, after the first franc
#47	oll-model (CI)	Keyless mock provider + docker boot+health gate in the quality-gate workflow.	none test-only hardening
#54	write (+CI)	write boots keyless (gateway-aware boot-gate; no `ANTHROPIC_API_KEY` needed when `OLL_MODEL_BASE_URL` is set) + a docker-boot gate for all product services.	none
#55	oll-model	Bundle Ollama as a NON-DEFAULT switchable provider (dev harness in `docker-compose.dev.yml`); groq stays the deployed default.	none ollama stays "skipped" in prod unless `OLLAMA_BASE_URL` is set
#56	e2e	Repeatable Ollama/groq model-path harness (the dev-tested gate).	none

Honest note: these 5 are CI-hardening + a dev-only Ollama harness + foto (not-live). None is itself the money path. The money path is the Stripe webhook (above) + humaniz going keyless-live (Tier 3 below).

Your to-do — ordered, stop at first blocker

Do these top-to-bottom. Tier 0 is the only thing between you and a paying customer; the rest makes shipping reliable + repeatable.

TIER 0 The money gate Sam console · ~3 min

Create the LIVE Stripe webhook → core.oll.am/api/billing/webhook.
Set STRIPE_WEBHOOK_SECRET on the Core Coolify app.
Verify: a real-card CHF payment flips a test account to pro.

Do this first. Without it, payment succeeds but the plan never flips.

TIER 1 Make deploys reliable + intentional Sam console · one-time

Turn Coolify git-auto-deploy OFF on the 4 service apps (oll-core · oll-model · foto · write); leave it ON for the site app.
Merge PR #45 (the gated Coolify-webhook deploy).
For each service you want CD to deploy, set repo secret COOLIFY_HOOK_<SVC> (the Coolify deploy-webhook URL) + repo variable DEPLOY_<SVC>_ENABLED=true.

After that, deploy = run the deploy-all.yml workflow (workflow_dispatch from main) — it fires the webhooks in dependency order T0 Core+Model → T1 foto/write/spec → T2 site, health-gated at each tier.

TIER 2 Release the queued oll-model hardening low-risk · reliability-positive

Promote stage → main for oll-model (#55 + #47), then CD oll-model.
No NEW required env var — groq stays default, GROQ_API_KEY already set, OLLAMA_BASE_URL stays unset so ollama stays "skipped".

A call with provider=ollama and no reachable endpoint returns a clean 502 LLM_UNAVAILABLE (verified). write (#54) and foto (#48) stay built-not-deployed until after the first franc, per the plan of record.

TIER 3 humaniz keyless-live — the first-franc consumer repo: humanize-me-clean

Set OLL_MODEL_BASE_URL=https://model.oll.am + OLL_MODEL_SERVICE_TOKEN on the humaniz BACKEND Coolify app (env FIRST — it's a boot-gate).
Merge humaniz PR #9.
Redeploy keyless on Groq.
Then flip humaniz to LIVE Stripe.

Scope guard. Promoting foto/write now (before a stranger pays) is the ~85%-then-stall pattern — hold them on stage. The reliability + webhook + humaniz-live work is the ship.

Env-var config per service — the "config good" reference

What each service actually demands at boot vs. what's optional. The boot-gates are provider-aware / gateway-aware, so you only set the secrets a given mode needs.

Service (app / domain)	Required at boot	Optional / conditional	Notes
oll-model `model.oll.am`	`OLL_MODEL_SERVICE_TOKEN` `GROQ_API_KEY` (because `LLM_PROVIDER` defaults to groq)	`LLM_PROVIDER` (default groq) · `OLLAMA_BASE_URL` (only if `LLM_PROVIDER=ollama` or a `provider=ollama` call — else stays skipped) · `ANTHROPIC_API_KEY` (only for `provider=claude`) · `GROQ_MODEL`/`OLLAMA_MODEL`/`LLM_TIMEOUT` overrides	Boot-gate is provider-aware — only the default provider's key/URL is demanded at boot.
Core `core.oll.am`	`DATABASE_URL` (→ oll_core Neon DB) · `AUTH_JWT_SECRET` · `STRIPE_SECRET_KEY` · `STRIPE_WEBHOOK_SECRET` ← the money gate · `STRIPE_PRO_PRICE_ID` · `RESEND_API_KEY` · `FROM_EMAIL` · `FRONTEND_URL` / `SITE_URL` · `CORS_ORIGINS`	—	Rotate to the LIVE Stripe key for real charges.
write `write-service` (when deployed)	`CORE_BASE_URL` · `OLL_MODEL_BASE_URL=https://model.oll.am` · `OLL_MODEL_SERVICE_TOKEN`	`ANTHROPIC_API_KEY` — fallback only	`ANTHROPIC_API_KEY` is NOT needed when `OLL_MODEL_BASE_URL` is set (keyless).
foto `foto-service` (P2)	`CORE_BASE_URL` · `FOTO_PUBLIC_BASE_URL` · `REPLICATE_API_TOKEN` (only when the real replicate provider is armed)	mock default needs no token	Mock default = zero network/spend; the real replicate provider is triple-guarded.
humaniz backend (separate repo)	`OLL_MODEL_BASE_URL=https://model.oll.am` · `OLL_MODEL_SERVICE_TOKEN`	—	Base already defaults to `model.oll.am`; the `TOKEN` is the secret to set.

How to dev-test the new release — verified entrypoints

Real commands, verified this session. Fake the model provider at the seam for CI; run the real gateway + real write-service for the model-path proof.

One-command model-path harness

Builds the real oll-model gateway (groq) + real write-service (its model call delegated to the gateway) + a stub Core on docker-compose.dev.yml, waits healthy, and prints the exact curl to drive a real Groq rewrite through write → gateway → groq:

export GROQ_API_KEY=...   # from bubls/.env, local only
sh scripts/start-stack.sh
# write on :5012 · gateway on :5113 · prints the driving curl
sh scripts/start-stack.sh down   # teardown

The #56 e2e harness proves both the ollama and groq model paths repeatably (the dev-tested gate).

Post-deploy smoke (prod)

Curl the four health endpoints from the strip at the top, then a real gateway call:

curl -s https://model.oll.am/api/text/complete \
  -H "X-Service-Token: $OLL_MODEL_SERVICE_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"say hi"}]}'
# expect: {text, provider:"groq", model, tokens_in/out, latency_ms}

A 401 without the token proves the internal auth gate is live.

Ship — definition of done: a brand-new stranger can pay on the live URL and get value, with Sam touching nothing.