Designing the CI/CD — intentional deployment behind staged quality gates

The one seam — merge ≠ deploy

The gates buy the trust to auto-merge — into stage, where merging changes the code of record and ships nothing. Quality buys speed; deploy stays intentional, on main. ↓ the full model next.

The branch model — stage integrates, main promotes (per-service gated deploy)

Read it left→right: every feature branch opens a PR into stage; the CI gate is the only thing that decides the merge; stage accumulates the whole line and never deploys. When a service is ready, you promote it to main and flip that one service's deploy flag — site/docs deploy freely (your interface stays live), services deploy only when their DEPLOY_<SVC>_ENABLED var + COOLIFY_HOOK_<SVC> secret are set. The pipeline never blocks on deploy again.

The promotion ritual — stage → main, one service at a time

One-time console setup — what actually unblocks the decouple

Stage as a deployed environment

The design decision — isolation must be structural, not a branch flip

By design, the platform starts on one Docker network with the prod domains — the minimal footprint for a solo operator, chosen deliberately. Adding a stage environment therefore carries an explicit, documented decision: isolation must be structural, because a service "deployed to stage" on the prod network would otherwise be reachable by prod, and vice-versa. Concretely, the rejected shortcut is pointing the existing prod gateway app at the stage branch — that does not create a stage environment; it makes the prod Model gateway (the one live product clients call) run stage code on the prod domain and prod network, conflating the two. The intended end-state below gives stage its own network and domains, so the two environments never touch — a stage that shares prod's network and domains would not be a stage at all.

The design — isolated, cheap for a solo builder

Two worlds, one VPS: prod apps watch main on the internal network and are never repointed at stage; stage apps watch stage on an isolated stage network, get stage.-prefixed domains, and resolve each other by their -stage names. The dashed wall is the whole point — no container on one network can reach the other. Promote-to-prod is the existing gated-deploy model with a stage rehearsal in front: verify on stage → promote stage→main for that one service + flip its DEPLOY_<SVC>_ENABLED gate → the prod app deploys.

Runtime-first walkthrough — the concrete first case

Stage stops being a promise the CI makes and becomes a place you can curl — isolated by its own network and its own domains, so proving a service never risks the ones a stranger is paying for.

The deploy sequence — dependency-ordered, one button (CD ≠ CI)

One press deploys everything armed, in the right order: Identity service + Model gateway first (the foundation a product would 404/500 against if it weren't up), each health-gated before the product tier starts; products next (they call the Identity service /me+billing+email and the Model gateway /text); site last. A tier proceeds when its deps succeeded or were skipped (flag off) — never when one failed, so a half-broken foundation can't drag a product live. Scaffolded as .github/workflows/deploy-all.yml.

CI proves the code; CD ships it — deliberately, in dependency order, one button, one service's gate at a time.

Failure modes & recovery — what the gates don't catch

Auto-merge to stage + automated remote testing — the verification rungs

Auto-merge to stage on green CI

PRs into stage squash-merge themselves the moment their CI is green — safe because stage never deploys; auto-merge only advances the integration line. Mechanism: .github/workflows/automerge-stage.yml — an Action that fires when a CI workflow completes, finds the matching green PR targeting stage, and squash-merges it (skips drafts + hold/wip/do-not-merge labels, and waits until every check is green).

⚠️ Constraint surfaced (a Sam decision): GitHub's native auto-merge + branch protection (required status checks) need GitHub Pro or a public repo — they 403 on a free private repo. So we use the Action path, which works on a free private repo via the workflow GITHUB_TOKEN. (workflow_run Actions run from the default branch, so this file ships to main with the CI/CD infra and governs PRs into stage.) If you'd rather have native auto-merge + protected branches, that's the one upgrade that unlocks it.

Automated remote testing on the stage environment — smoke + e2e (next)

The design intent: once the stage environment exists, enable autonomous testing on a remote dev-test target — smoke + e2e. Today verification is local + mocked (the dev-test stacks). The next rung stands up a stage/preview deploy of the line, then runs against its live URLs, automatically, before promotion:

This turns the dashed rungs 4–5 of the pipeline (drawn above) real: verification moves from mocked-local to real-remote and automated, and nothing is promoted to production that hasn't proven itself on a live stage first — promotion of the money spine still waits on a human.

The gate ladder — the "thorough quality check"

Rungs 0–4 are pre-merge (the trust); rung 5 is post-deploy and decoupled. Anti-coverage-theater: per-service floors (not a repo average that lets the spine hide an untested product), branch coverage, diff-coverage as the real teeth, and the omit/pragma config itself reviewed on every PR (the "ignore the auth tests" failure mode).

Reality check — the core subset vs the fuller map

This page describes the delivery model in full; deliberately, it is grown to fit real load rather than gold-plated ahead of it — so a focused core subset carries the weight and the rest is staged until concurrency and traffic justify it. Here is the core set of CI workflows the design defines under .github/workflows/:

✓ core of the design: the identity spine plus its full gate set · the path-filtered CI (spine · services ruff+matrix · the site design gate) · automerge-stage.yml + the one-button deploy-all.yml (activated from the default branch, since workflow_run/CD fire there). ○ fuller map: a merge queue, CODEOWNERS, services.yml docker-smoke rigor, and remote-stage smoke+e2e.

How we got here — options & best practice, condensed

Explored: incremental → reusable-matrix → merge-queue. Not either/or — they're the sequenced best-fit below, adopted in that order as scale demands.

2026 best practice — the cited verdict, 5 bullets

• Skip Nx / Turborepo / Bazel below ~5 services — dorny/paths-filter (mirroring Coolify Watch Paths) is right at our size.
• The gatekeeper-job is mandatory: required checks don't understand skipped path-filtered jobs (a skip sits "pending" forever). Fix = one always-running job that needs: all conditional jobs and is the only required check.
• Native merge queue + branch protection + native auto-merge would be the textbook tool for an agent PR flood (batches against future-main) — but they require GitHub Pro or a public repo and 403 on a free private repo. That constraint is exactly why we land via the Action-based automerge-stage.yml (squash on green CI) instead. Mergify/Trunk are paid alternatives only if CI later becomes the cost bottleneck; Kodiak is unmaintained — avoid.
• Keep the contract/Schemathesis gate required — contract/spec drift is the #1 failure mode of agent PRs that slips past unit tests; our DTO-drift + Schemathesis gate is exactly the guard, and green-on-it is what arms the auto-merge into stage.
• Tier trust by path (CODEOWNERS) + reusable workflow_call + pip caching + concurrency: cancel-in-progress + quarantine flaky tests before trusting auto-merge.

The recommended design — sequenced best-fit

Deploy & freeze — the intentional, dependency-ordered ship

The freeze cycle — frozen by default, promoted deliberately, re-frozen

So merge moves stage; promotion moves main; deploy is a separate, manual, dependency-ordered, per-service-gated act — armed by a var, fired by a secret, ordered by needs:. Automated agents can flood stage with green PRs all night and nothing reaches prod until a deliberate deploy-all run says it may.

The controlled-change protocol — extending a frozen service

1. Additive-only, on a feat/ branch into stage. A new endpoint / provider / field / migration — never a rewrite of the existing contract. Existing consumers must be unaffected. (The Identity service is OpenAPI-first → the DTO-drift + Schemathesis gates prove backward-compatibility; the Model gateway adds providers as one entry in _PROVIDERS, call sites unchanged.)
2. Full gate ladder green (the same ladder above): lint · tests · contract/Schemathesis · docker boot+health. For a money-path or auth change, an adversarial multi-agent review (a documented pre-merge verification pass) before merge.
3. Intentional, flagged deploy. Promote stage → main (merge ≠ deploy). To ship: flip that service's DEPLOY_<SVC>_ENABLED → true for the one deploy, run deploy-all.yml, then flip it back OFF → re-frozen. Never a standing auto-deploy on a frozen service.
4. Re-verify live + record the new frozen commit. Health green, the new surface works, existing surface unchanged. The service is frozen again at the new commit.

Planned extensions (the only sanctioned changes to the frozen set)

The everyday state of a frozen-infrastructure service is exactly that — frozen. Changing one is a deliberate, planned event, never an ad-hoc push: it is proposed, then executed by this protocol, then re-frozen.

The maturity roadmap — automated recovery under human oversight

The full PR → prod loop — automated path, human-gated deploy

The agent arms gh pr merge --auto on open (the ladder is the approval) → the merge queue rebuilds each candidate on main-as-it-will-be so two green agent PRs can't break main together → Coolify spins a preview at a per-PR subdomain against a fresh copy-on-write Postgres branch (destructive money-path tests on real-shaped data that vanishes on PR close) → the deploy-probe curls /health/{db,payments} + the passwordless-sign-in→checkout→webhook→plan-flip smoke; green clears the image for promotion — and the money spine still waits on a human to authorise the deploy. Mechanical where it's safe, human-gated where it's irreversible.

Automated recovery — what notices at 3am, rolls back, and hands a human the incident

Detection is cheap; automated recovery is the goal — main reverted, prod rolled back, and a fix-it incident issue waiting for a human, before anyone woke up.

The dev-test stacks & test doubles — documented