This section is a precise read of .github/workflows/*, the CI/CD constitution (main is the Truth), and the platform topology — not the aspiration, the code on disk.
CI is the gate; it ships nothing. Each service climbs its own path-scoped ladder before a commit is eligible to deploy. This is the strongest part of the whole system.
| Service · workflow | Gate ladder (in order) | Job dependency |
|---|---|---|
oll-coreoll-core.yml |
1. DTO-drift — regen pydantic models from openapi.yaml, git diff --exit-code fails on drift → 2. ruff lint → 3. alembic migrate + pytest --cov on a real postgres:17-alpine service container (matches Neon's major) → 4. Schemathesis contract-fuzz vs the sliced spec on a booted gunicorn Core (not_a_server_error + response_schema_conformance, health paths excluded) |
Steps 1–4 are one test job; docker-smoke runs in parallel as its own job — builds the real image via docker-compose.core.yml, asserts host-port liveness and container Health.Status=healthy. Both jobs must pass. |
productsservices.yml(foto · oll-write · oll-model) |
1. ruff lint → 2. pytest (mocks Core + providers, keyless — conftest sets env) → 3. docker boot + health — builds each real image, boots keyless via mock/gateway env, container's own Dockerfile HEALTHCHECK must report healthy |
Matrix over the 3 services. docker-boot needs: test — the boot gate runs only after lint+test pass. |
sitesite.yml |
design-language conformance — design-audit.sh: token values · font stacks · dark-theme chrome · internal-link integrity · nav presence · naming canon. Fails the build on any drift. |
Single job. |
Its comment names the exact incident it prevents: "the crash that took humaniz.me down — a boot-gate that demanded a key the keyless path never uses." Unit tests pass in a world of mocks; the boot gate builds the real production image and proves it comes up healthy with only the env a PaaS will give it. It catches the class of failure that unit tests structurally cannot — a missing boot env var, a $PORT/bind mismatch, a broken healthcheck — before it can reach prod. This is executable architecture governance, the pipeline analog of springular's ArchUnit rules and our own design-audit.sh: a convention that isn't mechanically enforced rots.
deploy-all.yml is the CD engine, and it is textbook-correct in structure. Reading it top to bottom:
v*or workflow_dispatch (services = all · changed · list)Three properties make this genuinely good, and worth understanding as why, not just what:
push: tags: ['v*'] and manual dispatch. A merge to main never deploys. This is the single most important design choice in the whole system; it is the industry's "merge ≠ deploy" principle made literal (OpenGitOps). The one hole: Coolify's own per-app git-auto-deploy is out-of-band and must be turned OFF in the console or the decoupling is a fiction — the constitution flags this as Sam's console action.preflight job queries the commit's check-runs and refuses if any concluded non-success. So a tag can't ship a red commit — CD verifies CI, it doesn't assume it./api/health + /health/neon + /health/stripe; oll-model: groq:ok; foto/write: status:ok; site: HTTP 200) before the next tier starts. A product never deploys onto a foundation that isn't up.The constitution defines three environments as deploy targets (production via <svc>.oll.am / tag-gated; staging via stage.<svc>.oll.am / auto-on-merge; optional PR previews) — "same service, two environments = two Coolify apps from one repo, differing only in injected config." That is 12-factor to the letter (factor III): config lives per-environment in Coolify, never in git; secrets are injected, not committed. Honest status: this is designed and documented but the staging apps are not yet stood up — today there is really one environment (production) plus the local full-stack mirror (docker-compose.local-full.yml). Two long-lived branches still exist (main + stage) with sync-stage.yml / automerge-stage.yml keeping them reconciled; the constitution's open decision is to retire the stage branch and make staging an environment.
To judge our pipeline you need the yardstick the field actually uses. This is the learning core — read it once and the assessment in Part 3 reads itself.
A decade of DORA / State of DevOps research found four (now five) metrics that predict organizational performance — and, crucially, that throughput and stability rise together in elite teams; you don't trade one for the other. The four you should track:
| Key | Definition | Elite band | How oll.am would measure it — for free |
|---|---|---|---|
| Deployment frequency | how often you successfully release to prod | on-demand / multiple per day | count successful deploy-all.yml runs (or Coolify webhook fires) per week |
| Lead time for changes | commit → running in prod | < 1 day | deploy_finish − git commit time; median it. Proxy: merge-to-main → deploy-finished |
| Change failure rate | share of deploys needing immediate rollback/hotfix | ~5% (0–15%) | deploys that tripped the post-deploy health gate or were followed by a rollback-tagged deploy ÷ total |
| Failed-deployment recovery time (the renamed MTTR) | time to recover from a bad deploy | < 1 hour | bad-deploy timestamp → next-good deploy; median |
Note the 2024 report shifted from fixed global cutoffs to relative performance clusters (~Elite 19% / High 22% / Medium 35% / Low 25%) and found, for the first time, the keys not all moving together — so treat the band table as directional, not gospel (2024 State of DevOps). The point for us: because our deploy already emits every timestamp these need, an append-only CSV line per deploy turns all four keys on with zero new tooling — and measuring is the precondition for improving. Today we measure none of them.
DORA identifies trunk-based development as a causal driver of performance, correlated specifically with < 3 active branches, branch lifetimes under a day, and no code-freeze periods. Long-lived branches (GitFlow's develop/release/*) delay integration, hide problems, and discourage refactoring (Fowler, "FeatureBranch"). The modern default is one always-releasable trunk + short-lived feat/* → PR → green CI → squash-merge. The companion principle: build one immutable artifact once and promote that same artifact across environments, varying only config — environments are deployment targets, not branches. Our constitution already reaches this conclusion independently; our residual gap is the second long-lived stage branch (which re-fuses "code line" with "environment") plus Coolify's out-of-band auto-deploy.
GitOps (a CNCF working group) formalizes what our deploy engine already does: version control holds the declarative desired state; deployment is a deliberate, auditable promotion, not a side-effect of a merge; a revert of the manifest is a rollback. Its four principles — declarative, versioned & immutable, pulled automatically, continuously reconciled — are the vocabulary for "deploy is a decision." Paired with 12-factor config (config that varies between deploys lives in the environment; the litmus test is "could this repo go public without leaking a credential?"), you get the clean separation we've designed toward.
The safe-rollout toolkit: blue-green (two prod environments, flip the router, instant rollback), canary (a small % of live traffic first), rolling (replace instances incrementally), health-gated (no traffic until the health check passes), and feature flags (decouple deploy from release — ship dormant code, turn it on later).
Coolify genuinely provides: zero-downtime rolling redeploys via start-before-stop (single-Dockerfile apps, health-gated) and one-click rollback to any prior image by commit hash — no rebuild.
One node cannot provide: true blue-green redundancy (that's a rolling swap on the same host, not two environments behind a router), survival of a host failure (one kernel/disk/NIC/power domain — if the box dies, every service dies and rollback can't help), or real traffic-% canary (needs ≥2 instances). Rolling redeploy protects against a bad deploy; it does nothing against a dead host. Note: Coolify does not do rolling updates for Docker Compose deployments (static container names → stop/start). The one progressive-delivery tool that is node-count-independent — and therefore our highest-leverage one — is the feature flag. (Coolify rolling-updates docs)
You don't need SLSA L3 to protect a solo shop; you need the four cheap controls that defend the two things most likely to actually hurt you — a leaked live Stripe key and a hijacked third-party Action.
| Control | What it does | Value / effort (solo) |
|---|---|---|
| Secret scanning + push protection | blocks a push containing a detected secret before it enters git history | very high / near-zero — directly serves the "exposed key = burned" rule (docs) |
Least-privilege GITHUB_TOKEN | a top-level permissions: block; setting any permission drops the rest to none | high / ~5 min — deploy-all.yml already scopes contents:read checks:read; the CI files don't (docs) |
| SHA-pin third-party Actions | a @v4 tag is mutable/hijackable; a full commit SHA is immutable | high / low — matters most for non-GitHub actions (docs) |
Dependency scanning (Dependabot / pip-audit) | auto-PRs for vulnerable deps; a CI audit step | high / low — the constitution names this as a known missing gate |
| CodeQL static analysis | finds code-level vulns | medium / low — free on public repos; enable if quiet |
| SLSA provenance / signed artifacts | attestation for downstream consumers of your artifacts | low value / high effort — you ship containers to your own VPS; no external consumer to verify them. Signed git tags are the sensible slice (SLSA levels) |
A health check answers "is it up?" (Kubernetes splits this into liveness / readiness / startup). Observability is larger: understanding internal state from outputs — logs, metrics, and traces — enough to ask novel "why?" questions (OpenTelemetry primer). The minimal starting alert set is Google SRE's four golden signals: latency (measure success and failure latency separately — a fast 500 hides otherwise), traffic, errors, saturation (alert on the rising trend, not at 100%). Around that:
RENAME couples code and DB rollback; expand-contract decouples them. Our constitution already mandates this.ionstarter — tag-release discipline + Vercel preview-per-PR: exactly the "environments-as-targets, ephemeral per-PR" model our constitution proposes, already proven in one of Sam's own projects. springular — a genuinely mature pipeline (in springular2/.github/workflows/): a single cicd.yml orchestrator that uses dorny/paths-filter for change-detection, composes reusable frontend/backend sub-pipelines via workflow_call, runs a docker-compose integration smoke, and ends in a flag-gated deploy job that GETs a Coolify webhook with Authorization: Bearer <token> — the exact pattern our deploy-all.yml deploy step is lifted from. It also governs architecture with ArchUnit-style executable rules — the same instinct as our design-audit.sh + docker-boot gates. Two ideas worth borrowing: reusable workflow_call sub-pipelines (to DRY our three CI workflows) and a central feature-flag block to toggle jobs. Where we're ahead: springular deploys on push to main (flag-gated); we deploy on a tag, CI-green-gated, dependency-ordered, and health-gated across four services.
Using the Continuous Delivery Maturity Model (five categories, five levels — Base → Beginner → Intermediate → Advanced → Expert), scored per category because the profile is deliberately uneven:
| Category | Level | Justification (grounded in our files) |
|---|---|---|
| Build & Deploy | Advanced (4) | Automated, path-scoped CI; deploy fully decoupled from VC (tag-triggered, no push-to-branch); CI-green preflight; dependency-ordered, health-gated CD; one-click rollback. "Release decoupled from deploy" is a Level-4 marker — we're there structurally. |
| Test & Verification | Advanced (4) (backend) | Contract-fuzz (Schemathesis), DTO-drift gate, real-Postgres pytest, and the docker-boot gate that reproduces prod boot. Strong. Held back only by no security-scan gate and no perf/load testing. |
| Design & Architecture | Advanced (4) | Loosely coupled services over HTTP, database-per-service, frozen-Core discipline, expand-contract migration policy — all DORA "loosely coupled architecture" capabilities. |
| Information & Reporting | Base–Beginner (1–2) | No metrics, no DORA measurement, no dashboards, no alerting, no external uptime monitor, no error tracking. Health checks exist but only fire during a deploy — nothing watches prod between deploys. This is the anchor dragging the composite down. |
| Culture & Reliability infra | Intermediate (3) | Excellent discipline and documentation for a solo builder, but the runtime substrate is a single VPS (SPOF) with untested backups — reliability engineering, as opposed to delivery engineering, is nascent. |
The pipeline (categories that decide how well you ship) is genuinely Level 4 and better than most teams with far more resources. The composite is a 3, not a 4, because Information & Reporting is a 1–2 and reliability rests on one un-redundant box with unproven backups. DORA's own guidance is a capability mindset over a maturity one: don't chase "Level 5," measure the four keys and fix the single capability that most moves a lagging one. For us that lagging capability is unambiguous — comprehensive monitoring & observability.
design-audit.sh + docker-boot fail the build on drift instead of trusting review (the springular / ArchUnit instinct).| # | Risk | Why it bites |
|---|---|---|
| 1 | No observability / alerting / uptime monitor | An outage is discovered by a user (or by Sam happening to look), not by the system. Health checks run only during a deploy; nothing watches prod between deploys. You cannot fix fast what you cannot see. This single gap is the difference between a hobby and a reliable service. |
| 2 | Single VPS = SPOF | One kernel / disk / NIC / power domain. If the Hetzner box dies, every service — including the money path — is down, and rollback can't help. All the pipeline's "zero-downtime" protects against bad deploys, not a dead host. Maintenance reboots = downtime with no failover. |
| 3 | Backups untested + 6h PITR window | Neon Free = a 6-hour restore window; a longer incident or a delayed discovery loses data beyond it. And a restore that's never been rehearsed is a hope, not a capability — the first real test must not be during an incident. |
| 4 | Flying blind on delivery (no DORA) | None of the four keys is measured, so "are we getting more reliable?" is unanswerable. The data is already in our deploy logs; we just don't capture it. |
| 5 | Manual tag + no live staging + fragile externals | Promotion is a manual git tag (fine, but human — easy to forget the CI-green ritual detail); the staging environment is designed but unbuilt, so integration proof happens in prod; oll-model's default provider rides the Groq free-tier rate limit (an in-process, non-durable quota — a traffic spike degrades the model path); and secret push-protection / a dependency-audit gate aren't enabled yet. |
Ordered so each step buys the most trust per hour. Effort tags: S = under an afternoon · M = a day or two · L = a project.
| # | Move | Effort | Value — what it buys |
|---|---|---|---|
| 1 | External uptime monitor + alert channel — UptimeRobot HTTP + keyword check per public service (oll.am, core, model, foto, write), plus TLS-expiry + a Heartbeat per cron. Alert to a channel you can't ignore. | S | Very high. Closes risk #1's worst edge: you learn of an outage in minutes, from the system, not a user. Highest reliability-per-hour move available. |
| 2 | Secret push-protection + least-privilege permissions: on the CI workflows + SHA-pin third-party actions + a pip-audit gate. | S | High. Under an hour total, all free/native; defends the leaked-Stripe-key and hijacked-action failure modes — the two most expensive solo mistakes. |
| 3 | Sentry on every service — capture unhandled exceptions with release + suspect-commit context. | S–M | High. Turns "something's wrong" into "this exception, this line, this release, N users." Pairs with #1 (black-box) as the white-box half. |
| 4 | Verify + document backup/restore — actually restore a Neon branch to PITR in a rehearsal; bump the retention window on the stateful services that matter; write a one-page DR runbook (what to do when the VPS dies). | S–M | High. Converts a hoped-for backup into a proven one and turns "the box died" from panic into a checklist. |
| 5 | DORA scorecard — append one CSV/JSON line per deploy in deploy-all.yml (commit time, deploy-finish, changed services, health-gate result); render the four keys on the Control Room. | S | Medium-high. Makes improvement measurable; the data already exists, we just capture it. Measurement precedes management. |
| 6 | Stand up the staging environment — the designed-but-unbuilt piece: stage.<svc>.oll.am Coolify apps on a Neon branch + test Stripe, auto-deployed on merge to main; retire the stage branch. Turn Coolify per-app git-auto-deploy OFF everywhere. | M | High. Moves integration proof out of prod, completes the environments-as-targets model, and closes the out-of-band Coolify auto-deploy hole. |
| 7 | Structured logging + a golden-signals view — request-id-correlated JSON logs (we have the audit-outcome-logging pattern), latency/error/traffic surfaced somewhere queryable. | M | Medium. The observability step beyond error tracking; makes "why is it slow?" answerable. |
| 8 | Harden the model path — give oll-model a fallback provider (or a paid tier) so the Groq free-tier rate limit isn't a single availability dependency. | S–M | Medium. Removes a non-durable in-process quota from the critical path for the AI products. |
| 9 | Redundancy for the money path — a second node + a load balancer (or a managed host for the one revenue-critical service) to survive a host failure. | L | High value, high effort. The only fix for risk #2 (SPOF). Honest "later" — justified once a stranger's dollars depend on uptime, not before. Ship-vs-build: do #1–#5 first; they buy most of the reliability at a fraction of the cost. |
Everything heavier — a second node, full metrics, blue-green — is real but premature. The pipeline is Level 4; the gap is that nothing watches the thing running. Fix seeing before you fix surviving.