Platform Extensions — The Overnight Build Ledger

Purpose & the pivot's why

One stateful service = document ingest + hybrid semantic retrieval + structured extraction, built on LlamaIndex (PGVectorStore on pgvector, SentenceSplitter, OllamaEmbedding). Only a thin house-style shell is hand-written — auth, the unified error envelope, health probes, the boot-gate factory — and extraction generation is routed through the oll-model gateway. It owns a private Neon oll_memory DB. The pivot's logic: don't reinvent RAG. The original #75/#76/#77 hand-rolled a chunker, pgvector SQL, and Reciprocal Rank Fusion; a mature library does that better, so the win is fewer deploys, one DB, one contract instead of three services to wire, provision and keep in sync.

With the deterministic hash embedder (the keyless CI default), a synonym-only query — "mountain gold vault custodian", where keyword_hits=0 so retrieval is pure dense — ranked the correct document #9 of 9: no signal. Switching only EMBED_PROVIDER to Ollama nomic-embed-text (768-dim) ranked the same document #1.

Real semantic retrieval — meaning matched with zero shared keywords. Embeddings default to a keyless deterministic local-hash provider (so CI runs real cosine maths, no keys), with nomic (self-hosted / EU) behind an env flag for real use.

• LlamaIndex engine (not hand-rolled RAG)
• 65 pytest vs real pgvector
• ruff clean
• docker boot + health against pgvector
• Keyless CI-equivalent verified — ingest / query / ACL / delete / extract
• nomic semantic win #9→#1 reproduced
• Dedicated test-memory pgvector CI job + integration ingest→query assertion
• All 26/26 gh pr checks green
• PR #80 UNMERGED — deploy is Sam's

Purpose

Schema-locked structured extraction. Give it text (or input) plus a JSON Schema; it returns validated, typed JSON that conforms to that schema — the messy-language-in, clean-fields-out primitive every document product needs (invoice fields, résumé parsing, contract terms). It calls the Model gateway for inference and never imports a model SDK of its own. It is stateless — no database: a pure request → typed-JSON transform.

• Contract-first OpenAPI slice → DTOs
• 33 pytest, keyless
• ruff clean
• docker build 270MB, non-root + HEALTHCHECK HEALTHY
• Real keyless hop to a live oll-model container proven
• Booted inside the CI integration compose
• All gh pr checks green — lint+test, pip-audit, docker boot+health, integration smoke
• PR #75 UNMERGED — deploy is Sam's

Purpose

Document / vision ingestion — the front-half of the pipeline. Give it a document (text, markdown, PDF or image); it returns clean, normalized text split into overlapping, paragraph-aware chunks with character offsets and metadata. It is the step that turns a messy source into retrieval-ready pieces — its chunks are what oll-rag embeds and searches. Like the other extensions it is stateless — no database: a pure document → chunks transform.

• Contract-first OpenAPI slice → DTOs
• 40 pytest, keyless
• ruff clean
• docker build 266MB, non-root + HEALTHCHECK HEALTHY
• Verified curl: 3 chunks, contiguous offsets 0..529, exact 40-char overlap, heading metadata present
• Booted inside the CI integration compose
• All 26/26 gh pr checks green — lint+test, pip-audit, docker boot+health, integration smoke
• PR #76 UNMERGED — deploy is Sam's

Purpose

Hybrid retrieval over a corpus — the retrieval back-half of the pipeline. Store embedded, chunked documents (oll-ingest's output) and answer a query by fusing vector + keyword search with chunk-level ACL, backed by Postgres + pgvector. Where the other extensions are stateless transforms, this is the first extension that owns its own DB — a corpus persists between an ingest and a query.

• Contract-first OpenAPI slice → DTOs
• 46 pytest incl. 14 DB-backed against real pgvector
• ruff clean
• docker non-root + HEALTHY two ways — real pgvector and keyless dummy DB
• Verified curl: RRF top-1 score 0.0328 (found by both retrievers) vs 0.0161 vector-only
• ACL hides then reveals the tagged chunk; delete works
• Dedicated test-rag pgvector CI job + integration smoke boots pgvector + oll-rag
• All 26/26 gh pr checks green — PR #77 UNMERGED

Purpose

An additive, backward-compatible routing tier that lets a caller require inference to run only on EU-hosted or on-device providers — the honest, sellable "your data never leaves the EU, or your device" capability, made routable. It does NOT add a new model backend; it rides the existing provider switch. Because the Model gateway is a frozen service, the whole design turns on one promise: a caller who ignores the new field sees no change whatsoever.

• Additive OpenAPI — optional field + new endpoint, no breaking change
• 84 pytest — 54 unchanged + 30 new, keyless
• ruff clean
• docker builds + boots HEALTHY keyless
• Verified curl: no-privacy call unchanged (no echo fields)
• Verified curl: on_device routes + echoes region
• Verified curl: eu + groq → 422 POLICY_VIOLATION
• Verified curl: nothing compliant → 503 NO_COMPLIANT_PROVIDER
• Verified curl: /api/text/policy matrix returns
• Integration smoke: no-privacy cross-service proves backward-compat
• All 20/20 gh pr checks green
• PR #78 UNMERGED — deploy is Sam's

Purpose

A single deployable Streamable-HTTP MCP server that exposes the whole platform as authenticated, entitlement-gated tools for Claude, ChatGPT and Cursor. This is the distribution + defensibility move from the research: the same primitives (extract / ingest / RAG / private inference) that any MCP builder can wire up, but wrapped in the Core billing + identity spine most of them lack — so a caller is a known, paying user, not an anonymous key holder. It is the one ASGI service in the platform (the MCP SDK brings its own uvicorn / Starlette); every other house convention is kept, and it is purely additive — no other service is touched.

• MCP Streamable-HTTP server ✓
• 38 pytest, keyless (+5 cross-user scoping regression, hardening pass)
• ruff clean · pip-audit clean
• docker non-root + HEALTHY ~3s (283MB)
• Real MCP client vs booted uvicorn: auth-reject 401
• list_tools = 7
• ingest → upsert → query chain — top chunk score 0.83
• complete_text privacy=eu → provider ollama / region eu
• free-plan caller → "upgrade required" tool error
• All 26/26 gh pr checks green
• PR #79 UNMERGED — deploy is Sam's

• Tag-less / omitted acl cannot read private chunks. The enforcement is cardinality(acl_tags)=0 OR acl_tags && caller_tags — a chunk is public only when it carries no tags. A caller with an empty tag array overlaps nothing, so it sees only public chunks. Private-by-default: a tagged (private) chunk is invisible unless the caller presents an intersecting tag.
• SQL injection via acl / filter / collection / query is inert. Every one of those inputs reaches Postgres as a bound parameter, never string-concatenated into SQL — the classic array-overlap and filter payloads do nothing.
• No cross-collection leakage. Every read, write and delete pins collection in its WHERE clause, so a query against collection A can never surface a chunk stored under collection B, regardless of tags.
• ACL enforced on BOTH branches. The same predicate guards the vector arm and the keyword arm before Reciprocal Rank Fusion — a chunk the caller may not see is filtered out of each retriever, so RRF can never resurrect it. A one-sided guard would have been the classic hybrid-search leak; it isn't here.

Scope note (by design, not a gap): within oll-rag a caller may present any tags — it is an internal service behind a service token, and binding tags → identity is the gateway's job, not the store's. That binding is exactly what the oll-mcp finding below is about.

The fix — pin, don't default

• Collection is namespaced under the caller. A client-requested collection is forced under the caller's own u{id} prefix — it can name a sub-collection, never a sibling's.
• ACL is forced to the caller's own tag. No membership / sharing model exists, so client ACLs are never trusted — the tag is set server-side from the authenticated identity, full stop.
• Applied to all three data tools — rag_query, rag_upsert_document, rag_delete_document — with a test_cross_user_scoping.py regression proving cross-user read, delete and overwrite are all blocked. oll-mcp is now 38 tests, gh pr checks 79 green.

Also attacked, and refuted

• Free / expired / garbage bearer reaching a paid tool. Every paid tool runs through the plan gate (/me → MCP_REQUIRED_PLAN); only platform_health is free. A non-pro or invalid identity gets a clean tool error, never data.
• Token-custody leak. Service tokens are added only inside the downstream boundary (downstream.py); errors surface code + message, not headers or URLs — a tool error can't exfiltrate a token or an internal endpoint.

1. Merge PR #80 (oll-memory). It supersedes #75/#76/#77 — close those three when #80 merges. #78 (sovereign tier) and #79 (oll-mcp) stay open but parked; nothing to deploy for them now.
2. One Coolify app for oll-memory. Base Directory services/oll-memory, matching Watch Paths, honour $PORT, set OLL_MEMORY_SERVICE_TOKEN and OLL_MODEL_* (base URL + token) so the /api/extract generation hop reaches the oll-model gateway.
3. Provision oll-memory's database. It is stateful: create a private Neon oll_memory DB and enable the vector extension (CREATE EXTENSION vector;) before first boot.
4. Wire embeddings. Real use runs on self-hosted Ollama nomic-embed-text — set EMBED_PROVIDER=ollama + OLLAMA_EMBED_BASE_URL (and EMBED_DIM=768). The keyless local-hash default stays for CI; the boot gate asserts the Ollama env when the real provider is selected — no silent fallback.
5. Deploy stays deliberate. A tag or deploy-all.yml, health-gated (/api/health/db + /api/health/model), never a merge-to-main. Then curl an ingest → query round-trip against the live app to confirm the semantic win holds with the real embedder.

The five have landed — Phase 2 (Vault Chat) is in progress

With all five extensions green, the payoff is a POC product in a SEPARATE repo that consumes the extended platform end-to-end — ingest a document, extract typed fields, retrieve grounded answers, complete text, meter it through Core, all EU / on-device by default. Tested end-to-end against the live platform, it proves the extensions are reusable, not just present. Phase 2 (Vault Chat) is now in progress — the deliberate next move now that the foundation is real and green.