diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md index 52462057..2cb6c2cc 100644 --- a/docs/plan/derived-blobs.md +++ b/docs/plan/derived-blobs.md @@ -1,12 +1,20 @@ # Plan — Derived content as blobs (tier-2 refactor) -**Status:** design captured 2026-08-02, not implemented. Follow-up to -`fix/services-use-blob-abstraction` — that PR normalised the -**read-side** (services consume blobs through `BlobStorageBackend` -uniformly). This plan tackles the **write-side**: services that -today write derived artifacts (thumbnails, transcodes) to a local -sidecar directory and would benefit from writing them through the -backend abstraction instead. +**Status:** design captured 2026-08-02, revised 2026-08-16 — keying +rule, CDC reuse, backend-dispatch rule, the +`content_derived_blobs` / `file_attached_blobs` pair, copy/version +semantics, a consistency coverage matrix with **three** hard +prerequisites (one of them a `dedup_gc` predicate that would delete +the entire derived tier), migration of the existing sidecar content, +and a schema trim down to the columns that carry information nothing +else owns. Not implemented. + +Follow-up to `fix/services-use-blob-abstraction` — that +PR normalised the **read-side** (services consume blobs through +`BlobStorageBackend` uniformly). This plan tackles the **write-side**: +services that today write derived artifacts (thumbnails, transcodes) +to a local sidecar directory and would benefit from writing them +through the backend abstraction instead. ## Context — the three-tier storage taxonomy @@ -27,6 +35,58 @@ thumbnails for every photo) but not data loss; conflating them means backup policies can't distinguish "must preserve" from "can rebuild". +## The relation map (after this refactor) + +Solid arrows **hold a reference** (bump a `ref_count`); dashed arrows +are **dependents** — they must be cleaned up when their target dies but +they keep nothing alive. + +```mermaid +flowchart TB + subgraph RES["RESOURCE LAYER · keyed by UUID"] + FILES["storage.files
id UUID PK
blob_hash VARCHAR(64)
name · folder_id · mime_type"] + FAB["storage.file_attached_blobs
(file_id, kind, variant) PK
blob_hash · uploaded_by
user-supplied · never shared"] + FMD["storage.file_metadata (EXIF)
file_id PK
⚠ content-derived, file-keyed"] + end + + subgraph CON["CONTENT LAYER · keyed by BLAKE3 of source bytes"] + CDB["storage.content_derived_blobs
(source_hash, kind, variant) PK
blob_hash
pure f(content) · dedupes"] + BET["storage.blob_extracted_text
blob_hash PK"] + FACES["faces.faces
blob_hash"] + end + + BLOB["BLOB — the content of a file
BLAKE3 of plaintext
storage.chunk_manifests
file_hash PK · chunk_hashes[]
ref_count"] + CHUNK["CHUNK — physical payload
BLAKE3 of the fragment
storage.blobs
hash PK · ref_count · orphaned_at"] + BACKEND[("BlobStorageBackend
Local .blobs/ · S3 · Azure
+encryption +retry +cache")] + + FILES -->|"FK file_id · CASCADE"| FAB + FILES -->|"FK file_id · CASCADE"| FMD + FILES -->|"blob_hash"| BLOB + FILES -.->|"legacy pre-CDC · no manifest"| CHUNK + CDB -.->|"source_hash · dependent"| BLOB + CDB -->|"blob_hash"| BLOB + FAB -->|"blob_hash"| BLOB + BET -.->|"dependent cache"| BLOB + FACES -.->|"dependent cache"| BLOB + BLOB -->|"chunk_hashes[] · 1..N ordered"| CHUNK + CHUNK -->|bytes| BACKEND +``` + +Three things to read off it: + +1. **`content_derived_blobs` touches the Blob layer twice with + opposite meanings** — `source_hash` is a dependent (it keeps + nothing alive; the file does), `blob_hash` is a reference holder. + Conflating them is how you get either a leak or a premature reap. +2. **Every new solid arrow into the Blob layer feeds + `chunk_manifests.ref_count`** — the counter nothing reconciles + today. See the prerequisites below. +3. **The two new tables meet the rest of the graph only at the Blob + layer.** `content_derived_blobs` has no edge to `storage.files` at + all: it reaches a file only by sharing that file's `blob_hash`. + That is exactly what makes it dedupe across files — and exactly why + it must never hold user-chosen bytes. + ## Multi-instance driver Single-instance: tier-2-as-local-cache works fine. Rebuild after @@ -61,36 +121,606 @@ Reusing it for derived artifacts means no second abstraction to build and maintain, and all the operational surface (audit, migration, key rotation) applies to derived content by default. -### Keying +### Keying — content-address only pure functions of the content -Content-addressable via BLAKE3, same as source blobs. For -server-derived content the hash is over the produced bytes (not -the source), so: +**The rule:** an artifact may be keyed by its source's content hash +**iff** it is a deterministic pure function of the source bytes. +Anything influenced by user choice must be keyed by the resource it +was attached to, never by content. -- Two files with **identical thumbnails** (e.g. same 256px WebP - crop of the same underlying image → identical bytes → identical - hash) share the physical blob. Dedup wins for free. -- Two files with **identical originals** but **different variant - specs** (256px vs 512px thumb) produce different blobs. Also - correct. +| Artifact | Function of | Content-keyable? | +|---|---|---| +| server thumbnail | `f(blob bytes, variant)` | ✅ any user uploading identical bytes derives identical output — nothing to poison | +| transcode | `f(blob bytes, target)` | ✅ | +| extracted text | `f(blob bytes)` | ✅ — `storage.blob_extracted_text` | +| face vectors | `f(blob bytes)` | ✅ — `faces.faces` | +| client-uploaded preview | `f(user's choice)` | ❌ **must be file-keyed** — `storage.file_attached_blobs`, see below | -The variant spec (what was rendered) lives in the referring DB row -alongside the blob hash — not in the storage key. Storage stays -one keyspace; ownership stays per-service. +This isn't a new pattern: `storage.blob_extracted_text` already +chose content-keying for the same reason, and the migration says so +(`migrations/20260701000000_content_search_index.sql:22-28`) — +"extraction is keyed by `blob_hash`, not by file: N copies of the +same PDF cost ONE extraction, and rename/move/copy never +re-extract." `faces.faces` is keyed on `blob_hash` too. Thumbnails +are the same class of artifact, and file-keying them would make +them the odd one out among three sibling features while costing: -### Client-uploaded thumbnails +- **the dedup fast path** — `ThumbnailRefreshHook::on_file_created` + returns early when `!is_new_blob`, so 100 users uploading the same + photo cost one render. File-keying means either N renders or a + join back through `files.blob_hash` (content-keying + through the back door, slower and with more code). +- **free copies and free versions** — `on_file_copied` is a no-op + today precisely because the key is content, and future versioning + inherits the same property. See the copy/version axes below. + +For the derived side the hash is over the **produced** bytes, so: + +- Two files with identical thumbnails (same variant of the same + source → identical bytes → identical hash) share the physical + blob. Dedup wins for free. +- Two variants of one source (256px vs 512px) produce different + blobs. Also correct. + +The variant spec lives in the referring DB row, not in the storage +key. Storage stays one keyspace; ownership stays per-service. + +### Corollary — point at a file, never at a blob + +Both tables in this plan exist because their content is *not* a file: +a thumbnail has no name, no folder and no place in a user's tree. When +a binary **can** be a file, make it one and point at it with a +`*_file_id` FK — `storage.files` is already a `BlobReferenceSource`, +already covered by every consistency edge, already GC-integrated, so a +file pointer costs **zero** new reference sources and zero new +consistency checks. + +That is the rule that stops the next person adding a fourth +blob-referencing table. It is what `docs/plan/hidden-system.md` +applies to user avatars, backgrounds and signatures, and it extends to +owners that are not users at all — +`carddav.contacts.photo_file_id` would retire the inlined +`photo_url TEXT` on the same terms. + +### Schema + +```sql +CREATE TABLE storage.content_derived_blobs ( + source_hash VARCHAR(64) NOT NULL, -- source Blob (no FK — see below) + kind TEXT NOT NULL, -- 'thumbnail' | 'transcode' + variant TEXT NOT NULL, -- 'icon' | 'preview' | 'large' | '720p' + blob_hash VARCHAR(64) NOT NULL, -- the DERIVED Blob + content_type TEXT NOT NULL, -- served directly; no byte-sniffing + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (source_hash, kind, variant) +); +CREATE INDEX ON storage.content_derived_blobs(blob_hash); +``` + +**`variant` is opaque text. New axes go inside it, never into new +columns.** This is the rule that keeps the table from growing, and it +disposes of three columns earlier drafts proposed: + +- **No `format` column.** WebP vs JPEG looks like a second axis, but + only the canonical rendering is persisted (below), so there is one + row per variant. If a format migration ever happens — AVIF is the + plausible one — it is `variant = 'preview-avif'` beside + `'preview'`. Data change, not a PK migration. +- **No `codec` column.** Transcoding here is a *playability + fallback*, not bandwidth optimisation: one widely-compatible + rendition (H.264/AAC in MP4), no negotiation, nothing to + distinguish. `