diff --git a/docs/plan/derived-blobs.md b/docs/plan/derived-blobs.md
index 52462057..2cb6c2cc 100644
--- a/docs/plan/derived-blobs.md
+++ b/docs/plan/derived-blobs.md
@@ -1,12 +1,20 @@
# Plan — Derived content as blobs (tier-2 refactor)
-**Status:** design captured 2026-08-02, not implemented. Follow-up to
-`fix/services-use-blob-abstraction` — that PR normalised the
-**read-side** (services consume blobs through `BlobStorageBackend`
-uniformly). This plan tackles the **write-side**: services that
-today write derived artifacts (thumbnails, transcodes) to a local
-sidecar directory and would benefit from writing them through the
-backend abstraction instead.
+**Status:** design captured 2026-08-02, revised 2026-08-16 — keying
+rule, CDC reuse, backend-dispatch rule, the
+`content_derived_blobs` / `file_attached_blobs` pair, copy/version
+semantics, a consistency coverage matrix with **three** hard
+prerequisites (one of them a `dedup_gc` predicate that would delete
+the entire derived tier), migration of the existing sidecar content,
+and a schema trim down to the columns that carry information nothing
+else owns. Not implemented.
+
+Follow-up to `fix/services-use-blob-abstraction` — that
+PR normalised the **read-side** (services consume blobs through
+`BlobStorageBackend` uniformly). This plan tackles the **write-side**:
+services that today write derived artifacts (thumbnails, transcodes)
+to a local sidecar directory and would benefit from writing them
+through the backend abstraction instead.
## Context — the three-tier storage taxonomy
@@ -27,6 +35,58 @@ thumbnails for every photo) but not data loss; conflating them
means backup policies can't distinguish "must preserve" from "can
rebuild".
+## The relation map (after this refactor)
+
+Solid arrows **hold a reference** (bump a `ref_count`); dashed arrows
+are **dependents** — they must be cleaned up when their target dies but
+they keep nothing alive.
+
+```mermaid
+flowchart TB
+ subgraph RES["RESOURCE LAYER · keyed by UUID"]
+ FILES["storage.files storage.file_attached_blobs user-supplied · never shared "]
+ FMD["storage.file_metadata (EXIF)⚠ content-derived, file-keyed "]
+ end
+
+ subgraph CON["CONTENT LAYER · keyed by BLAKE3 of source bytes"]
+ CDB["storage.content_derived_blobs pure f(content) · dedupes "]
+ BET["storage.blob_extracted_text faces.faces BLOB — the content of a filestorage.chunk_manifests ref_count "]
+ CHUNK["CHUNK — physical payloadstorage.blobs ref_count · orphaned_at"]
+ BACKEND[("BlobStorageBackend ` probes the codec from the container, so the
+ header only owes `video/mp4`. A second codec would be
+ `variant = '720p-av1'`. (HLS/DASH manifests do need declared
+ codecs, but that is a segmented architecture this table doesn't
+ model and would need its own structure regardless.)
+- **No `renderer` / spec-version column.** It would exist to
+ invalidate rows when render parameters change (400 px → 512 px,
+ q82 → q85) — but such a change is global, so invalidation is
+ `DELETE … WHERE kind = 'thumbnail'` with or without it. The
+ `blob_extracted_text.extractor` precedent does **not** transfer:
+ that column exists because extraction caches *terminal negative
+ results* per blob (`'failed'`, `'unsupported'`) that must be
+ retried on an extractor bump. Thumbnails never persist a failure —
+ the failure path stores a zero-weight moka sentinel that evicts
+ immediately. No cached negatives, nothing version-dependent.
+
+**Only the canonical rendering is persisted to tier 3.** WebP covers
+~97% of clients; persisting the JPEG fallback too would add one
+object per size per source — a 50% increase in derived object count
+and in the `blobs_consistency` probe cost, to serve 3% of requests.
+Re-encoding JPEG from an already-decoded 400 px WebP is
+sub-millisecond, so the fallback is generated on demand and held in
+the moka RAM tier only. Same principle as the "persist only `large`,
+derive icon/preview" option below: **tier 3 stores the canonical
+rendering; everything else is derived on demand.**
+
+**No `size` column.** The bytes are content-addressed, so their
+length is an immutable fact the blob layer already owns
+(`chunk_manifests.total_size` / `storage.blobs.size`, reachable via
+`blob_hash`). Copying it here would create a second source of truth
+for something this table has no authority to assert. Every consumer
+is covered without it: `Content-Length` comes from the bytes in hand,
+storage reporting joins on `blob_hash`, thumbnails are never
+range-served, and `read_blob_bytes` already takes its buffer hint
+from the manifest row.
+
+`content_type` is the one thing worth storing rather than
+recomputing: the handler byte-sniffs every response today
+(`mime_detect::thumbnail_content_type`), and this deletes that. It is
+a per-row fact the producer knows for certain. It is *not* part of
+the key — identity is (source, kind, variant).
+
+One table with a `kind` discriminator rather than separate
+`storage.thumbnails` / `storage.transcodes`: `count_references`,
+`list_referenced_blobs`, the GC cascade and the
+`backend_consistency` walk are byte-identical between the two, so
+two tables means maintaining a duplicate of that SQL — which
+`AGENTS.md § Code duplication` forbids. `kind` costs one column.
+
+**No FK on `source_hash` or `blob_hash`**, for the reason the
+search-index migration already documents: a file hash resolves to
+either `storage.blobs` (legacy whole blob) or
+`storage.chunk_manifests` (CDC file hash), so the reference can't be
+expressed as a single FK. Orphans are reclaimed by GC instead.
+
+**No `origin` column.** The 2026-08-02 draft had
+`origin = 'server_derived' | 'client_provided'` so consistency-check
+severity could differ. With client previews excluded from this table
+(below), every row is server-derived and the severity is uniformly
+"warning, regenerable" — the column carries no information. Re-add
+it only if that changes.
+
+**Deletion is app-layer.** `on_blob_deleted(source_hash)` does
+`DELETE FROM storage.content_derived_blobs WHERE source_hash = $1 RETURNING blob_hash`
+then `remove_reference()` per row — a one-for-one replacement of
+today's `delete_blob_thumbnails` unlink loop, no new mechanism. A raw
+SQL `ON DELETE CASCADE` would drop the mapping row without
+decrementing the refcount, but *not* silently: once the reference
+registry exists the refcount is derivable, so the drift surfaces as a
+`refcount_mismatch` finding, and the interim state is an **over**-count
+(blob retained longer than needed) never an under-count (live data
+reaped). So a cascade is tolerable where it's ergonomic — see
+`file_attached_blobs` below — provided the registry is in place to
+reconcile it.
+
+### The pair — `content_derived_blobs` and `file_attached_blobs`
+
+Two tables, one keying difference, and that difference *is* the
+security boundary (see the client-preview section):
+
+```
+storage.content_derived_blobs (source_hash, kind, variant)
+storage.file_attached_blobs (file_id, kind, variant)
+```
+
+**The name states the key**, because the key is the only thing that
+differs and choosing the wrong one is a silent poisoning bug rather
+than a compile error. An earlier draft called these
+`derived_blobs` / `attached_blobs`; that pairing was rejected because
+`derived` vs `attached` is the wrong axis of symmetry — a thumbnail is,
+in plain English, "attached to" a file, so both words plausibly
+describe either table and the names carry no signal about keying. The
+`content_` / `file_` prefixes are non-overlapping and answer the only
+question an implementor needs to ask.
+
+```sql
+CREATE TABLE storage.file_attached_blobs (
+ file_id UUID NOT NULL REFERENCES storage.files(id) ON DELETE CASCADE,
+ kind TEXT NOT NULL CHECK (kind IN ('preview', 'subtitle', 'cover_art')),
+ variant TEXT NOT NULL, -- 'preview' | 'en' | 'cover'
+ blob_hash VARCHAR(64) NOT NULL, -- content-addressed bytes (dedup preserved)
+ content_type TEXT NOT NULL,
+ uploaded_by UUID NOT NULL REFERENCES auth.users(id),
+ created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+ PRIMARY KEY (file_id, kind, variant)
+);
+CREATE INDEX ON storage.file_attached_blobs(blob_hash);
+```
+
+Same trims as `content_derived_blobs` — no `size`, no `format`, no
+`codec`. Two deliberate differences:
+
+- **No `renderer`-style column even in principle.** These bytes
+ aren't rendered, they're supplied; there is no spec to version.
+ That falls straight out of the purity rule.
+- **`uploaded_by` is kept.** It is the one column here that is
+ genuinely new information rather than a copy of something the blob
+ layer owns, and it is the only trace that an Editor on a shared
+ file replaced the owner's preview — untraceable today.
+
+**Routing rule — put this as a comment on both tables:**
+
+> Bytes are a pure deterministic function of the file's content →
+> `content_derived_blobs` (content-keyed, dedupes across files, regenerable).
+> Bytes are user-supplied or user-chosen → `file_attached_blobs`
+> (file-keyed, never shared across files, not regenerable).
+
+Generic naming rather than `file_previews` because the family is
+real, and each member would otherwise be a new table plus a new
+`BlobReferenceSource` plus a new term in the consistency recompute.
+With `kind` it's a one-line `ALTER … CHECK`:
+
+| Kind | Why it lands here |
+|---|---|
+| `preview` | client-uploaded thumbnail (today's case) |
+| `e2e_thumbnail` | strongest future case — in an E2E drive the server *cannot* derive thumbnails, so the client must upload them. "Vault" is already reserved as a future E2E drive kind (`project_drive_naming_and_vault_reservation`) |
+| `subtitle` | user-supplied caption tracks, one per language (`variant = 'en'`) |
+| `cover_art` | user override of embedded/derived art; user-chosen video poster frame |
+| `metadata_sidecar` | XMP / `.nfo` uploaded alongside a photo |
+| `signature` | detached signature over the file, per signer |
+
+Avoid `file_sidecars` as a name despite it being the natural
+media-world term: this codebase already uses "sidecar" for the
+tier-2 local directories (see the sidecar section below), and
+overloading it would undo that vocabulary.
+
+Note the `(…, kind, variant)` shape is identical across both tables,
+so one generic reference-source implementation parameterised by
+table + column covers both — the same one-implementation property
+the rest of this plan is built on.
+
+**Naming and comments are advisory; the boundary needs a test.** The
+one guard that actually stops a future implementor is an integration
+test asserting both halves at once: two users uploading byte-identical
+content resolve to the **same** `content_derived_blobs` rows (dedup
+works), and a client preview uploaded by one of them produces **no**
+row reachable by the other (no cross-file sharing). That test fails
+loudly the moment someone rekeys either table. Ship it with step 4,
+not after.
+
+Structural help, in descending order of reliability: the test; the
+column types (`file_id UUID` vs `source_hash VARCHAR(64)`, so a row
+cannot be copied between tables); the absence of any `file_id` column
+on `content_derived_blobs`; the table comments.
+
+### Write path — reuse `store_from_stream`, don't special-case CDC
+
+Derived blobs go through `DedupService::store_from_stream()`
+**unchanged**. An earlier draft of this plan proposed a dedicated
+single-chunk write path to avoid the manifest row; that was
+optimising the wrong thing. What it costs to reuse the standard
+path:
+
+- **+1 `chunk_manifests` row per derived blob.** `CDC_MIN_CHUNK` is
+ 65_536, and WebP q82 thumbnails land at roughly 3–8 KB (icon),
+ 15–30 KB (preview), 40–90 KB (large) — so icon and preview are
+ always below the minimum chunk size and emit exactly one chunk;
+ `large` occasionally splits into two.
+- **One manifest lookup per read**, served from RAM by
+ `manifest_cached` — not a per-read query.
+- **A CDC pass over ~12 KB** — below min-chunk, so a single pass
+ with no boundary search. Negligible.
+
+What it buys: zero new write path, zero new read path, and
+ref-counting, GC, `add_reference` / `remove_reference` and both
+consistency jobs all work on derived blobs with no changes, because
+they are already manifest-aware. One `store_from_stream` call == one
+reference == one `content_derived_blobs` row, symmetric on delete.
+
+Two gotchas that come with the reuse:
+
+1. `store_from_stream` fires `fire_blob_creation_hooks`. The only
+ non-dispatcher `BlobLifecycleHook` implementor today is
+ `ThumbnailService`, whose `on_blob_created` is a no-op — so no
+ spurious work now. But creating a thumbnail now fires
+ blob-creation hooks, so any future hook (search indexing, face
+ detection) must not treat every new blob as user content. Add a
+ `kind`/content-type guard to the hook contract before the second
+ implementor lands.
+2. GC of a *derived* blob fires `on_blob_deleted(derived_hash)`,
+ which looks for derived-of-derived rows, finds none and stops.
+ One level deep, terminates — but that's incidental rather than
+ designed. Comment it at the recursion point.
+
+### No backend-type branching in the service
+
+`src/AGENTS.md` already forbids the shape this refactor must avoid:
+
+> - **Never hand-craft blob paths.** No `blob_root: PathBuf` fields […]
+> - **Persistent state = backend**, not `/*` sidecars.
+
+`thumbnail_service` is cited there as a read-side reference impl,
+and the read side is compliant. The write side is the violation:
+`thumbnails_root: PathBuf` is exactly the banned field, and
+`get_thumbnail_path()` is the hand-crafted path.
+
+There must be **no `if backend is local { .thumbnails/… } else { blob }`
+anywhere in the service.** `ThumbnailService` holds
+`Arc`, reads and writes through it, and never learns
+which backend it is sitting on. The local-vs-remote difference is
+expressed once, as decorator composition in `common/di.rs`:
+
+```rust
+if self.config.storage.cache.enabled && active_backend_kind != StorageBackendType::Local {
+ blob_backend = Arc::new(CachedBlobBackend::new(blob_backend, &cfg));
+}
+```
+
+Local deployments write derived blobs into `/.blobs/` via
+`LocalBlobBackend` with no cache decorator (a cache would be a
+byte-identical second copy on the same disk). Remote deployments get
+the cache. Same service code both ways — and it is the same branch
+that already governs source blobs, not a new one.
+
+What this deletes from `thumbnail_service.rs`:
+
+- `thumbnails_root` field, `get_thumbnail_path()`,
+ `ThumbnailSize::dir_name()` (becomes `variant()`, feeding the DB
+ column)
+- `initialize()`'s `create_dir_all` loop
+- every `fs::read` / `fs::write` / `fs::metadata` / `remove_file`
+- the three `all_exist` stat loops → one indexed query each
+- `delete_blob_thumbnails`'s unlink loop and the duplicate of it
+ inside `on_blob_deleted`
+
+Net deletion, which is the main argument for this shape.
+
+Two adjacent cleanups in the same file: `stream_blob_to_temp` uses
+`self.thumbnails_root` as its temp directory and must move to
+`AppConfig::temp_dir` per the `OXICLOUD_TEMP_DIR` rule; and
+`store_external_thumbnail`'s `ext-{file_id}.jpg` write is the last
+hand-crafted path once the rest is converted.
+
+### Client-uploaded thumbnails — file-keyed, and NOT in `content_derived_blobs`
Some clients (NC desktop, mobile apps) upload their own encoded
previews alongside the file. These are **not derivable** — losing
them means asking the client to regenerate, which may not be
-possible (client offline, original file no longer present on
-device).
+possible.
-Same storage shape: BLAKE3 of the client-provided bytes → blob.
-The DB row distinguishes `origin = 'server_derived' | 'client_provided'`
-so consistency-check policy can differ (missing client-provided
-thumbnail = data loss finding; missing server-derived = warning,
-regenerable).
+They are also **not a function of the content**, and that makes
+content-keying a cross-user poisoning vector:
+
+1. User A uploads file X plus a preview that does not depict X.
+ There is no validation that can catch this — verifying a preview
+ faithfully represents its source means re-deriving and comparing,
+ at which point accepting the client's upload is pointless.
+2. User B uploads the same file X. Dedup matches on `source_hash`.
+3. B is served A's preview.
+
+So client previews **stay file-keyed, in `storage.file_attached_blobs`,
+and out of `storage.content_derived_blobs`.** This is a precondition for
+`content_derived_blobs` having no `source_file_id` column, not an independent
+choice — the two decisions must land together.
+
+Worked example. User A uploads `image.png` (file id `7f3e…9c`,
+content hash `a1b2c3…`), then `PUT`s their own preview:
+
+```
+storage.content_derived_blobs -- server-derived, content-keyed
+ source_hash | kind | variant | blob_hash | content_type
+ a1b2c3… | thumbnail | icon | 9a8b… | image/webp
+ a1b2c3… | thumbnail | preview | d4e5f6… | image/webp
+ a1b2c3… | thumbnail | large | c7d8… | image/webp
+
+storage.file_attached_blobs -- client-supplied, file-keyed
+ file_id | kind | variant | blob_hash | content_type | uploaded_by
+ 7f3e…9c | preview | preview | e1f2… | image/jpeg | A
+```
+
+Both sets of bytes travel the same dispatch
+(`store_from_stream` → blob → backend → encryption), so the *bytes*
+stay content-addressed and dedupe: two users uploading
+byte-identical previews converge on one object at `ref_count = 2`.
+Only the **mapping** is per-file — and the mapping is the part that
+carries the trust problem. When user B uploads the same
+`image.png`, B matches `a1b2c3…` in `content_derived_blobs` and gets the
+server-derived thumbnails; B has no `file_attached_blobs` row, so A's
+preview is unreachable.
+
+Read precedence is unchanged from today (the client's preview wins):
+`file_attached_blobs` for `(file_id, 'preview', …)` first, else
+`content_derived_blobs` for `(source_hash, 'thumbnail', 'preview')`.
+Both fold into the query the handler already issues.
+
+`uploaded_by` is new. Today there is no provenance at all on a
+client preview, and an Editor on a shared file can overwrite the
+owner's — same family as the known Editor-can-rename gap
+(`bug_drive_rename_editor_can_do_it`), and worth the same decision.
+
+Today's code is already safe, implicitly, via its choice of
+filename; the risk is losing that in the migration:
+
+- write is file-keyed — `store_external_thumbnail` writes only
+ `ext-{file_id}.jpg`, never into the `{blob_hash}.{ext}` space
+- read checks the file-keyed path *before* the content-keyed one in
+ `get_cached_thumbnail`, so a preview only surfaces for its own
+ file
+- `PUT …/thumbnail/{size}` requires `Permission::Update` on the
+ target file
+
+Two things keep the boundary after the refactor:
+
+- **The schema is self-guarding.** With no `source_file_id` column
+ there is nowhere to put a client preview, so making the mistake
+ requires writing a migration — which gets reviewed. Keeping a
+ nullable column would be an attractive nuisance; omitting it *is*
+ the enforcement.
+- **State the invariant in the migration**, in the style
+ `content_search_index.sql` already uses: *keyed by `source_hash`
+ because derived content is a pure function of the source bytes;
+ client-uploaded previews are NOT derived, are user-chosen, and
+ must never be keyed here or one user's preview would be served for
+ another user's identical file.*
+
+Note the pressure this is under: a "unify the two write paths" pass
+would produce exactly the vulnerability. The two axes are separate —
+client previews share the **dispatch** (they become blobs via
+`store_from_stream` like everything else, satisfying the
+no-backend-branching rule) while keeping **file keying** in their own
+small mapping table. Storage dedup is preserved either way, because
+the derived bytes are still content-addressed: two byte-identical
+previews converge on one object at `ref_count = 2`. Only the mapping
+is per-file, and the mapping is the part that carries the trust
+problem.
+
+**This is a live, shipping feature — not a deferred one.**
+`frontend/src/lib/utils/thumbnail.ts` generates all three canonical
+sizes client-side and `PUT`s them back, for
+`SUPPORTED_MIME_TYPE = [image/*, application/pdf, video/*]`, using
+vendored pdf.js to render page 1 of a PDF. It is the fallback for
+mime types the server cannot handle.
+
+PDF is the case that matters: **the backend has no PDF rasteriser**
+(no pdfium, no poppler), so a client-supplied PDF thumbnail is the
+only one that will ever exist. Unlike video frames, which
+`generate_video_thumbnails_background` can rebuild with ffmpeg, a
+dropped PDF thumbnail is gone until some user happens to reopen that
+document in the SPA. So `storage.file_attached_blobs` is **required
+by the migration** and ships with the thumbnail slice, not after it.
+
+PDF also gives the keying rule its sharpest example. A client-rendered
+page 1 *feels* like a pure function of the content — far more so than
+a video poster frame — which is exactly why the rule is about **who
+controls the bytes**, not about whether an honest implementation would
+be deterministic. Nothing stops a client `PUT`ting an arbitrary JPEG
+as "page 1". Content-keyed, every user holding that PDF sees the
+forgery; file-keyed, it stays with the uploader's own file.
+
+`uploaded_by` follows the `storage.shares.created_by` convention —
+`NOT NULL`, **no FK** — so the audit fact survives the uploader's
+account being deleted (users are hard-deleted here). A FK would force
+either `ON DELETE CASCADE`, destroying other people's thumbnails when
+an account closes, or `ON DELETE SET NULL`, destroying the audit trail
+this column exists for. Rows imported from the sidecar, which records
+no uploader, get an all-zeros sentinel; the UI's "removed user"
+fallback renders both cases.
+
+Unrelated bug spotted in the same function: the `PUT` uses a raw
+`fetch` with `getCsrfHeaders()` rather than `apiFetch`, so no DPoP
+proof is attached — the raw-fetch bypass class that lands on Gate C.
+It also `Promise.all`s the responses without checking status, so a
+rejection would be silent. Worth verifying against current DPoP
+enforcement on `/api/files/*`.
+
+## Copy and version semantics — the second and third axes
+
+Keying is not the only property a satellite table has. Two more must
+be declared explicitly, because neither is derivable from the key and
+getting either wrong is silent data loss:
+
+| | keyed by | on copy | on new version |
+|---|---|---|---|
+| thumbnails, transcodes | content | free — shares hash | free — new hash gets its own rows; old rows stay valid for the old version |
+| previews, subtitles, cover art | file (→ *version*, see below) | **duplicate** + take a reference | **invalidate** / attach to the new version |
+| comments | file | **do NOT duplicate** — a copy is a new artifact; the discussion belongs to the original | unaffected — a conversation spans versions |
+| WebDAV dead properties | file | duplicate (RFC 4918 §8.8) | unaffected — properties describe the resource |
+| EXIF (`storage.file_metadata`) | content ⚠ *file-keyed today* | would be free | would be free |
+
+**`on_file_copied` stops being a no-op.** It is one today only because
+every derived artifact is content-keyed, so a copy shares them for
+free. The moment `file_attached_blobs` exists, copy must duplicate its
+rows and take a blob reference for each.
+
+### The copy fan-out is exactly two sites — and they have already drifted
+
+| | single-file copy | folder-tree copy (cascading) |
+|---|---|---|
+| impl | `copy_file` CTE, `file_blob_write_repository.rs:592` | `storage.copy_folder_tree` (latest def: migration `20260902000001`) |
+| `storage.files` row | ✅ in CTE | ✅ |
+| dead properties | ✅ `dead_prop_copy` CTE arm | ✅ folder + file props |
+| blob reference | ✅ `dedup.add_reference()` — manifest-aware | ❌ hand-rolled `UPDATE storage.blobs` — **misses manifests** |
+
+That last cell is a live data-loss bug, not a plan concern — see the
+note at the end of this section. It is the same logic written twice,
+once correctly and once not, which is exactly what
+`AGENTS.md § Code duplication` forbids. Adding file-keyed tables
+without consolidating first means writing it a third and fourth time.
+
+So single-source the fan-out:
+
+```sql
+-- ONE place that knows what follows a file on copy.
+CREATE FUNCTION storage.copy_file_satellites(old_file_id UUID, new_file_id UUID)
+RETURNS void AS $$
+BEGIN
+ INSERT INTO storage.webdav_dead_properties (file_id, namespace, local_name, value)
+ SELECT new_file_id, namespace, local_name, value
+ FROM storage.webdav_dead_properties WHERE file_id = old_file_id;
+
+ INSERT INTO storage.file_attached_blobs
+ (file_id, kind, variant, blob_hash, size, uploaded_by)
+ SELECT new_file_id, kind, variant, blob_hash, size, uploaded_by
+ FROM storage.file_attached_blobs WHERE file_id = old_file_id;
+ -- then take a MANIFEST-AWARE reference per inserted blob_hash
+
+ -- storage.comments: deliberately NOT copied. See the table above.
+END;
+$$ LANGUAGE plpgsql;
+```
+
+Both paths call it — the tree function per row in `_copy_file_map`,
+the single-file path as one statement. Three properties worth having:
+
+1. **The function body *is* the copy-semantics declaration.** Comments
+ are absent, and that absence is the documented decision rather than
+ an omission someone has to notice. The taxonomy becomes executable.
+2. Adding a file-keyed table is one edit, so the two paths cannot
+ drift again.
+3. The refcount bump lives there once, manifest-aware, which fixes the
+ tree-copy bug as a consequence of consolidating rather than as a
+ separate patch.
+
+References inside it must go through the same manifest-first logic as
+`add_reference`, never `UPDATE storage.blobs`. That argues for a
+`storage.add_blob_reference(hash)` SQL helper so the contract exists
+exactly once in the database too.
+
+### Versioning (future) — free on the content side
+
+When file versioning lands, each version has its own content hash.
+The content-keyed half needs **nothing**: version N's thumbnails are
+the `content_derived_blobs` rows with `source_hash = H_N`, already
+present, already shared with any other file holding that content.
+Rolling back to an old version has its thumbnails instantly, with no
+re-derivation. Versioning and dedup turn out to be the same mechanism
+— which is the strongest retroactive argument for content-keying:
+file-keyed thumbnails would need a re-derived set per version with no
+sharing between a version and a copy of it.
+
+What propagates is a *reference*, not a copy: each `file_versions` row
+holds a reference on its Blob, which transitively keeps that version's
+derived rows alive, since `on_blob_deleted(source_hash)` only fires
+once the source Blob genuinely dies. So `file_versions` is simply
+another `BlobReferenceSource` — same shape, no new machinery.
+
+The file-keyed half is what versioning complicates: a client-uploaded
+preview belongs to a *version*, not a file, because version 2's
+content makes version 1's preview wrong. Expect
+`file_attached_blobs`'s key to migrate from `file_id` to `version_id`
+at that point. Today's `on_file_updated` (delete, then regenerate) is
+the degenerate case — versioning with history depth 1. Comments stay
+keyed on `file_id`.
+
+### ⚠ Pre-existing bug found while mapping this
+
+`storage.copy_folder_tree` bumps refcounts with
+`UPDATE storage.blobs … WHERE b.hash = hc.blob_hash`
+(migration `20260902000001`, lines 141-153). A CDC file's `blob_hash`
+names a **manifest**, so for a multi-chunk file that predicate matches
+zero rows and the copy takes **no reference at all**. Deleting the
+original then drops the manifest 1 → 0, deletes it and dereferences
+every chunk, and GC reaps them after the grace window — leaving the
+copy pointing at nothing.
+
+Single-chunk files (under `CDC_MIN_CHUNK`) escape by accident: their
+whole-file hash equals their lone chunk's hash, so the UPDATE matches,
+and `read_blob_bytes`'s no-manifest fallback still finds the bytes. So
+the symptom is size-dependent — copy a folder of files ≥ 64 KB, delete
+the original, lose the copies.
+
+The Rust caller (`file_blob_write_repository.rs:1038-1067`) only
+invokes the function, so nothing compensates at the application layer.
+Derived from reading the SQL against the refcount contract, **not from
+a reproduction** — needs a test before anyone acts on it. Same root
+cause as prerequisite 2 below: the manifest counter is written by
+`dedup_service` and reconciled by nothing, so this has been invisible.
+
+Related, much narrower: the single-file path calls `add_reference`
+*after* its CTE, best-effort with a warning on failure
+(`:708-715`), so a failure there leaves a file row holding no
+reference. Belongs in the same transaction.
## `BlobReferenceSource` — reference tracking abstraction
@@ -105,14 +735,32 @@ The extension point:
pub trait BlobReferenceSource: Send + Sync {
/// Short stable identifier for logs / consistency finding
/// `source` fields. Suggested: `"files"`, `"chunks"`,
- /// `"thumbnails"`, `"transcodes"`.
+ /// `"derived"`.
fn source_name(&self) -> &'static str;
/// Count of references this source holds on `blob_hash`.
- /// Called by `blobs_consistency` when recomputing
- /// `refcount_mismatch` findings.
+ /// **On-demand path only** (`dedup_gc` checking one reap
+ /// candidate). MUST NOT be used by the consistency sweep — see
+ /// `ref_count_sql` below.
async fn count_references(&self, blob_hash: &str) -> Result;
+ /// A correlated-subquery fragment counting this source's
+ /// references **at `level`** to the outer row's hash, e.g.
+ /// `"(SELECT COUNT(*) FROM storage.content_derived_blobs d WHERE d.blob_hash = m.file_hash)"`.
+ /// `None` when this source holds no references at that level.
+ ///
+ /// The registry sums the fragments per level into that level's
+ /// existing per-page SELECT, so each sweep stays ONE query per
+ /// page instead of degrading to (sources × rows) round-trips.
+ /// Identifiers only — never interpolate caller input.
+ ///
+ /// A source may contribute at BOTH levels, so this is **not** a
+ /// per-source constant: `FilesReferenceSource` references a chunk
+ /// for manifest-less legacy rows and a Blob for CDC rows. An
+ /// earlier draft modelled this as `fn ref_level(&self) -> RefLevel`
+ /// and was wrong for exactly that source.
+ fn ref_count_sql(&self, level: RefLevel, outer_alias: &str) -> Option;
+
/// Iterate the source's referenced blobs, paged by the
/// implementation's natural cursor (typically a DB PK). Used
/// by `backend_consistency` to walk the backend against the
@@ -151,42 +799,327 @@ registrations:
- `FilesReferenceSource` — wraps `storage.files.blob_hash`
- `ChunksReferenceSource` — wraps `storage.chunk_manifests.chunk_hashes[]`
-Tier-2 migration adds:
+Tier-2 migration adds two more — one per table, not one per `kind`,
+and both satisfied by the same generic implementation parameterised
+by table + column:
-- `ThumbnailsReferenceSource` — wraps a new
- `storage.thumbnails(hash, blob_hash, variant_spec, origin)` table
-- `TranscodesReferenceSource` — wraps
- `storage.transcodes(hash, blob_hash, target_format)` table
+- `ContentDerivedReferenceSource` — `storage.content_derived_blobs.blob_hash`
+- `FileAttachedReferenceSource` — `storage.file_attached_blobs.blob_hash`
+
+`ChunksReferenceSource` stays bespoke (array containment,
+`b.hash = ANY(m.chunk_hashes)`).
+
+### Two counters — get the level right
+
+`add_reference` bumps `chunk_manifests.ref_count` first and only
+falls back to `storage.blobs.ref_count`. So references land at
+whichever level the hash names:
+
+| Reference holder | References a… | Feeds |
+|---|---|---|
+| `chunk_manifests.chunk_hashes[]` | chunk | `storage.blobs.ref_count` |
+| `files.blob_hash` (legacy, no manifest) | whole-file blob | `storage.blobs.ref_count` |
+| `files.blob_hash` (CDC) | Blob via manifest | `chunk_manifests.ref_count` |
+| `content_derived_blobs.blob_hash` | Blob via manifest | `chunk_manifests.ref_count` |
+| `file_attached_blobs.blob_hash` | Blob via manifest | `chunk_manifests.ref_count` |
+
+Both new tables reference *manifests*, never chunks — hence the
+`level` parameter on `ref_count_sql`. They return `None` for the chunk
+level. Adding their fragments to the chunk-level recompute would
+double-count systematically.
+
+**The aliasing trap is the norm here, not an edge case.** Today's
+recompute carries a `NOT EXISTS` clause because a single-chunk file's
+`file_hash` equals its lone chunk's hash (~40% of uploads per the
+comment at `blobs_consistency_service.rs:388`). Derived blobs sit
+below `CDC_MIN_CHUNK`, so ~100% of them are single-chunk and hit that
+aliasing case. Level-correctness is not optional.
Then:
- **`dedup_gc`** — orphan iff `registry.total_references(hash) == 0`
- (with the existing grace window). No per-service GC changes.
-- **`blobs_consistency`** — `refcount_mismatch` recomputes via
- `registry.total_references`. New services register → automatically
- covered.
+ (with the existing grace window). Per-hash `count_references` is
+ fine here: candidates are already filtered to `ref_count = 0` past
+ the grace window, so the set is small.
+- **`blobs_consistency`** — `refcount_mismatch` sums the sources'
+ `ref_count_sql` fragments into its existing per-page SELECT
+ (`blobs_consistency_service.rs:395-411`), which is already
+ set-based. It must NOT be rewritten to call
+ `registry.total_references` per row — that would turn one query per
+ page into (sources × blobs) round-trips.
- **`backend_consistency`** — walks the backend and unions all
`list_referenced_blobs` streams for the "did we lose bytes"
check.
+### Consistency coverage — what is and isn't tracked
+
+Findings each job reports today, and where the new tables land:
+
+| # | Edge | Direction | Mechanism | Status |
+|---|---|---|---|---|
+| 1 | backend → `storage.blobs` | orphan bytes | `orphan_blob` (backend_consistency) | ✓ |
+| 2 | `storage.blobs` → backend | missing bytes | `blob_missing_from_backend` | ✓ |
+| 3 | chunk bytes | corruption | `blob_corrupted`, `blob_unreadable` | ✓ |
+| 4 | manifest → chunks | chunk reaped | `chunk_missing` (files_consistency) | ✓ |
+| 5 | `files` → Blob | dangling | `missing_blob` (files_consistency) | ✓ |
+| 6 | `storage.blobs.ref_count` | recompute | `refcount_mismatch` | ✓ chunk level only |
+| 7 | `chunk_manifests.ref_count` | recompute | — | ✗ gap, pre-existing |
+| 8 | manifest orphan reaping | GC predicate | `OR NOT EXISTS(files)` | ⚠ **breaks — see below** |
+| 9 | derived/attached → Blob | dangling | — | ✗ new check needed |
+| 10 | `content_derived_blobs.source_hash` → Blob | orphan mapping | — | ✗ new check needed |
+| 11 | chunk at `ref_count = 0` past grace, still present | GC lag | — | ✗ a stalled GC is silent |
+| 12 | `blob_extracted_text`, `faces.faces` orphans | dependents | search worker self-janitors; `faces` unverified | ~ verify |
+| 13 | `file_attached_blobs.file_id` → `files` | dangling | FK `ON DELETE CASCADE` | ✓ DB-enforced |
+
+**`backend_consistency` needs no change.** Its probe asks "does this
+backend object have a `storage.blobs` row?" — and the backend holds
+**chunks**, which neither new table ever references. A derived blob's
+bytes are registered there by `store_from_stream` like any other
+chunk. Adding the new tables to that probe would look up Blob hashes
+in a table of chunk hashes and, for multi-chunk blobs, match nothing.
+
+Rows 9-11 are new work; row 12 wants a check. Row 7 is the
+pre-existing hole. Row 8 is the blocker:
+
+### ⚠ Blocker — `dedup_gc` will delete every derived blob
+
+The zero-ref manifest sweep (`dedup_service.rs:2574`) is:
+
+```sql
+DELETE FROM storage.chunk_manifests
+ WHERE m.ref_count <= 0
+ OR NOT EXISTS (SELECT 1 FROM storage.files f WHERE f.blob_hash = m.file_hash)
+```
+
+That `OR` hardcodes **"`storage.files` is the only thing that can
+reference a manifest."** A thumbnail's manifest has `ref_count = 1`,
+held by its `content_derived_blobs` row, so the first condition is
+false — but no `storage.files` row points at a thumbnail's Blob hash,
+so `NOT EXISTS` is true, the `OR` fires, and the manifest is deleted,
+its chunks dereferenced and the bytes reaped.
+
+Ship `content_derived_blobs` without changing this and **the first
+`dedup_gc` run destroys the entire derived tier.** Not drift —
+immediate, total loss.
+
+It is also the missing half of row 7: that `OR` is the self-healing
+hack that makes an unreconciled `chunk_manifests.ref_count`
+survivable. Adding referrers breaks the assumption the hack rests on,
+which is why 7 and 8 have to be fixed together. The predicate becomes
+registry-driven — the union of every source's manifest-level
+`ref_count_sql`, not a hardcoded table name.
+
+### Three hard prerequisites, not sequencing preferences
+
+0. **Fix the `dedup_gc` manifest predicate** (row 8) **before a single
+ derived blob is written.** Everything else in this plan is
+ recoverable; this one silently deletes data.
+1. **Registry before the tables.** `blobs_consistency` derives
+ expected refcounts from `storage.files` + manifests only. Ship
+ `storage.content_derived_blobs` first and *every* derived blob becomes a
+ `refcount_mismatch` finding — a flood, and one an operator might
+ "repair".
+2. **`chunk_manifests.ref_count` must become verified.** It is
+ currently maintained by `dedup_service` and reconciled by
+ *nothing*: `blobs_consistency` only recomputes
+ `storage.blobs.ref_count`, and the manifest-level integrity it
+ defers to `files_consistency::chunk_missing` is a different check
+ (manifests pointing at reaped chunks). Since both new tables feed
+ the manifest counter, registering a source would give the
+ *illusion* of coverage, not coverage. Failure mode: a manifest
+ stuck at `ref_count > 0` forever, so its chunks are never
+ reclaimed. A manifest-level recompute is part of this work.
+
+## Read path and caching
+
+Request carries `(file_id, size, format)`; the derived hash is
+BLAKE3 of bytes that don't exist yet, so it is not computable from
+the request. Read order:
+
+1. **moka RAM tier** — keyed by `(source_hash, size, format)`.
+ (Today it is keyed by `file_id`; rekeying to `source_hash` costs
+ nothing — the handler already has the hash from the row it
+ loaded — and stops N copies of one photo occupying N entries for
+ identical bytes.)
+2. **DB** — `content_derived_blobs` lookup for the variant. This is free:
+ the miss path already queries `get_blob_hash(&id)`, so a
+ `LEFT JOIN` on `content_derived_blobs` returns the source hash and the
+ derived hash in one query. Net DB cost unchanged from today.
+3. **`dedup.read_blob_bytes(derived_hash)`** — through the normal
+ backend stack, which is where the disk cache lives.
+4. Generate only if step 2 found no row.
+
+**The disk cache is `CachedBlobBackend`, reused unchanged.** No
+thumbnail-specific cache, no second root path. Routing derived
+blobs through the same stack gets, for free:
+
+- **single-flight per hash** — a gallery cold-load where 50 clients
+ race one thumbnail collapses to one S3 GET
+- **write-through on put** — the instance that generated the
+ thumbnail already has it locally, so upload→view-gallery never
+ round-trips to S3
+- byte-budget eviction with unlink, and a restart-survivable index
+
+**One thing to test rather than assume.** Thumbnails and source
+blobs have opposite cache profiles: small / hot / expensive to
+regenerate versus large / cold / cheap to re-fetch. Sharing one LRU
+budget (`OXICLOUD_STORAGE_CACHE_MAX_SIZE`, default 50 GB) means a
+sequential multi-GB video read is exactly the scan pattern that
+flushes a working set — and flushing thumbnails costs a re-render,
+not a re-download. moka 0.12's TinyLFU admission *should* resist
+this (a one-shot large entry denied rather than evicting
+frequently-hit small ones), and the eviction listener unlinks on
+`RemovalCause::Size` so a denied entry shouldn't leak its file. Both
+deserve a test, because the failure mode is silent: unexplained CPU
+on the thumbnail path, not a cache metric.
+
+If it does interfere, the fix that preserves the one-implementation
+rule is **two instances of `CachedBlobBackend` with separate
+budgets** — same type, same factory, different config — not a second
+cache type. Honest cost: `DedupService` would need a second backend
+handle plus a content-class selector, since derived blobs have
+manifests and must still be reassembled through `DedupService`. Ship
+the shared cache, measure, split only if the test says so. The knob
+would be `OXICLOUD_STORAGE_DERIVED_CACHE_MAX_SIZE` alongside the
+existing `OXICLOUD_STORAGE_CACHE_MAX_SIZE`.
+
+## Cost consequences to budget for
+
+1M photos × 3 sizes ≈ **3M additional backend objects and 3M
+additional `storage.blobs` + `chunk_manifests` rows** (~100 KB of
+derived content per photo, so ~100 GB total) — 3M and not 6M because
+only the canonical WebP is persisted; the JPEG fallback never reaches
+tier 3. Two costs land:
+
+- **PUT requests at upload** — one-off, modest (~$15 per 3M on AWS
+ pricing). Use `put_blob_from_bytes_unsynced` + a batched
+ `sync_blobs`, not `put_blob_from_bytes`: the latter does a
+ `head_object` before every PUT on S3, doubling the request count
+ for an idempotency check content-addressing already guarantees.
+- **`blobs_consistency` request amplification** — it does one
+ `blob_exists` HEAD per blob row, so 4× the rows is 4× the S3
+ requests, forever. This is the dominant recurring cost. Fix by
+ diffing against `list_blob_hashes` pages in bulk (one LIST per
+ 1000 keys instead of 1000 HEADs) rather than per-row probes.
+ Alternative: exempt derived rows from the byte-level check, since
+ they are regenerable — but the bulk-LIST fix is better and helps
+ source blobs too.
+
+Note for object-store deployments: IA/Glacier tiers bill a 128 KB
+minimum per object, so an 8 KB icon is billed at 128 KB. **Open
+option** (config, not a code branch): persist only the `large`
+variant to tier 3 and derive icon/preview from it on demand — the
+render path already decodes once for all sizes, and resampling an
+800px WebP is sub-millisecond. That is 1M objects instead of 3M. It
+changes only *how many variants get a `content_derived_blobs` row*, so it
+stays a single code path.
+
## Sidecar directories after this refactor
| Sidecar today | After |
|---|---|
-| `.thumbnails/` | Persisted as derived blobs in tier 3. `.thumbnails/` becomes a pure read-through cache (tier 1-ish; ephemeral, per-instance). |
-| `.transcoded/` | Same shape as thumbnails. |
-| `.blob-cache/` | Already a cache; stays. Owned by `CachedBlobBackend`. |
+| `.thumbnails/` | **Gone.** Derived blobs live in tier 3; caching is `CachedBlobBackend` in `.blob-cache/`, keyed by hash like every other blob. |
+| `.transcoded/` | Gone, same shape. |
+| `.blob-cache/` | Stays. Owned by `CachedBlobBackend`, path via `OXICLOUD_STORAGE_CACHE_PATH`. Now serves derived blobs too. |
| `.search-index/` | Open question — see non-goals. |
| `.plugin-logs/` | Ops-local; stays. |
| `.uploads/` | Tier 1 already; migrates to `OXICLOUD_TEMP_DIR`. |
-The persistent-spool env var reserved:
+**`OXICLOUD_SPOOL_DIR` is probably no longer worth adding.** The
+2026-08-02 draft reserved it as the home for `.thumbnails/`,
+`.transcoded/` and `.blob-cache/`. The first two now disappear
+entirely rather than becoming local caches, and `.blob-cache/`
+already has its own `OXICLOUD_STORAGE_CACHE_PATH`. That leaves
+`.search-index/` (a non-goal) and `.plugin-logs/` (ops-local) — not
+enough to justify a new config surface. Either drop step 2 below or
+reduce it to documenting the existing `OXICLOUD_STORAGE_CACHE_PATH`
+as the tier-2 relocation knob.
-- **`OXICLOUD_SPOOL_DIR`** — path for the local read-through
- caches (`.thumbnails/`, `.transcoded/`, `.blob-cache/`). Default
- `/spool`. Ops can point it at a different disk
- than tier-3 storage; multi-instance deployments accept per-
- instance rebuild OR mount a shared FS here.
+## Migrating the existing sidecar content
+
+### Inventory → destination
+
+| On disk today | Goes to |
+|---|---|
+| `.thumbnails/{icon\|preview\|large}/{blob_hash}.webp` | `content_derived_blobs(source_hash = stem, kind='thumbnail', variant = dir, blob_hash = BLAKE3(bytes), content_type='image/webp')` |
+| `.thumbnails/{…}/{blob_hash}.jpg` | **Deleted, not imported** — tier 3 is canonical-only |
+| `.thumbnails/{…}/ext-{file_id}.jpg` | `file_attached_blobs(file_id = stem, kind='preview', variant = dir, content_type='image/jpeg', uploaded_by = sentinel)` |
+| `.transcoded/webp/{file_id}…` | Deferred to the transcode slice — and it is a **re-keying**, since transcodes are file-keyed today (`get_cache_path(file_id, …)`) and must join `storage.files` to resolve `blob_hash` |
+| `.transcoded/` skip markers | Not imported — a negative verdict costs one transcode attempt to recompute |
+
+The server-derived thumbnail case is lossless: the path already
+encodes `(source_hash, variant)`, and the derived hash is just BLAKE3
+of the file's own bytes. **No source read, no decode.**
+
+### Why import rather than regenerate
+
+Regenerating 1M photos × 3 sizes means 1M GETs of **full-size
+originals** — multi-MB each, from S3 on remote backends — plus 1M
+decodes and 3M encodes. Importing reads ~20 KB local files, hashes,
+and PUTs. It is the difference between a background job and a
+maintenance window, and on object storage it is a real egress and
+request bill.
+
+### The `ext-` case, unfolded
+
+This is the hard one, and it triages by the *file's* mime type:
+
+- **`application/pdf` — must import.** Irreplaceable: no server-side
+ rasteriser exists. Dropping it loses the thumbnail until a user
+ reopens the document in the SPA.
+- **`video/*` — should import.** `generate_video_thumbnails_background`
+ could rebuild it, but an ffmpeg run per video costs far more than
+ copying a 20 KB JPEG.
+- **`image/*` — safe to drop.** The server regenerates from source and
+ produces a *better* result (WebP rather than the client's JPEG).
+ Importing is still cheaper; either is defensible.
+
+Three complications specific to this path:
+
+1. **Three files per file**, not one — the client uploads icon,
+ preview and large. On a document-heavy install
+ `file_attached_blobs` can rival the derived table in row count.
+2. **`file_id` may be stale.** The file may since have been deleted.
+ Verify against `storage.files.id`; skip and log otherwise.
+3. **No uploader is recorded.** Use the all-zeros sentinel (possible
+ only because `uploaded_by` has no FK).
+
+### Strategy — read-through, then a batch tail
+
+**Phase 1 (release N): dual read.** The read path checks the new
+tables first, falls back to the legacy sidecar on a miss, and imports
+that entry inline before serving. Hot content migrates itself under
+real traffic. Writes go only to the new tables. The legacy path lives
+in **one clearly-marked module** — it temporarily reintroduces the
+`thumbnails_root` construction this plan exists to delete, which is
+acceptable only with a named removal release.
+
+**Phase 2: `derived_import`**, a registered `JobRegistry` job (*not* a
+sqlx migration — migrations are SQL-only and run at boot; this walks a
+filesystem and a remote backend for hours). It sweeps the cold tail:
+
+- **Resumable** — checkpoint a `(dir, filename)` cursor, same shape as
+ the consistency jobs.
+- **Idempotent** — `INSERT … ON CONFLICT DO NOTHING`, and on conflict
+ *release* the reference the blob write just took, or re-runs inflate
+ refcounts.
+- **Verifies the source exists** before inserting. `.thumbnails/` can
+ hold orphans if `on_blob_deleted` ever failed, and `source_hash` has
+ no FK — importing one creates a row pointing at nothing that holds a
+ reference forever.
+- Reports imported / skipped-orphan / skipped-jpeg / failed counts.
+
+**Phase 3 (release N+1): delete** the fallback module and the sidecar
+directories, gated on the job reporting an empty tail.
+
+### The "just delete it" opt-out is no longer universally safe
+
+For a small install with no PDFs, `rm -rf .thumbnails/` and lazy
+regeneration is still fine — the content is regenerable by
+definition. **PDFs break that**, since nothing server-side can rebuild
+them. So the opt-out is safe only where `application/pdf` thumbnails
+don't exist, and the importer should refuse to run in delete-only mode
+if it finds any.
## Delivery order
@@ -196,19 +1129,60 @@ hardcoded SQL). New sources bolt on independently.
1. **`BlobReferenceSource` trait + registry** in
`application/ports/`. `FilesReferenceSource` and
- `ChunksReferenceSource` implementations mirroring current SQL;
- wire into `dedup_gc` + `blobs_consistency` behind an integration
- test that proves the union equals the pre-refactor count on a
- real DB.
-2. **`OXICLOUD_SPOOL_DIR`** — config + `example.env` + docs +
- `AppConfig::spool_dir`. Migrate `CachedBlobBackend` cache path
- default to `/blob-cache/`.
-3. **`ThumbnailService` writes go through the backend**. New
- `storage.thumbnails` table + `ThumbnailsReferenceSource`. Local
- `.thumbnails/` sidecar becomes a read-through cache pattern.
-4. **`ImageTranscodeService`** — same shape as thumbnails.
-5. **Client-uploaded thumbnails** — new `origin` column + upload
- API path if needed.
+ `ChunksReferenceSource` implementations mirroring current SQL,
+ with per-level `ref_count_sql` fragments summed into the existing
+ per-page SELECT; wire into `dedup_gc` + `blobs_consistency` behind
+ an integration test that proves the union equals the pre-refactor
+ count on a real DB. **Blocks step 5** (prerequisite 1).
+2. **`dedup_gc` manifest predicate becomes registry-driven** —
+ replaces the hardcoded `OR NOT EXISTS (… storage.files …)` with the
+ union of every source's manifest-level fragment. **Hard blocker**
+ (prerequisite 0): until this lands, writing a derived blob means
+ the next GC run deletes it. Ship with an integration test that
+ creates a manifest referenced *only* from `content_derived_blobs`
+ and asserts GC leaves it alone.
+3. **Manifest-level refcount verification** — recompute
+ `chunk_manifests.ref_count` against its actual referrers, the
+ counter nothing reconciles today. **Also blocks step 5**
+ (prerequisite 2): without it, derived-blob refcount drift is
+ invisible, and step 2 removes the `OR` that used to mask it.
+4. ~~`OXICLOUD_SPOOL_DIR`~~ — reduced to a docs change, or dropped;
+ see the sidecar section.
+5. **`ThumbnailService` writes go through `DedupService`**. New
+ `storage.content_derived_blobs` table + `ContentDerivedReferenceSource`.
+ Deletes `thumbnails_root`, `get_thumbnail_path`, and every
+ filesystem call in the service. Fold the `content_derived_blobs` lookup
+ into the handler's existing `get_blob_hash` query.
+6. **`blobs_consistency` bulk-LIST diff** — before, or immediately
+ after, step 5 lands at scale; per-row HEADs do not survive a 4×
+ row count.
+7. **`ImageTranscodeService`** — same shape, `kind = 'transcode'`,
+ no new table.
+8. **`storage.copy_file_satellites` consolidation** — collapse the two
+ copy paths onto one helper, with a manifest-aware reference bump.
+ **Blocks step 9**: adding a file-keyed table before this means
+ writing the same cascade a third and fourth time, into a pair of
+ sites that have already drifted once.
+9. **`storage.file_attached_blobs`** — **required, not deferred**: the
+ client-side generator already ships and PDF thumbnails have no
+ server-side regeneration path, so the migration depends on this
+ table. Lands with step 5. File-keyed, never in
+ `content_derived_blobs`. Register it in `copy_file_satellites` and
+ declare its version semantics.
+10. **`derived_import` job + the dual-read fallback** — see the
+ migration section. Phase 3 (deleting the fallback and the sidecar
+ dirs) is a separate later release, gated on an empty tail.
+11. **`DedupService` → `BlobHandler` rename** — decided, mechanical,
+ 34 files. Standalone commit, `src/AGENTS.md` updated with it. Can
+ land at any point; last is easiest, since every earlier slice
+ would otherwise rebase across it.
+
+Tracked separately, **not** part of this plan: the
+`storage.copy_folder_tree` refcount bug (see the copy section). It is
+a production data-loss bug on a path this plan doesn't otherwise
+touch, so it wants its own PR and its own reproduction test — but step
+8 subsumes the fix, so sequence them together to avoid conflicting
+edits to the same function.
Each slice is independently mergeable. Delivery span: rough
estimate ~2 weeks end-to-end.
@@ -229,15 +1203,27 @@ GC). The name `DedupService` narrates HOW it works, not WHAT it
is — new service authors read the name and don't realise they
should be routing every blob read through it.
-Suggested rename: **`BlobHandler`** (or `ContentStore` /
-`BlobStore` — pick one and commit). Public surface stays
-identical; consumers write `Arc` and call
-`blob_handler.read_blob_bytes(hash)`. Internal doc-comments
-document dedup + CDC + GC as strategies.
+**Decided 2026-08-16: `BlobHandler`.** (`ContentStore` / `BlobStore`
+were the alternatives.) Public surface stays identical; consumers
+write `Arc` and call `blob_handler.read_blob_bytes(hash)`.
+Internal doc-comments document dedup + CDC + GC as strategies.
-Scope: ~35 files (grep `DedupService|dedup_service`), mechanical.
-Keep as one commit inside the tier-2 refactor so reviewers see
-"rename" independently from the substantive changes.
+Scope, measured: **34 files, 209 occurrences** of
+`DedupService|dedup_service` under `src/`. Mechanical. Keep it as one
+standalone commit so reviewers see "rename" independently from the
+substantive changes — it is the noisiest diff in this plan and the
+least interesting.
+
+**`src/AGENTS.md` must change in the same commit.** Lines 20-21 name
+`Arc` as *the* canonical read abstraction and list its
+methods; leaving them would point the rule at a type that no longer
+exists — and that rule is what stops new services from taking
+`Arc` directly. The local variable name
+`dedup` is used pervasively in call sites; renaming it to
+`blob_handler` is what produces most of the 209 occurrences, so decide
+up front whether the variable follows the type (recommended — the
+whole point is that readers stop thinking "dedup" when they mean
+"read file content").
### 2. `blob` overloaded across two scales
@@ -291,6 +1277,36 @@ lands so we don't stack schema changes.
- **Client thumbnail negotiation protocol** — the wire-level API
for how clients push their previews. Design piece for the
photo/mobile team when there's a real feature ask.
+- **A per-derived-content cache type.** Explicitly rejected: the
+ disk cache is `CachedBlobBackend`, instantiated by the existing
+ DI branch. Splitting budgets means a second *instance*, never a
+ second implementation.
+- **Key/value metadata on files.** A separate plan. It is not a blob
+ table: k/v pairs are rows, so they belong in their own schema, NOT
+ as a `file_attached_blobs` row with `kind = 'metadata'`. Both
+ tables here map a key to exactly one blob hash; arbitrary k/v has
+ different cardinality, indexing and query needs.
+
+ But it will hit **the same content-vs-file split**, so it should
+ inherit the convention rather than invent one:
+ `storage.content_derived_*` for anything that is a pure function of
+ the bytes (EXIF, dimensions, duration, extracted text — the split
+ already exists as `storage.blob_extracted_text` and `faces.faces`),
+ `storage.file_attached_*` for anything user-supplied or
+ per-resource (tags, descriptions, custom properties — WebDAV dead
+ properties are already this shape). The purity rule in the Keying
+ section is not blob-specific; it governs any content-addressed
+ store.
+
+ One hazard is sharper for metadata than for thumbnails. Content-
+ keying is safe when the datum is a function of bytes the caller
+ already holds — a thumbnail of your own file reveals nothing new.
+ A content-keyed datum that encodes *another user's input* leaks
+ across the dedup boundary twice over: user B reads A's text, and
+ learns that someone else holds identical content. So for k/v the
+ purity rule carries an anti-enumeration duty too, alongside the
+ poisoning one — cf. the dedup blob anti-enumeration policy in
+ `project_d7_policy_calls`.
## References
@@ -301,6 +1317,10 @@ lands so we don't stack schema changes.
- `docs/plan/storage-key-rotation.md` — encryption/rotation applies
to derived blobs too.
- `src/AGENTS.md` — the read-side rule enforcing backend
- abstraction (already shipped alongside this plan doc).
+ abstraction, the no-hand-crafted-paths rule, and the
+ persistent-state-is-backend rule this plan implements.
+- `migrations/20260701000000_content_search_index.sql` — the
+ content-keying precedent (`storage.blob_extracted_text`) and its
+ rationale.
- Memory note `project_services_bypassing_blob_backend` — audit
history of the pre-normalisation bypasses.
diff --git a/docs/plan/hidden-system.md b/docs/plan/hidden-system.md
new file mode 100644
index 00000000..c9b35aaf
--- /dev/null
+++ b/docs/plan/hidden-system.md
@@ -0,0 +1,246 @@
+# Plan — Hidden system drive for user-owned objects
+
+**Status:** design captured 2026-08-21. Not implemented. Sibling to
+`docs/plan/derived-blobs.md`, which answers "where does *derived*
+content live"; this one answers "where do *user-owned binaries* live".
+The two share one rule — **point at a file, never at a blob** — and
+that rule is the reason neither needs new blob-referencing tables.
+
+## Problem — binaries in the users row
+
+`auth.users.image TEXT` (migration `20260526000000_add_user_image.sql`)
+holds the avatar inline, up to 512 KiB. It is the wrong home, and the
+cost is already measured rather than theoretical:
+
+- **It TOASTs, and every wide read pays.** The repository comment on
+ `get_users_by_ids` records a group fan-out that "detoasted + shipped
+ + parsed M avatars purely to discard them", fixed by adding a narrow
+ projection (`benches/ROUND12.md §Q1`, `ROUND13.md §Q1`). That
+ workaround exists *because* the column is in the wrong place; the
+ rule it leaves behind — "add a wide sibling rather than widening
+ this one back" — is a permanent tax on every future query.
+- **Base64 inflation.** A ~384 KB image becomes ~512 KB of TEXT.
+- **No dedup.** N users sharing a default or IdP-supplied avatar cost
+ N copies.
+- **None of the storage stack applies** — no `EncryptedBlobBackend`,
+ no backend migration, no key rotation, no local cache, no
+ consistency coverage.
+- **Backups and replication carry it.** Binary weight lands in the
+ logical dump and on every replica, forever.
+
+The same pressure is coming for the UI background and a signature
+image, so this needs a general answer, not another column.
+
+## The rule — point at a file, never at a blob
+
+```sql
+ALTER TABLE auth.users
+ ADD COLUMN avatar_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL,
+ ADD COLUMN background_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL,
+ ADD COLUMN signature_file_id UUID REFERENCES storage.files(id) ON DELETE SET NULL;
+```
+
+`storage.files` is already a `BlobReferenceSource`, already covered by
+every consistency edge, already GC-integrated, already copy- and
+version-aware. A pointer to a *file* therefore adds **zero new
+reference sources and zero new consistency edges** — `auth.users`
+holds no blob reference at all, only a pointer to a row that does.
+Deleting the file decrements the blob refcount through the existing
+file-deletion path.
+
+Point at `id`, never at a name or path: a rename must not break a
+profile.
+
+For a small fixed set, **columns beat a table.** A table becomes right
+only when the object set is open-ended, and it would cost exactly what
+the pointer avoids.
+
+### Rejected alternatives
+
+| Option | Why not |
+|---|---|
+| **New table → blobs/chunks** | Needs a new `BlobReferenceSource`, a new fragment in the manifest sweep, and a new dangling check. Precisely the complexity the pointer removes. |
+| **Direct backend paths** (`profile/{uuid}/avatar.png`) | `backend_migration` enumerates *blobs*, so a Local→S3 cutover **silently drops every avatar**. `EncryptedBlobBackend` is hash-keyed, so writes either bypass encryption or need a parallel path. `backend_consistency` raises `unknown_backend_file` (severity `anomaly`, "non-canonical file in blob namespace") per object on every sweep. And a fixed key overwritten in place breaks the immutability everything else rests on, killing `Cache-Control: immutable`. |
+| **Reserved folder in the user's own drive** (`.profile/`, `.oxiprofile/`) | The user can write to it, so there are **two write paths and only one is validated** — the upload endpoint's format/size checks are bypassable over WebDAV and sync. It is in the sync tree, so "hidden" is not hidden in the protocols that matter. Existence-by-path drags back the extension problem. And any reserved name squats a namespace users own — `.profile` is a POSIX shell file, so it collides by default for anyone syncing a Linux home. |
+| **One drive per user** | Per-user creation at signup, backfill for existing users, per-user quota exemption, and a cascade on account deletion. All avoidable — see below. |
+
+Underneath all of it: these objects are owned by the **application on
+the user's behalf**, not by the user as documents. Putting them in a
+document tree conflates the two, and every problem above follows.
+
+## The hidden system drive
+
+**One shared drive, not one per user.** `kind = 'system'`, alongside
+today's `CHECK (kind IN ('personal', 'shared'))`. Files inside are
+owned by their respective users via the normal `created_by` /
+ownership columns; the drive is only a container.
+
+Sharing one drive drops per-user creation at signup, backfill for
+existing users, and per-user quota exemption. Deleting a user becomes
+a query over `storage.files` rather than a drive cascade.
+
+Properties it needs:
+
+- **Hidden at drive enumeration.** This is the single filter point,
+ and it is why the drive beats a folder: a folder must be filtered in
+ directory listings, search results, recent items, trash, photo
+ indexing and sync deltas, whereas a drive is filtered once where
+ drives are listed. Every surface must honour it — REST, WebDAV,
+ NextCloud, search, quota reporting. **A missed filter is the
+ characteristic bug of this design**, so it deserves a test per
+ surface rather than per call site.
+- **Trash disabled.** Otherwise every replaced avatar lands in a trash
+ nobody can see, holding a blob reference that GC cannot reclaim
+ while retention keeps it alive — invisible storage growth with no
+ signal. Deletion here is immediate.
+- **Exempt from the user quota envelope.** Nobody should pay quota for
+ their own avatar.
+- **Created at install, fail-fast at boot.** If the drive is missing,
+ panic rather than silently disabling profile objects — a silently
+ absent avatar surface is worse than a refusal to start.
+- **Visible to admins.** Ops need to see it for storage accounting
+ even though it is hidden from users.
+
+## Visibility — per kind, in code
+
+Reads go through a service method carrying an explicit policy, audited
+like any other authorization decision. Because the column set is fixed
+and small, the policy is a `match`, not stored data — there is nothing
+to misconfigure, and adding a column forces adding an arm:
+
+| Object | Who may read | Why |
+|---|---|---|
+| `avatar` | **the same rule as profile visibility** | Not "any authenticated user". `AGENTS.md` has `user_profile.rejected` return **404, never 403**, for an external caller with no relationship, specifically so existence cannot be confirmed. An avatar endpoint answering 200 for any caller is an oracle around that control. |
+| `background` | owner only | Nobody else has a reason to fetch it. |
+| `signature` | owner only, plus the document render path | A handwritten signature is forgery material. It is "public" only in the sense that it appears on documents you may already read — which argues for rendering it into those documents, not exposing it as a directly-readable object. |
+
+Note the consequence: the drive's own permission model is **not** what
+governs these reads. The object lives in a drive and is read through a
+different door. That is a deliberate choice, not an oversight — record
+it so nobody later "fixes" it by granting cross-user drive access.
+
+## What must NOT live here
+
+> **If losing control of it is a security incident rather than a
+> cosmetic bug, it stays in the database.** The system drive is for
+> user-facing binaries.
+
+So E2E/Vault key material — public key bundle, passphrase-wrapped
+private key, recovery kit — stays in `auth.users` columns. Four
+reasons, the last decisive:
+
+1. **Failure-mode asymmetry.** The characteristic bug here is a missed
+ listing filter. For a wallpaper that is cosmetic; for key material
+ it is disclosure.
+2. **Atomicity.** Rotating a passphrase rewraps the private key
+ *together with* the credential change. A DB column makes that one
+ transaction; a file write plus a column update cannot be atomic.
+3. **Size.** A few KB — blob storage buys nothing.
+4. **`EncryptedBlobBackend` encrypts under a key the server holds.**
+ For E2E material the whole premise is that the server *cannot*
+ decrypt. Routing a wrapped private key through the blob layer
+ encrypts it twice, once under a key the operator controls, adding
+ no protection while creating the impression of it.
+
+Users who want to store genuinely private *files* already have the
+personal drive, with the full AuthZ engine behind it. There is no gap.
+
+## Migrating the avatar off `auth.users.image`
+
+Volume is one row per user, so unlike the thumbnail migration this
+needs **no read-through phase** — a single batch job is enough.
+
+**Phase 1.** Add the pointer columns and the system drive. Write path
+switches to files; read path prefers `avatar_file_id` and falls back
+to `image` when null.
+
+**Phase 2.** `profile_image_import`, a registered `JobRegistry` job
+(subject-first naming, per convention). For each user with a non-null
+`image`:
+
+1. Decode the data URI; skip and log if it does not parse, rather than
+ failing the batch.
+2. `store_from_stream` the decoded bytes → derived blob + manifest.
+3. Insert a `storage.files` row in the system drive, owned by that
+ user.
+4. Set `avatar_file_id`.
+
+Idempotent (`WHERE avatar_file_id IS NULL`), resumable via a user-id
+cursor, and reports imported / skipped-unparseable / failed counts.
+
+**Phase 3.** Drop `auth.users.image` and the fallback, gated on the
+job reporting zero remaining. Dropping the column is what actually
+reclaims the TOAST weight and retires the narrow-projection rule in
+`get_users_by_ids`.
+
+**IdP-sourced avatars.** OIDC login already refreshes the avatar
+("same IdP avatar, already verified" — `user_pg_repository.rs:1435`).
+That path must be converted at Phase 1, not Phase 3, or it keeps
+writing to a column the migration is draining.
+
+## Object catalogue
+
+**Now:** avatar, UI background, signature image.
+
+**Strong future candidates** — these are what justify a drive rather
+than three columns and a corner:
+
+| Object | Why it fits |
+|---|---|
+| **Data exports** (GDPR takeout, drive-export zip) | Generated async, large, downloadable, should expire. Today there is nowhere to put them. |
+| **Staged imports** (Google Takeout, NextCloud export) | Multi-step ingestion needs durability beyond a temp file. |
+| **Share-page branding / logo** | Per-user or per-org, served on public share pages. |
+
+**Same problem, different owner — this drive does not help:**
+`carddav.contacts.photo_url TEXT` (contact photos, today a URL or an
+inlined data URI) and CalDAV `ATTACH` event attachments. They are keyed
+by contact and by event, not by user. But the *pointer* generalises:
+`contacts.photo_file_id UUID REFERENCES storage.files(id)` solves them
+with no new blob-referencing table either — they simply live in the
+address book's or calendar's own drive rather than here. Own plan.
+
+## Operational details
+
+- **Replace must delete.** Write new file → update pointer →
+ hard-delete the previous file. `ON DELETE SET NULL` protects the
+ pointer when a file vanishes, but nothing deletes the old file
+ because the pointer moved.
+- **Validation lives at the endpoint** and is now the only write path,
+ which is the point of not using a user-writable location. Enforce
+ format, dimensions and size there.
+- **Account deletion** deletes the user's files in the system drive
+ explicitly; the pointer columns are on the row being deleted anyway.
+
+## Non-goals
+
+- **Secrets of any kind.** See the discriminator above.
+- **Per-user system drives.** One shared drive; revisit only if
+ per-user quota or trash semantics ever become necessary.
+- **Contact photos and event attachments.** Same pointer pattern,
+ different owner, different drive — separate plan.
+- **A generic "user objects" API.** The column set is fixed and small
+ on purpose. Reach for a table only when it demonstrably is not.
+
+## Open questions
+
+- Who owns the system drive row itself, and what does
+ `drives_consistency` expect of a drive with no human owner?
+- Does the signature object survive the "owner only" rule, or does
+ document rendering need a broader read path than expected?
+- Should exports live here or in a short-lived namespace with its own
+ expiry, given they are the only candidate with a natural TTL?
+
+## References
+
+- `docs/plan/derived-blobs.md` — the sibling plan; shares the
+ point-at-a-file rule and documents the consistency coverage matrix
+ these objects inherit for free.
+- `migrations/20260526000000_add_user_image.sql` — the column being
+ retired.
+- `migrations/20260802100000_drives_schema_additive.sql` — the
+ `kind IN ('personal','shared')` constraint this extends.
+- `src/AGENTS.md` — the backend-abstraction rules, and the
+ anti-enumeration pattern the avatar visibility rule follows.
+- Memory `project_drive_naming_and_vault_reservation` — "Vault"
+ reserved for the future E2E kind whose key material this plan
+ explicitly keeps out of the drive.
diff --git a/frontend/src/lib/api/endpoints/adminJobs.ts b/frontend/src/lib/api/endpoints/adminJobs.ts
index 71bf7d65..5722462e 100644
--- a/frontend/src/lib/api/endpoints/adminJobs.ts
+++ b/frontend/src/lib/api/endpoints/adminJobs.ts
@@ -60,11 +60,21 @@ export function listJobs(): Promise {
}
/**
- * `POST /api/admin/jobs/{name}/trigger?force=X&deep=X` — dispatch a job
- * on-demand. `force` bypasses per-tenant idempotency checks (e.g.
- * `trash_cleanup` skipping when nothing is due). `deep` opts into slow
- * variants (currently only `storage_consistency`, propagated by
- * `consistency_batch` to every child).
+ * `POST /api/admin/jobs/{name}/trigger?force=X&deep=X&repair=X` —
+ * dispatch a job on-demand.
+ *
+ * - `force` bypasses per-tenant idempotency checks (e.g. `trash_cleanup`
+ * skipping when nothing is due).
+ * - `deep` opts into slow variants (currently only `storage_consistency`,
+ * propagated by `consistency_batch` to every child).
+ * - `repair` opts into corrective action on the refcount consistency
+ * tenants (`blobs_consistency`, `manifests_consistency`, and
+ * `consistency_batch` which fans out to both). Content-safe: only the
+ * stored counter changes to match the auditor's computed value. Race-
+ * safe: the corrective UPDATE recomputes the auditor formula in the
+ * same statement, so a concurrent write can't leave a stale value.
+ * Default `false` preserves discovery-only behaviour — surface a
+ * confirm-first flow when calling with `repair: true`.
*
* Throws on 4xx / 5xx with the backend's error message when present.
* A 404 means the job name isn't registered — surface that specifically
@@ -72,11 +82,12 @@ export function listJobs(): Promise {
*/
export async function triggerJob(
name: string,
- opts: { force?: boolean; deep?: boolean; storage?: string } = {}
+ opts: { force?: boolean; deep?: boolean; storage?: string; repair?: boolean } = {}
): Promise {
const params = new URLSearchParams();
if (opts.force) params.set('force', 'true');
if (opts.deep) params.set('deep', 'true');
+ if (opts.repair) params.set('repair', 'true');
// `storage` scopes tenants that respect JobRunArgs.storage —
// currently blobs_consistency / backend_consistency (probes the
// named entry instead of the live backend). See
diff --git a/frontend/src/lib/components/AdminJobsPanel.svelte b/frontend/src/lib/components/AdminJobsPanel.svelte
index 7a70be95..d247ba58 100644
--- a/frontend/src/lib/components/AdminJobsPanel.svelte
+++ b/frontend/src/lib/components/AdminJobsPanel.svelte
@@ -21,6 +21,7 @@
import { SvelteSet } from 'svelte/reactivity';
import Icon from '$lib/icons/Icon.svelte';
import Modal from '$lib/components/Modal.svelte';
+ import { confirmDialog } from '$lib/stores/dialogs.svelte';
import { t } from '$lib/i18n/index.svelte';
import { errorMessage } from '$lib/utils/errors';
import { ui } from '$lib/stores/ui.svelte';
@@ -66,6 +67,43 @@
else busyKeys.delete(key);
}
+ // Per-job "Run" split-button menu state. Keyed by job name so
+ // two rows can open their menus independently (though the
+ // outside-click handler below closes all on any click outside
+ // any menu — matching the /files upload dropdown pattern). Only
+ // rows with `supportsDeep` OR `supportsRepair` render a chevron;
+ // the plain-Run rows (drives/folders/files/backend/… consistency,
+ // trash_cleanup, dedup_gc, …) show a bare "Run" button with no
+ // menu, keeping the common case one-click.
+ let runMenuOpen = $state>({});
+ function toggleRunMenu(name: string) {
+ runMenuOpen = { ...runMenuOpen, [name]: !runMenuOpen[name] };
+ }
+ function closeAllRunMenus() {
+ runMenuOpen = {};
+ }
+ // Global outside-click + Escape dismiss. Only registered while at
+ // least one menu is open — a background admin tab doesn't hold
+ // listeners.
+ $effect(() => {
+ const anyOpen = Object.values(runMenuOpen).some((v) => v);
+ if (!anyOpen) return;
+ const onDown = (e: MouseEvent) => {
+ if (!(e.target as HTMLElement).closest('.jobs-panel__split')) {
+ closeAllRunMenus();
+ }
+ };
+ const onKey = (e: KeyboardEvent) => {
+ if (e.key === 'Escape') closeAllRunMenus();
+ };
+ window.addEventListener('pointerdown', onDown);
+ window.addEventListener('keydown', onKey);
+ return () => {
+ window.removeEventListener('pointerdown', onDown);
+ window.removeEventListener('keydown', onKey);
+ };
+ });
+
// Purge-modal state. Null = closed; otherwise carries the
// draft retention days the operator's picking. Kept separate
// from the top-bar action state so mouse-away doesn't lose
@@ -234,8 +272,10 @@
// ─── Actions ───────────────────────────────────────────────────────
- async function onTrigger(name: string, opts: { deep?: boolean } = {}) {
- const key = `trigger:${name}${opts.deep ? ':deep' : ''}`;
+ async function onTrigger(name: string, opts: { deep?: boolean; repair?: boolean } = {}) {
+ // Key suffix has to keep every dispatched variant distinct so the
+ // button-disabled state of one doesn't lock out another mid-flight.
+ const key = `trigger:${name}${opts.deep ? ':deep' : ''}${opts.repair ? ':repair' : ''}`;
markBusy(key, true);
try {
// Fire the trigger + a follow-up loadJobs after a short delay
@@ -265,10 +305,49 @@
if (!res.outcome) {
// dispatched (detached) — no outcome to render
} else if (res.outcome.outcome === 'ok') {
- ui.notify(
- t('admin.jobs.triggered_ok', { name }, '{{name}} triggered successfully'),
- 'success'
- );
+ // Repair runs surface a rollup so the operator sees
+ // whether corrective UPDATEs actually fired. `extra`
+ // carries `repaired_count` on the two refcount tenants
+ // directly, and nested under `per_check[*].extra` when
+ // dispatched via `consistency_batch`. Sum across the
+ // per_check dict if present, else read the top-level.
+ let repairedTotal = 0;
+ let sawRepair = false;
+ const extra = (res.outcome.extra ?? {}) as {
+ repair_requested?: boolean;
+ repaired_count?: number;
+ per_check?: Record<
+ string,
+ { extra?: { repair_requested?: boolean; repaired_count?: number } }
+ >;
+ };
+ if (extra.repair_requested) {
+ sawRepair = true;
+ repairedTotal += extra.repaired_count ?? 0;
+ }
+ if (extra.per_check) {
+ for (const child of Object.values(extra.per_check)) {
+ if (child?.extra?.repair_requested) {
+ sawRepair = true;
+ repairedTotal += child.extra.repaired_count ?? 0;
+ }
+ }
+ }
+ if (sawRepair) {
+ ui.notify(
+ t(
+ 'admin.jobs.triggered_ok_repair',
+ { name, n: repairedTotal },
+ '{{name}}: {{n}} counter(s) repaired'
+ ),
+ 'success'
+ );
+ } else {
+ ui.notify(
+ t('admin.jobs.triggered_ok', { name }, '{{name}} triggered successfully'),
+ 'success'
+ );
+ }
} else {
ui.notify(
t(
@@ -557,6 +636,31 @@
return name === 'consistency_batch' || name === 'blobs_consistency';
}
+ // Jobs whose handler consults `args.repair` and applies a
+ // corrective UPDATE against the finding it just emitted. Only the
+ // two ref_count tenants today; `consistency_batch` also accepts
+ // the flag (fans out to both) and is surfaced separately as the
+ // top-bar "Repair ref_counts" button. Keep this list narrow —
+ // adding a job here without a matching backend handler produces a
+ // silently no-op button that confuses operators.
+ function supportsRepair(name: string): boolean {
+ return name === 'blobs_consistency' || name === 'manifests_consistency';
+ }
+
+ async function onTriggerWithRepairConfirm(name: string) {
+ const ok = await confirmDialog({
+ title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'),
+ message: t(
+ 'admin.jobs.run_repair_confirm_body_scoped',
+ { name },
+ 'Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.'
+ ),
+ confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
+ danger: true
+ });
+ if (ok) await onTrigger(name, { repair: true });
+ }
+
function isRunning(job: JobSummary): boolean {
return job.running;
}
@@ -614,6 +718,39 @@
{
+ const ok = await confirmDialog({
+ title: t('admin.jobs.run_repair_confirm_title', 'Repair drifted ref_counts?'),
+ message: t(
+ 'admin.jobs.run_repair_confirm_body',
+ 'Runs the audit against every blob + manifest and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only the counter changes; blob content and file rows are untouched. Safe to run at any time; a discovery-only run happens first so you can see the drift before this repair overwrites it.'
+ ),
+ confirmText: t('admin.jobs.run_repair_confirm', 'Repair'),
+ danger: true
+ });
+ if (ok) await onTrigger('consistency_batch', { repair: true });
+ }}
+ >
+
{/if}
+ {@const hasRunVariants = supportsDeep(job.name) || supportsRepair(job.name)}
+
onTrigger(job.name, { deep: true })}
+ class:jobs-panel__split-main={hasRunVariants}
+ disabled={busyKeys.has(`trigger:${job.name}`)}
+ onclick={() => {
+ closeAllRunMenus();
+ void onTrigger(job.name);
+ }}
>
- {t('admin.jobs.run_deep', 'Run deep')}
+ {t('admin.jobs.run', 'Run')}
- {/if}
+ {#if hasRunVariants}
+ toggleRunMenu(job.name)}
+ >
+
+ {#if runMenuOpen[job.name]}
+
+ {/if}
+ {/if}
+
{/if}
{#if isRunning(job) && canExpand}
{#if isRecoverable(job)}
@@ -1304,6 +1499,90 @@
color: var(--color-danger-text-alt);
}
+ /* Warn variant — used for actions that mutate data but are content-
+ safe / reversible-in-outcome (e.g. Repair ref_counts). Signals
+ "read the tooltip and the confirm before clicking" without the
+ danger red reserved for destructive delete-style buttons. */
+ .jobs-panel__btn--warn {
+ border-color: var(--color-warning-border);
+ color: var(--color-warning-text);
+ }
+
+ /* Split-button — inline flex holding a primary "Run" (fires default
+ action) and a chevron (opens the variants menu). `position:
+ relative` anchors the menu below the toggle. Only rendered on
+ rows whose job supports at least one variant; plain-Run rows
+ sidestep this whole structure. */
+ .jobs-panel__split {
+ display: inline-flex;
+ position: relative;
+ }
+
+ /* Attached-button trick: main loses its right border-radius, toggle
+ loses its left. Toggle also loses its left border so the two
+ don't render a double-thick divider. */
+ .jobs-panel__split-main {
+ border-top-right-radius: 0;
+ border-bottom-right-radius: 0;
+ }
+
+ .jobs-panel__split-toggle {
+ border-top-left-radius: 0;
+ border-bottom-left-radius: 0;
+ border-left: none;
+ padding-left: 0.35rem;
+ padding-right: 0.35rem;
+ }
+
+ /* The variants menu — dropdown below the toggle, right-aligned so
+ it doesn't overflow the Actions column edge into the next row's
+ badge cell. Shadow + surface bg mirror the /files upload
+ dropdown (`upload-dropdown-menu`); using local CSS here rather
+ than the ported class so the jobs-panel keeps its scoped styling. */
+ .jobs-panel__run-menu {
+ position: absolute;
+ top: calc(100% + 2px);
+ right: 0;
+ z-index: 30;
+ min-width: 10rem;
+ background: var(--color-bg-surface);
+ border: 1px solid var(--color-border);
+ border-radius: var(--radius-md, 6px);
+ box-shadow: var(--shadow-md);
+ padding: 0.25rem 0;
+ }
+
+ .jobs-panel__run-menu-item {
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ width: 100%;
+ padding: 0.4rem 0.75rem;
+ background: transparent;
+ border: none;
+ text-align: left;
+ font: inherit;
+ color: var(--color-text);
+ cursor: pointer;
+ white-space: nowrap;
+ }
+
+ .jobs-panel__run-menu-item:hover:not(:disabled) {
+ background: var(--color-bg-hover);
+ }
+
+ .jobs-panel__run-menu-item:disabled {
+ opacity: 0.5;
+ cursor: not-allowed;
+ }
+
+ /* Warn colour on the menu item mirrors the button variant so the
+ Repair option carries the same "attention-worthy but not
+ destructive" visual weight as its top-bar counterpart. */
+ .jobs-panel__run-menu-item--warn {
+ color: var(--color-warning-text);
+ }
+
.jobs-panel__pill {
display: inline-block;
padding: 0.1rem 0.5rem;
diff --git a/frontend/static/locales/en.json b/frontend/static/locales/en.json
index fc20b4d3..09a10822 100644
--- a/frontend/static/locales/en.json
+++ b/frontend/static/locales/en.json
@@ -1275,6 +1275,14 @@
"run_all_consistency": "Run all consistency checks",
"run_deep": "Run deep",
"run_deep_hint": "Also runs slow variants (blob re-hash, bitrot detection).",
+ "run_repair": "Repair ref_counts",
+ "run_repair_hint": "Corrects any drifted ref_counts (blobs + manifests) found by the audit. Content-safe — only counters change, not data.",
+ "run_repair_confirm_title": "Repair drifted ref_counts?",
+ "run_repair_confirm_body": "Runs the audit against every blob + manifest and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only the counter changes; blob content and file rows are untouched. Safe to run at any time; a discovery-only run happens first so you can see the drift before this repair overwrites it.",
+ "run_repair_confirm_body_scoped": "Runs {{name}} and applies a corrective UPDATE to any counter that disagrees with its live reference count. Content-safe: only counters change; blob content and file rows are untouched.",
+ "run_variants_menu": "Run variants menu",
+ "run_repair_confirm": "Repair",
+ "triggered_ok_repair": "{{name}}: {{n}} counter(s) repaired",
"col_name": "Name",
"col_cadence": "Cadence",
"col_last_run": "Last run",
diff --git a/frontend/static/locales/fr.json b/frontend/static/locales/fr.json
index 7b51a642..15b38fbe 100644
--- a/frontend/static/locales/fr.json
+++ b/frontend/static/locales/fr.json
@@ -1197,6 +1197,14 @@
"run_all_consistency": "Exécuter tous les contrôles de cohérence",
"run_deep": "Analyse approfondie",
"run_deep_hint": "Exécute également des variantes lentes (re-hachage de blob, détection bitrot).",
+ "run_repair": "Réparer les compteurs",
+ "run_repair_hint": "Corrige les compteurs de références (blobs + manifestes) désynchronisés détectés par l'audit. Sûr pour les données — seuls les compteurs changent, pas le contenu.",
+ "run_repair_confirm_title": "Réparer les compteurs de références ?",
+ "run_repair_confirm_body": "Lance l'audit sur chaque blob et manifeste, puis applique un UPDATE correctif à chaque compteur qui ne correspond pas au nombre réel de références. Sans risque pour les données : seuls les compteurs changent ; le contenu des blobs et les enregistrements de fichiers ne sont pas touchés. Vous pouvez exécuter cela à tout moment ; un passage en lecture seule s'exécute d'abord pour visualiser l'écart avant que la réparation ne l'écrase.",
+ "run_repair_confirm_body_scoped": "Lance {{name}} et applique un UPDATE correctif à chaque compteur qui ne correspond pas au nombre réel de références. Sans risque pour les données : seuls les compteurs changent ; le contenu et les enregistrements de fichiers ne sont pas touchés.",
+ "run_variants_menu": "Menu des variantes d'exécution",
+ "run_repair_confirm": "Réparer",
+ "triggered_ok_repair": "{{name}} : {{n}} compteur(s) réparé(s)",
"col_name": "Nom",
"col_cadence": "Cadence",
"col_last_run": "Dernière exécution",
diff --git a/migrations/20261016000000_copy_folder_tree_manifest_refcount.sql b/migrations/20261016000000_copy_folder_tree_manifest_refcount.sql
new file mode 100644
index 00000000..6c3010ba
--- /dev/null
+++ b/migrations/20261016000000_copy_folder_tree_manifest_refcount.sql
@@ -0,0 +1,211 @@
+-- Fix: `storage.copy_folder_tree` never incremented `chunk_manifests.ref_count`.
+--
+-- The function bumped only `storage.blobs`:
+--
+-- UPDATE storage.blobs b SET ref_count = ref_count + hc.cnt
+-- FROM (...) hc WHERE b.hash = hc.blob_hash;
+--
+-- but a CDC file's `blob_hash` names a MANIFEST, not a chunk. For any
+-- multi-chunk file that predicate matches nothing, so a folder copy took
+-- NO reference. Delete the original afterwards and `remove_reference`
+-- walks the manifest to 0, `dedup_gc` reaps the manifest and every chunk
+-- behind it — and the copy is unreadable. Silent data loss on an ordinary
+-- UI operation.
+--
+-- Single-chunk files escaped by accident: their whole-file hash equals
+-- their lone chunk's hash, so the UPDATE did match — bumping the wrong
+-- counter, which shows up as a manifest under-count plus a blob
+-- over-count rather than as loss.
+--
+-- Reproduced on a 5 MiB / 18-chunk file copied through the UI:
+-- `chunk_manifests.ref_count` stayed at 1 while two `storage.files` rows
+-- referenced it; `manifests_consistency` reported
+-- `manifest_refcount_mismatch` with `delta: 1, reap_risk: true`.
+--
+-- This migration only rewrites the reference-counting block; everything
+-- else is `20260902000001_copy_folder_tree_drop_user_id.sql` verbatim.
+--
+-- NOTE: existing drift is NOT repaired here. Run `manifests_consistency`
+-- to find it — a data fix belongs with the recovery framework, not in a
+-- schema migration that cannot know which counter is authoritative.
+
+CREATE OR REPLACE FUNCTION storage.copy_folder_tree(
+ p_source_id UUID,
+ p_target_parent_id UUID, -- NULL = copy to root (keeps source drive)
+ p_dest_name TEXT DEFAULT NULL -- NULL = keep source folder name
+) RETURNS TABLE(new_root_id TEXT, folders_copied BIGINT, files_copied BIGINT) AS $$
+DECLARE
+ v_root_lpath ltree;
+ v_root_depth INT;
+ v_max_depth INT;
+ v_level INT;
+ v_folders BIGINT := 0;
+ v_files BIGINT := 0;
+ v_inserted BIGINT;
+ v_new_root UUID;
+ v_dest_drive_id UUID;
+BEGIN
+ -- Validate source exists.
+ SELECT fo.lpath, nlevel(fo.lpath)
+ INTO v_root_lpath, v_root_depth
+ FROM storage.folders fo
+ WHERE fo.id = p_source_id AND NOT fo.is_trashed;
+
+ IF v_root_lpath IS NULL THEN
+ RAISE EXCEPTION 'Source folder not found: %', p_source_id
+ USING ERRCODE = 'P0002'; -- no_data_found
+ END IF;
+
+ -- Resolve destination drive_id once up front (cross-drive copy path).
+ IF p_target_parent_id IS NULL THEN
+ SELECT fo.drive_id INTO v_dest_drive_id
+ FROM storage.folders fo
+ WHERE fo.id = p_source_id;
+ ELSE
+ SELECT fo.drive_id INTO v_dest_drive_id
+ FROM storage.folders fo
+ WHERE fo.id = p_target_parent_id AND NOT fo.is_trashed;
+ IF v_dest_drive_id IS NULL THEN
+ RAISE EXCEPTION 'Target parent folder not found: %', p_target_parent_id
+ USING ERRCODE = 'P0002';
+ END IF;
+ END IF;
+
+ -- Temp mapping: every folder in the subtree → new UUID.
+ CREATE TEMP TABLE IF NOT EXISTS _copy_map(
+ old_id UUID PRIMARY KEY,
+ new_id UUID NOT NULL DEFAULT gen_random_uuid()
+ ) ON COMMIT DROP;
+ TRUNCATE _copy_map;
+
+ INSERT INTO _copy_map(old_id)
+ SELECT fo.id
+ FROM storage.folders fo
+ WHERE NOT fo.is_trashed
+ AND fo.lpath <@ v_root_lpath;
+
+ SELECT cm.new_id INTO v_new_root
+ FROM _copy_map cm WHERE cm.old_id = p_source_id;
+
+ SELECT MAX(nlevel(fo.lpath))
+ INTO v_max_depth
+ FROM storage.folders fo
+ JOIN _copy_map cm ON fo.id = cm.old_id;
+
+ -- ── Insert folders level by level ──
+ -- Post-D7: `user_id` intentionally omitted from the column list so
+ -- copied rows leave the (now-nullable) column NULL. Provenance is
+ -- carried by `created_by` / `updated_by` (§14 columns) — preserved
+ -- from source so authorship survives the copy.
+ FOR v_level IN v_root_depth .. v_max_depth LOOP
+ INSERT INTO storage.folders(
+ id, name, parent_id,
+ drive_id, created_by, updated_by
+ )
+ SELECT cm.new_id,
+ CASE WHEN fo.id = p_source_id AND p_dest_name IS NOT NULL
+ THEN p_dest_name ELSE fo.name END,
+ CASE WHEN fo.id = p_source_id THEN p_target_parent_id
+ ELSE pm.new_id END,
+ v_dest_drive_id,
+ fo.created_by,
+ fo.updated_by
+ FROM storage.folders fo
+ JOIN _copy_map cm ON fo.id = cm.old_id
+ LEFT JOIN _copy_map pm ON fo.parent_id = pm.old_id
+ WHERE NOT fo.is_trashed
+ AND nlevel(fo.lpath) = v_level;
+
+ GET DIAGNOSTICS v_inserted = ROW_COUNT;
+ v_folders := v_folders + v_inserted;
+ END LOOP;
+
+ -- Temp mapping for files src→dst (dst ids pre-allocated so we can
+ -- reference them in the dead-property duplication below).
+ CREATE TEMP TABLE IF NOT EXISTS _copy_file_map(
+ old_id UUID PRIMARY KEY,
+ new_id UUID NOT NULL DEFAULT gen_random_uuid()
+ ) ON COMMIT DROP;
+ TRUNCATE _copy_file_map;
+
+ INSERT INTO _copy_file_map(old_id)
+ SELECT f.id
+ FROM storage.files f
+ JOIN _copy_map cm ON f.folder_id = cm.old_id
+ WHERE NOT f.is_trashed;
+
+ -- ── Batch copy all files (zero-copy: same blob_hash) ──
+ -- Post-D7: `user_id` omitted. Provenance via `created_by`/`updated_by`.
+ INSERT INTO storage.files(
+ id, name, folder_id, blob_hash, size, mime_type,
+ media_sort_date, drive_id, created_by, updated_by
+ )
+ SELECT fm.new_id, f.name, cm.new_id, f.blob_hash, f.size,
+ f.mime_type, f.media_sort_date, v_dest_drive_id, f.created_by,
+ f.updated_by
+ FROM storage.files f
+ JOIN _copy_map cm ON f.folder_id = cm.old_id
+ JOIN _copy_file_map fm ON fm.old_id = f.id
+ WHERE NOT f.is_trashed;
+
+ GET DIAGNOSTICS v_files = ROW_COUNT;
+
+ -- Batch increment reference counts — MANIFEST FIRST, blobs only as
+ -- fallback. This mirrors `DedupService::add_reference`, and the order
+ -- is the whole point:
+ --
+ -- A CDC file's `blob_hash` names a MANIFEST (`chunk_manifests.file_hash`),
+ -- not a chunk. The previous version of this block updated only
+ -- `storage.blobs`, so for a multi-chunk file the predicate
+ -- `b.hash = hc.blob_hash` matched ZERO rows and the copy took no
+ -- reference at all. Deleting the original then walked the manifest's
+ -- ref_count to 0, dedup_gc reaped the manifest and every chunk behind
+ -- it, and the copy became unreadable. Reproduced via the UI folder
+ -- copy on a 5 MiB (18-chunk) file: ref_count stayed 1 with two files
+ -- referencing it.
+ --
+ -- The `NOT EXISTS (bumped)` guard on the blobs branch is load-bearing.
+ -- For a SINGLE-chunk file the whole-file hash equals its lone chunk's
+ -- hash, so without it the copy would be counted at both levels and
+ -- turn an under-count into an over-count.
+ IF v_files > 0 THEN
+ WITH hc AS (
+ SELECT f.blob_hash, COUNT(*)::int AS cnt
+ FROM storage.files f
+ JOIN _copy_map cm ON f.folder_id = cm.new_id
+ WHERE NOT f.is_trashed
+ GROUP BY f.blob_hash
+ ),
+ bumped AS (
+ UPDATE storage.chunk_manifests m
+ SET ref_count = m.ref_count + hc.cnt
+ FROM hc
+ WHERE m.file_hash = hc.blob_hash
+ RETURNING m.file_hash
+ )
+ UPDATE storage.blobs b
+ SET ref_count = b.ref_count + hc.cnt,
+ -- Matches add_reference: a blob resurrected inside its GC
+ -- grace window must lose its orphan stamp.
+ orphaned_at = NULL
+ FROM hc
+ WHERE b.hash = hc.blob_hash
+ AND NOT EXISTS (SELECT 1 FROM bumped WHERE file_hash = hc.blob_hash);
+ END IF;
+
+ -- Duplicate dead properties per RFC 4918 §8.8 — id-keyed store.
+ INSERT INTO storage.webdav_dead_properties
+ (folder_id, namespace, local_name, value)
+ SELECT cm.new_id, dp.namespace, dp.local_name, dp.value
+ FROM storage.webdav_dead_properties dp
+ JOIN _copy_map cm ON dp.folder_id = cm.old_id;
+
+ INSERT INTO storage.webdav_dead_properties
+ (file_id, namespace, local_name, value)
+ SELECT fm.new_id, dp.namespace, dp.local_name, dp.value
+ FROM storage.webdav_dead_properties dp
+ JOIN _copy_file_map fm ON dp.file_id = fm.old_id;
+
+ RETURN QUERY SELECT v_new_root::text, v_folders, v_files;
+END;
+$$ LANGUAGE plpgsql;
diff --git a/migrations/20261017000000_file_delete_trigger_manifest_aware.sql b/migrations/20261017000000_file_delete_trigger_manifest_aware.sql
new file mode 100644
index 00000000..c166143f
--- /dev/null
+++ b/migrations/20261017000000_file_delete_trigger_manifest_aware.sql
@@ -0,0 +1,108 @@
+-- Fix: `trg_files_decrement_blob_ref` decremented the wrong counter for
+-- CDC files.
+--
+-- The original trigger (2026-03-07 initial schema) unconditionally ran:
+--
+-- UPDATE storage.blobs
+-- SET ref_count = GREATEST(ref_count - 1, 0)
+-- WHERE hash = OLD.blob_hash;
+--
+-- That's correct for a legacy whole-file blob, where `OLD.blob_hash`
+-- names a `storage.blobs` row directly. For a CDC file, `OLD.blob_hash`
+-- names a `storage.chunk_manifests.file_hash` — the blob table row (if
+-- one exists at all) holds a DIFFERENT counter, incremented by the
+-- MANIFEST's presence in its own `chunk_hashes[]`, not by the file.
+--
+-- Consequences before this fix:
+-- 1. `storage.chunk_manifests.ref_count` never decremented on file
+-- DELETE → over-count grows unboundedly across delete/purge
+-- cycles.
+-- 2. `storage.blobs.ref_count` decremented for hashes it shouldn't
+-- (CDC whole-file hashes) → the counter drops toward 0 while the
+-- manifest still legitimately references the chunk. GC then reaps
+-- a live blob → downloadable-then-404 data loss.
+--
+-- Both bugs surfaced by `tests/api/refcount_cascade.hurl` on the
+-- 135-byte fixture (single-chunk CDC file, worst case for confusion
+-- because the whole-file hash equals its lone chunk's hash). The
+-- 2026-08-22 sandbox drift (`storage.blobs.ref_count = 0`,
+-- `actual_auditor = 1`) is the same bug at rest.
+--
+-- Sibling fix: `20261016000000_copy_folder_tree_manifest_refcount.sql`
+-- fixed the mirror-image INCREMENT bug in `storage.copy_folder_tree`.
+-- This migration closes the decrement half.
+--
+-- Cross-references:
+-- - `DedupService::add_reference` (dedup_service.rs:1703) — app-layer
+-- twin for the increment direction: manifest first, blob fallback.
+-- - `manifests_consistency` tenant (2026-08-23) — surfaces any
+-- residual drift after this fix lands.
+--
+-- ── DESIGN NOTE — decrement only, no manifest reap here ──
+--
+-- The trigger DELIBERATELY does not delete manifests or walk chunks on
+-- a last-ref decrement. Both actions used to live inside
+-- `DedupService::cleanup_if_orphaned` and its callee
+-- `remove_manifest_reference`, and both fire `fire_blob_hooks` —
+-- the Rust callback that reaps disk artefacts keyed by the whole-file
+-- content hash (thumbnails, face embeddings, audio tags, media
+-- metadata). SQL triggers can't invoke Rust callbacks, so if this
+-- trigger reaped the manifest itself, dedup_gc Phase 1
+-- (`dedup_service.rs:2660-2772`) — the ONLY code path that knows to
+-- fire `fire_blob_hooks` for a reaped manifest's `file_hash` — would
+-- find nothing to do on its next sweep, and every derived artefact
+-- would leak on disk. `storage_cleanup_check.sh`'s "N thumbnail
+-- file(s) remain on disk" gate catches this class immediately.
+--
+-- Contract: trigger decrements the correct counter atomically inside
+-- the DELETE txn. GC (`dedup_gc`) is responsible for:
+-- • finding manifests whose ref_count hit 0 (or that no reference
+-- source references, covering bulk-delete paths),
+-- • deleting them,
+-- • decrementing each chunk in `chunk_hashes[]`,
+-- • firing `fire_blob_hooks(file_hash)` so Rust callbacks reap
+-- derived disk artefacts,
+-- • the corresponding legacy-blob path for ref_count = 0 blobs.
+--
+-- NOTE: pre-existing drift is NOT repaired here. Run `manifests_
+-- consistency` + `blobs_consistency` after deploy; feed the findings
+-- into the recovery framework.
+
+CREATE OR REPLACE FUNCTION storage.decrement_blob_ref()
+RETURNS trigger AS $$
+BEGIN
+ -- Manifest-first, mirroring the increment side. We touch ONE
+ -- counter and return — the manifest reap + chunk walk + hook
+ -- firing lives in `dedup_gc` where Rust callbacks can run.
+ IF EXISTS (
+ SELECT 1 FROM storage.chunk_manifests
+ WHERE file_hash = OLD.blob_hash
+ ) THEN
+ UPDATE storage.chunk_manifests
+ SET ref_count = GREATEST(ref_count - 1, 0)
+ WHERE file_hash = OLD.blob_hash;
+ ELSE
+ -- Legacy whole-file blob path: no manifest, blob is referenced
+ -- directly by this file row. Preserves the original behaviour
+ -- verbatim for the pre-CDC path.
+ UPDATE storage.blobs
+ SET ref_count = GREATEST(ref_count - 1, 0),
+ orphaned_at = CASE
+ WHEN GREATEST(ref_count - 1, 0) = 0
+ THEN now()
+ ELSE orphaned_at
+ END
+ WHERE hash = OLD.blob_hash;
+ END IF;
+
+ RETURN OLD;
+END;
+$$ LANGUAGE plpgsql;
+
+COMMENT ON FUNCTION storage.decrement_blob_ref() IS
+ 'Decrement the correct ref_count when a file is deleted. '
+ 'Manifest-aware (2026-10-17): dispatches to chunk_manifests.ref_count '
+ 'when the file''s blob_hash names a manifest, else to '
+ 'storage.blobs.ref_count for legacy whole-file blobs. Decrement-only: '
+ 'physical cleanup + Rust lifecycle hooks fire from dedup_gc, which '
+ 'can invoke callbacks a SQL trigger cannot.';
diff --git a/migrations/20261017000002_repair_existing_refcount_drift.sql b/migrations/20261017000002_repair_existing_refcount_drift.sql
new file mode 100644
index 00000000..11043211
--- /dev/null
+++ b/migrations/20261017000002_repair_existing_refcount_drift.sql
@@ -0,0 +1,94 @@
+-- One-time repair of ref_count drift accumulated under the pre-fix
+-- copy/delete code paths.
+--
+-- Why this is atomic with the upgrade rather than a manual admin action
+-- ─────────────────────────────────────────────────────────────────────
+-- The two prior migrations on this branch:
+-- * `20261016000000_copy_folder_tree_manifest_refcount.sql`
+-- (fix the INCREMENT path — copy was bumping the wrong counter for
+-- CDC files)
+-- * `20261017000000_file_delete_trigger_manifest_aware.sql`
+-- (fix the DECREMENT path — trigger was decrementing the wrong
+-- counter for CDC files; folder-cascade + trash-empty paths
+-- inherited that drift silently)
+-- both close the bugs going forward, but production DBs upgrading
+-- through this branch may carry accumulated drift from every prior
+-- copy → delete cycle a CDC file went through. Under-count is the
+-- dangerous direction: the next `dedup_gc` pass would reap a live
+-- blob → user-facing 404 → silent data loss.
+--
+-- Waiting for an operator to open the admin panel and click "Repair
+-- ref_counts" is the wrong default for a data-loss-preventing fix.
+-- Ed's rule (`[[feedback_no_silent_auto_repair]]`): consistency
+-- tenants must default to discovery-only so future bugs surface — but
+-- fixing KNOWN pre-existing drift on the upgrade itself is the
+-- bounded exception, because at that specific moment the source of
+-- drift is known + closed, and there is no upstream mystery to
+-- preserve.
+--
+-- Content-safety guarantees:
+-- * Only counter columns change (`storage.chunk_manifests.ref_count`,
+-- `storage.blobs.ref_count`). No file rows, no blob rows, no
+-- manifest rows, no chunk arrays, no backend files.
+-- * The corrective UPDATE sets `stored = actual` where `actual` is
+-- computed from the SAME auditor formulas that
+-- `manifests_consistency` / `blobs_consistency` use, so this
+-- migration and those tenants agree by construction.
+-- * Race-safe against concurrent writes (migrations run
+-- single-connection at startup before the server serves any
+-- traffic; nobody else is writing).
+-- * Idempotent — fresh installs and already-clean DBs no-op (both
+-- `stored` and `actual` are equal, the `WHERE <>` filters
+-- everything out).
+--
+-- The panel button + `?repair=true` on the trigger endpoints stay for
+-- FUTURE drift (regression detector; not for repeat use on this
+-- accumulated set).
+
+DO $$
+DECLARE
+ v_m_fixed int;
+ v_b_fixed int;
+BEGIN
+ -- Manifest counter: `actual` = # files whose blob_hash names this
+ -- manifest's file_hash. Same formula as
+ -- `manifests_consistency_service::manifest_page_sql` (via the
+ -- BlobReferenceRegistry at RefLevel::Manifest) — inline here
+ -- because migrations can't call Rust.
+ UPDATE storage.chunk_manifests m
+ SET ref_count = (SELECT COUNT(*) FROM storage.files
+ WHERE blob_hash = m.file_hash)
+ WHERE m.ref_count <> (SELECT COUNT(*) FROM storage.files
+ WHERE blob_hash = m.file_hash);
+ GET DIAGNOSTICS v_m_fixed = ROW_COUNT;
+
+ -- Blob counter: two-term formula mirroring
+ -- `blobs_consistency_service.rs:395-408`:
+ -- (files pointing at this blob AND having NO manifest for their
+ -- blob_hash — legacy whole-file path)
+ -- + (manifests including this hash as a chunk in chunk_hashes[])
+ UPDATE storage.blobs b
+ SET ref_count = (
+ (SELECT COUNT(*) FROM storage.files f
+ WHERE f.blob_hash = b.hash
+ AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests m
+ WHERE m.file_hash = f.blob_hash))
+ + (SELECT COUNT(*) FROM storage.chunk_manifests m
+ WHERE b.hash = ANY(m.chunk_hashes))
+ )
+ WHERE b.ref_count <> (
+ (SELECT COUNT(*) FROM storage.files f
+ WHERE f.blob_hash = b.hash
+ AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests m
+ WHERE m.file_hash = f.blob_hash))
+ + (SELECT COUNT(*) FROM storage.chunk_manifests m
+ WHERE b.hash = ANY(m.chunk_hashes))
+ );
+ GET DIAGNOSTICS v_b_fixed = ROW_COUNT;
+
+ -- Landed in the deploy log so an operator upgrading a huge instance
+ -- can see the migration did work — silent no-op on fresh installs.
+ RAISE NOTICE '[refcount_repair] fixed % manifest(s), % blob(s)',
+ v_m_fixed, v_b_fixed;
+END;
+$$;
diff --git a/src/application/ports/blob_reference_ports.rs b/src/application/ports/blob_reference_ports.rs
new file mode 100644
index 00000000..aa949ba3
--- /dev/null
+++ b/src/application/ports/blob_reference_ports.rs
@@ -0,0 +1,313 @@
+//! `BlobReferenceSource` — the extension point that teaches ref-counting
+//! and the consistency jobs about a table holding blob references.
+//!
+//! Before this port, "who references this hash" was hardcoded SQL in two
+//! places (`dedup_gc`'s reap predicate and `blobs_consistency`'s refcount
+//! recompute), both naming `storage.files` and `storage.chunk_manifests`
+//! directly. Any new blob-owning table therefore risked silent orphaning:
+//! `dedup_gc` sees `ref_count = 0`, or a manifest with no `storage.files`
+//! row behind it, and reaps live content.
+//!
+//! See `docs/plan/derived-blobs.md` for the design and the coverage matrix.
+//!
+//! # Two levels, and why a source may span both
+//!
+//! [`DedupService::add_reference`] bumps `chunk_manifests.ref_count` first
+//! and only falls back to `storage.blobs.ref_count`. So a reference lands
+//! on whichever counter its hash names, and the two must be recomputed
+//! separately — mixing them double-counts, systematically:
+//!
+//! * A **Blob** (`chunk_manifests.file_hash`) is "the content of a file".
+//! * A **Chunk** (`storage.blobs.hash`) is a physical byte payload.
+//! * For a single-chunk Blob the two hashes are **equal**, because both are
+//! BLAKE3 over the same bytes. That aliasing is why today's chunk-level
+//! recompute carries a `NOT EXISTS` clause, and why every fragment here
+//! must be level-correct rather than merely plausible.
+//!
+//! A source is not confined to one level: [`RefLevel::Chunk`] and
+//! [`RefLevel::Manifest`] fragments are requested independently, and
+//! `storage.files` legitimately contributes to both — a manifest-less
+//! legacy row references a chunk, a CDC row references a Blob.
+//!
+//! # Why SQL fragments rather than a per-hash count
+//!
+//! `blobs_consistency` recomputes refcounts with **one query per page**,
+//! the expected count inlined as correlated subqueries. Asking each source
+//! for a count per hash would turn that into `sources × rows` round-trips —
+//! a catastrophic regression on a table with millions of rows. So sources
+//! contribute a *fragment* that the registry sums into the existing page
+//! query, and [`BlobReferenceSource::count_references`] exists only for the
+//! on-demand path (`dedup_gc` checking a single reap candidate, where the
+//! candidate set is already filtered to `ref_count = 0`).
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+
+use crate::domain::errors::DomainError;
+
+/// Which counter a source's references land on.
+///
+/// Not a property of the source — see the module docs; the same source may
+/// contribute at both levels.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub enum RefLevel {
+ /// References a physical chunk. Feeds `storage.blobs.ref_count`.
+ Chunk,
+ /// References a Blob via its manifest. Feeds
+ /// `chunk_manifests.ref_count`.
+ Manifest,
+}
+
+impl RefLevel {
+ /// Both levels, for callers that sweep each in turn.
+ pub const ALL: [RefLevel; 2] = [RefLevel::Chunk, RefLevel::Manifest];
+
+ /// Stable name for logs and consistency-finding fields.
+ pub fn as_str(self) -> &'static str {
+ match self {
+ RefLevel::Chunk => "chunk",
+ RefLevel::Manifest => "manifest",
+ }
+ }
+}
+
+/// One table that holds references to blob hashes.
+///
+/// Implementors are registered on [`BlobReferenceRegistry`] during DI.
+/// Adding a blob-owning table **without** registering it is the failure
+/// this port exists to prevent.
+#[async_trait]
+pub trait BlobReferenceSource: Send + Sync {
+ /// Short stable identifier for logs and consistency-finding `source`
+ /// fields — `"files"`, `"chunks"`, `"content_derived"`, …
+ ///
+ /// Stable across releases: log aggregators key off it.
+ fn source_name(&self) -> &'static str;
+
+ /// A correlated-subquery fragment counting this source's references
+ /// **at `level`** to `outer_hash_expr`, or `None` when this source
+ /// holds no references at that level.
+ ///
+ /// `outer_hash_expr` is the SQL expression naming the hash of the row
+ /// being recomputed — `"b.hash"` when sweeping `storage.blobs`,
+ /// `"m.file_hash"` when sweeping `storage.chunk_manifests`. The
+ /// fragment must be a parenthesised scalar subquery so the registry can
+ /// join fragments with `+`.
+ ///
+ /// **Identifiers only.** `outer_hash_expr` is supplied by the sweep, never
+ /// by a request; no fragment may interpolate caller input.
+ fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option;
+
+ /// Existence form of [`Self::ref_count_sql`] — a boolean fragment, true
+ /// when this source holds at least one reference at `level`.
+ ///
+ /// Defaults to `() > 0`. Override when the source can express a
+ /// short-circuiting `EXISTS`, which the planner can stop at the first
+ /// matching row: `dedup_gc`'s reap predicate runs this per candidate
+ /// manifest, and a heavily-deduplicated blob has many referrers, so
+ /// counting all of them where existence would do is a real regression.
+ fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option {
+ self.ref_count_sql(level, outer_hash_expr)
+ .map(|fragment| format!("{fragment} > 0"))
+ }
+
+ /// Count of references this source holds on `blob_hash`, across both
+ /// levels.
+ ///
+ /// **On-demand path only** — `dedup_gc` checking a single reap
+ /// candidate. The consistency sweeps must use [`Self::ref_count_sql`];
+ /// calling this per row would turn one query per page into
+ /// `sources × rows` round-trips.
+ async fn count_references(&self, blob_hash: &str) -> Result;
+
+ /// Iterate the hashes this source references, paged by the
+ /// implementation's natural cursor (typically a primary key).
+ ///
+ /// Used by `backend_consistency` to walk the backend against the union
+ /// of all sources. Returns the page plus the cursor to resume from,
+ /// `None` when exhausted.
+ async fn list_referenced_blobs(
+ &self,
+ cursor: Option>,
+ limit: usize,
+ ) -> Result<(Vec, Option>), DomainError>;
+
+ /// Notification that `dedup_gc` reaped this blob.
+ ///
+ /// Sources maintaining a denormalised refcount can clean up here. Most
+ /// leave the default noop — the mapping row is normally deleted by the
+ /// owning service's `on_blob_deleted` hook instead.
+ fn on_blob_reaped(&self, _blob_hash: &str) {}
+}
+
+/// The set of registered [`BlobReferenceSource`]s.
+///
+/// Assembled once during DI and shared (`Arc`) by `dedup_gc` and the
+/// consistency jobs, so all three agree on what "referenced" means.
+#[derive(Default)]
+pub struct BlobReferenceRegistry {
+ sources: Vec>,
+}
+
+impl BlobReferenceRegistry {
+ pub fn new() -> Self {
+ Self::default()
+ }
+
+ /// Register a source. Order is irrelevant — fragments are summed and
+ /// counts added.
+ pub fn register(&mut self, source: Arc) {
+ self.sources.push(source);
+ }
+
+ pub fn sources(&self) -> &[Arc] {
+ &self.sources
+ }
+
+ /// The summed SQL expression counting every source's references at
+ /// `level` to `outer_hash_expr`.
+ ///
+ /// Returns `"0"` when no source contributes at this level, which keeps
+ /// the caller's query valid without a special case.
+ pub fn ref_count_expr(&self, level: RefLevel, outer_hash_expr: &str) -> String {
+ let fragments: Vec = self
+ .sources
+ .iter()
+ .filter_map(|s| s.ref_count_sql(level, outer_hash_expr))
+ .collect();
+
+ if fragments.is_empty() {
+ "0".to_string()
+ } else {
+ fragments.join("\n + ")
+ }
+ }
+
+ /// Predicate selecting rows that **no** registered source references at
+ /// `level` — i.e. reap candidates.
+ ///
+ /// Returns `None` when no source contributes at this level, and callers
+ /// **must** treat that as "refuse to act" rather than substituting a
+ /// default. The natural default would be the sum-equals-zero form, which
+ /// on an empty registry reduces to `0 = 0` — vacuously true for every
+ /// row, i.e. "delete everything". Returning `None` makes that
+ /// unrepresentable at the call site instead of merely discouraged.
+ pub fn no_reference_predicate(&self, level: RefLevel, outer_hash_expr: &str) -> Option {
+ let fragments: Vec = self
+ .sources
+ .iter()
+ .filter_map(|s| s.ref_exists_sql(level, outer_hash_expr))
+ .collect();
+
+ if fragments.is_empty() {
+ return None;
+ }
+ Some(format!("NOT ({})", fragments.join("\n OR ")))
+ }
+
+ /// Total references held on `hash` across every source.
+ ///
+ /// On-demand path only — see [`BlobReferenceSource::count_references`].
+ pub async fn total_references(&self, hash: &str) -> Result {
+ let mut total = 0u64;
+ for source in &self.sources {
+ total = total.saturating_add(source.count_references(hash).await?);
+ }
+ Ok(total)
+ }
+
+ /// Fan out a reap notification to every source.
+ pub fn notify_reaped(&self, blob_hash: &str) {
+ for source in &self.sources {
+ source.on_blob_reaped(blob_hash);
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ struct Stub {
+ name: &'static str,
+ chunk: Option<&'static str>,
+ manifest: Option<&'static str>,
+ count: u64,
+ }
+
+ #[async_trait]
+ impl BlobReferenceSource for Stub {
+ fn source_name(&self) -> &'static str {
+ self.name
+ }
+
+ fn ref_count_sql(&self, level: RefLevel, outer: &str) -> Option {
+ let tmpl = match level {
+ RefLevel::Chunk => self.chunk?,
+ RefLevel::Manifest => self.manifest?,
+ };
+ Some(tmpl.replace("{outer}", outer))
+ }
+
+ async fn count_references(&self, _blob_hash: &str) -> Result {
+ Ok(self.count)
+ }
+
+ async fn list_referenced_blobs(
+ &self,
+ _cursor: Option>,
+ _limit: usize,
+ ) -> Result<(Vec, Option>), DomainError> {
+ Ok((Vec::new(), None))
+ }
+ }
+
+ fn registry() -> BlobReferenceRegistry {
+ let mut r = BlobReferenceRegistry::new();
+ r.register(Arc::new(Stub {
+ name: "a",
+ chunk: Some("(SELECT 1 WHERE {outer} = 'x')"),
+ manifest: None,
+ count: 2,
+ }));
+ r.register(Arc::new(Stub {
+ name: "b",
+ chunk: Some("(SELECT 2 WHERE {outer} = 'y')"),
+ manifest: Some("(SELECT 3 WHERE {outer} = 'z')"),
+ count: 5,
+ }));
+ r
+ }
+
+ #[test]
+ fn chunk_level_sums_every_contributing_source() {
+ let expr = registry().ref_count_expr(RefLevel::Chunk, "b.hash");
+ assert!(expr.contains("b.hash = 'x'"), "{expr}");
+ assert!(expr.contains("b.hash = 'y'"), "{expr}");
+ assert!(expr.contains('+'), "fragments must be summed: {expr}");
+ }
+
+ /// A source returning `None` for a level must contribute nothing there —
+ /// this is what keeps manifest-only tables out of the chunk recompute,
+ /// where they would double-count against the single-chunk hash alias.
+ #[test]
+ fn manifest_level_skips_non_contributing_sources() {
+ let expr = registry().ref_count_expr(RefLevel::Manifest, "m.file_hash");
+ assert!(expr.contains("m.file_hash = 'z'"), "{expr}");
+ assert!(!expr.contains('+'), "only one source contributes: {expr}");
+ }
+
+ /// An empty level must still yield a valid scalar expression, so callers
+ /// need no special case before a registry is fully populated.
+ #[test]
+ fn empty_level_yields_zero_literal() {
+ let r = BlobReferenceRegistry::new();
+ assert_eq!(r.ref_count_expr(RefLevel::Chunk, "b.hash"), "0");
+ }
+
+ #[tokio::test]
+ async fn total_references_adds_across_sources() {
+ assert_eq!(registry().total_references("deadbeef").await.unwrap(), 7);
+ }
+}
diff --git a/src/application/ports/mod.rs b/src/application/ports/mod.rs
index 3ee7c4dc..53c8e8db 100644
--- a/src/application/ports/mod.rs
+++ b/src/application/ports/mod.rs
@@ -1,6 +1,7 @@
pub mod auth_ports;
pub mod authorization_ports;
pub mod blob_lifecycle;
+pub mod blob_reference_ports;
pub mod blob_storage_ports;
pub mod cache_ports;
pub mod calendar_ports;
diff --git a/src/common/di.rs b/src/common/di.rs
index 9f3fc5b7..0a47d418 100644
--- a/src/common/di.rs
+++ b/src/common/di.rs
@@ -440,6 +440,22 @@ impl AppServiceFactory {
// `blob_backend` into DedupService.
let blob_backend_for_consistency = blob_backend.clone();
+ // Every table holding blob references. Built ONCE and shared by the
+ // GC reap predicate and the consistency recompute so the two cannot
+ // disagree about what "referenced" means — a disagreement reaps live
+ // content. New blob-owning tables register here.
+ // See docs/plan/derived-blobs.md.
+ let blob_reference_registry = {
+ use crate::infrastructure::repositories::pg::blob_reference_sources::{
+ ChunksReferenceSource, FilesReferenceSource,
+ };
+ let mut registry =
+ crate::application::ports::blob_reference_ports::BlobReferenceRegistry::new();
+ registry.register(Arc::new(FilesReferenceSource::new(db_pool.clone())));
+ registry.register(Arc::new(ChunksReferenceSource::new(db_pool.clone())));
+ Arc::new(registry)
+ };
+
// Deduplication service — PRIMARY blob storage engine (PostgreSQL-backed index)
let dedup_service = Arc::new(
crate::infrastructure::services::dedup_service::DedupService::new(
@@ -447,7 +463,8 @@ impl AppServiceFactory {
db_pool.clone(),
maintenance_pool.clone(),
)
- .with_blob_lifecycle(blob_lifecycle),
+ .with_blob_lifecycle(blob_lifecycle)
+ .with_reference_registry(blob_reference_registry.clone()),
);
dedup_service.initialize().await?;
@@ -1460,6 +1477,22 @@ impl AppServiceFactory {
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
.await;
+ // Reconciles `chunk_manifests.ref_count` — the SECOND reference
+ // counter, and the one nothing verified before. add_reference bumps
+ // it first and only falls back to storage.blobs.ref_count, so every
+ // CDC file (and every derived artifact, once those land) counts here
+ // rather than at the chunk level. Uses the same registry dedup_gc
+ // reaps from, so the two cannot disagree.
+ // See docs/plan/derived-blobs.md.
+ let _ = Arc::new(
+ crate::infrastructure::services::manifests_consistency_service::ManifestsConsistencyCheck::new(
+ maintenance_pool.clone(),
+ core.dedup_service.reference_registry(),
+ ),
+ )
+ .register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
+ .await;
+
// Third recoverable-run tenant. Iterates `storage.files`
// and reports parent-folder-trashed cascade misses,
// `missing_blob` (data-loss indicator — file references
@@ -1493,6 +1526,9 @@ impl AppServiceFactory {
core.blob_backend.clone(),
core.config.storage_entries.clone(),
self.storage_path.clone(),
+ // Same registry instance GC reaps from — see
+ // DedupService::reference_registry.
+ core.dedup_service.reference_registry(),
),
)
.register_recoverable_job(&core.job_registry, &job_store_provider_dyn)
diff --git a/src/infrastructure/repositories/pg/blob_reference_sources.rs b/src/infrastructure/repositories/pg/blob_reference_sources.rs
new file mode 100644
index 00000000..4424671b
--- /dev/null
+++ b/src/infrastructure/repositories/pg/blob_reference_sources.rs
@@ -0,0 +1,353 @@
+//! The two implicit blob-reference sources, made explicit.
+//!
+//! Before this module, "who references this hash" lived as hardcoded SQL
+//! inside `blobs_consistency`'s refcount recompute and `dedup_gc`'s reap
+//! predicate. These two implementations reproduce that SQL **exactly** —
+//! the fragments below sum to today's `actual_ref_count` expression — so
+//! the registry can be wired in without changing any observed count.
+//!
+//! See `docs/plan/derived-blobs.md` and
+//! [`crate::application::ports::blob_reference_ports`].
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use sqlx::{PgPool, Row};
+use uuid::Uuid;
+
+use crate::application::ports::blob_reference_ports::{BlobReferenceSource, RefLevel};
+use crate::domain::errors::DomainError;
+
+/// Aliases used inside the emitted fragments.
+///
+/// Deliberately distinct from the aliases the sweeps use for their outer
+/// row (`b` for `storage.blobs`, `m` for `storage.chunk_manifests`): a
+/// fragment reusing `m` would shadow the outer alias in the manifest-level
+/// sweep and silently correlate against itself.
+const FILES_ALIAS: &str = "cnt_f";
+const MANIFEST_ALIAS: &str = "cnt_m";
+
+/// Fragment for [`FilesReferenceSource`], as a free function so the SQL
+/// shape can be tested without constructing a pool — it is a property of
+/// the module, not of an instance.
+fn files_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option {
+ let f = FILES_ALIAS;
+ match level {
+ // Legacy whole-file blobs only — CDC files are counted at the
+ // manifest level, and counting them here too would double up on the
+ // single-chunk hash alias.
+ RefLevel::Chunk => Some(format!(
+ "(SELECT COUNT(*) FROM storage.files {f}
+ WHERE {f}.blob_hash = {outer_hash_expr}
+ AND NOT EXISTS (
+ SELECT 1 FROM storage.chunk_manifests {MANIFEST_ALIAS}
+ WHERE {MANIFEST_ALIAS}.file_hash = {f}.blob_hash
+ ))"
+ )),
+ RefLevel::Manifest => Some(format!(
+ "(SELECT COUNT(*) FROM storage.files {f}
+ WHERE {f}.blob_hash = {outer_hash_expr})"
+ )),
+ }
+}
+
+/// Short-circuiting existence form of [`files_ref_sql`].
+///
+/// `dedup_gc` evaluates this per candidate manifest, so counting every
+/// referrer where existence would do is a real cost on a heavily-deduplicated
+/// blob. This is also the exact shape the reap predicate used before the
+/// registry existed, so wiring it in changes no plan.
+fn files_exists_sql(level: RefLevel, outer_hash_expr: &str) -> Option {
+ let f = FILES_ALIAS;
+ match level {
+ RefLevel::Chunk => Some(format!(
+ "EXISTS (SELECT 1 FROM storage.files {f} \
+ WHERE {f}.blob_hash = {outer_hash_expr} \
+ AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests {MANIFEST_ALIAS} \
+ WHERE {MANIFEST_ALIAS}.file_hash = {f}.blob_hash))"
+ )),
+ RefLevel::Manifest => Some(format!(
+ "EXISTS (SELECT 1 FROM storage.files {f} WHERE {f}.blob_hash = {outer_hash_expr})"
+ )),
+ }
+}
+
+/// Fragment for [`ChunksReferenceSource`]. See [`files_ref_sql`].
+fn chunks_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option {
+ match level {
+ RefLevel::Chunk => {
+ let m = MANIFEST_ALIAS;
+ Some(format!(
+ "(SELECT COUNT(*) FROM storage.chunk_manifests {m}
+ WHERE {outer_hash_expr} = ANY({m}.chunk_hashes))"
+ ))
+ }
+ // A manifest is never referenced by another manifest.
+ RefLevel::Manifest => None,
+ }
+}
+
+// ─── storage.files ───────────────────────────────────────────────────────
+
+/// References held by `storage.files.blob_hash`.
+///
+/// Contributes at **both** levels, which is why `RefLevel` is a parameter
+/// rather than a property of the source:
+///
+/// * [`RefLevel::Manifest`] — a CDC file's `blob_hash` names a manifest.
+/// * [`RefLevel::Chunk`] — a pre-CDC legacy file, whose `blob_hash` names a
+/// whole-file blob with no manifest behind it. The `NOT EXISTS` guard is
+/// load-bearing: for a single-chunk file the whole-file hash *equals* its
+/// lone chunk's hash, so without it the row would be counted at both
+/// levels.
+pub struct FilesReferenceSource {
+ pool: Arc,
+}
+
+impl FilesReferenceSource {
+ pub fn new(pool: Arc) -> Self {
+ Self { pool }
+ }
+}
+
+#[async_trait]
+impl BlobReferenceSource for FilesReferenceSource {
+ fn source_name(&self) -> &'static str {
+ "files"
+ }
+
+ fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option {
+ files_ref_sql(level, outer_hash_expr)
+ }
+
+ fn ref_exists_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option {
+ files_exists_sql(level, outer_hash_expr)
+ }
+
+ async fn count_references(&self, blob_hash: &str) -> Result {
+ // No level split here: the question is "how many file rows name this
+ // exact hash", and a hash names either a manifest or a legacy blob,
+ // never both at once from the caller's point of view.
+ let n: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.files WHERE blob_hash = $1")
+ .bind(blob_hash)
+ .fetch_one(self.pool.as_ref())
+ .await
+ .map_err(|e| {
+ DomainError::internal_error("BlobRefSource", format!("files count: {e}"))
+ })?;
+ Ok(n.max(0) as u64)
+ }
+
+ async fn list_referenced_blobs(
+ &self,
+ cursor: Option>,
+ limit: usize,
+ ) -> Result<(Vec, Option>), DomainError> {
+ // Paged by the file's own PK so the cursor is stable under concurrent
+ // inserts; `blob_hash` is not unique and would skip or repeat rows.
+ let after: Option = match cursor {
+ Some(bytes) => Some(decode_uuid_cursor(&bytes)?),
+ None => None,
+ };
+
+ let rows = sqlx::query(
+ "SELECT id, blob_hash FROM storage.files
+ WHERE ($1::uuid IS NULL OR id > $1)
+ ORDER BY id
+ LIMIT $2",
+ )
+ .bind(after)
+ .bind(limit as i64)
+ .fetch_all(self.pool.as_ref())
+ .await
+ .map_err(|e| DomainError::internal_error("BlobRefSource", format!("files page: {e}")))?;
+
+ let next = rows
+ .last()
+ .map(|r| r.get::("id").as_bytes().to_vec())
+ .filter(|_| rows.len() == limit);
+ let hashes = rows
+ .iter()
+ .map(|r| r.get::("blob_hash"))
+ .collect();
+ Ok((hashes, next))
+ }
+}
+
+// ─── storage.chunk_manifests ─────────────────────────────────────────────
+
+/// References held by `storage.chunk_manifests.chunk_hashes[]`.
+///
+/// Chunk level only — a manifest never references another manifest, so
+/// [`RefLevel::Manifest`] yields `None` and this source contributes nothing
+/// to the manifest recompute.
+pub struct ChunksReferenceSource {
+ pool: Arc,
+}
+
+impl ChunksReferenceSource {
+ pub fn new(pool: Arc) -> Self {
+ Self { pool }
+ }
+}
+
+#[async_trait]
+impl BlobReferenceSource for ChunksReferenceSource {
+ fn source_name(&self) -> &'static str {
+ "chunks"
+ }
+
+ fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option {
+ chunks_ref_sql(level, outer_hash_expr)
+ }
+
+ async fn count_references(&self, blob_hash: &str) -> Result {
+ let n: i64 = sqlx::query_scalar(
+ "SELECT COUNT(*) FROM storage.chunk_manifests WHERE $1 = ANY(chunk_hashes)",
+ )
+ .bind(blob_hash)
+ .fetch_one(self.pool.as_ref())
+ .await
+ .map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks count: {e}")))?;
+ Ok(n.max(0) as u64)
+ }
+
+ async fn list_referenced_blobs(
+ &self,
+ cursor: Option>,
+ limit: usize,
+ ) -> Result<(Vec, Option>), DomainError> {
+ // Paged by the manifest PK, not by the unnested chunk hash: a single
+ // manifest expands to many hashes, so the page boundary has to fall
+ // between manifests or the cursor cannot be resumed unambiguously.
+ let after: Option = match cursor {
+ Some(bytes) => Some(String::from_utf8(bytes).map_err(|e| {
+ DomainError::internal_error("BlobRefSource", format!("bad chunk cursor: {e}"))
+ })?),
+ None => None,
+ };
+
+ let rows = sqlx::query(
+ "SELECT file_hash, chunk_hashes FROM storage.chunk_manifests
+ WHERE ($1::text IS NULL OR file_hash > $1)
+ ORDER BY file_hash
+ LIMIT $2",
+ )
+ .bind(after)
+ .bind(limit as i64)
+ .fetch_all(self.pool.as_ref())
+ .await
+ .map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks page: {e}")))?;
+
+ let next = rows
+ .last()
+ .map(|r| r.get::("file_hash").into_bytes())
+ .filter(|_| rows.len() == limit);
+ let hashes = rows
+ .iter()
+ .flat_map(|r| r.get::, _>("chunk_hashes"))
+ .collect();
+ Ok((hashes, next))
+ }
+}
+
+fn decode_uuid_cursor(bytes: &[u8]) -> Result {
+ let raw: [u8; 16] = bytes.try_into().map_err(|_| {
+ DomainError::internal_error(
+ "BlobRefSource",
+ format!("bad uuid cursor: expected 16 bytes, got {}", bytes.len()),
+ )
+ })?;
+ Ok(Uuid::from_bytes(raw))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::application::ports::blob_reference_ports::BlobReferenceRegistry;
+
+ /// The registry sums whatever the sources emit; these helpers exercise the
+ /// same code path without needing a pool, since `ref_count_sql` is pure.
+ fn summed(level: RefLevel, outer: &str) -> String {
+ let frags: Vec = [files_ref_sql(level, outer), chunks_ref_sql(level, outer)]
+ .into_iter()
+ .flatten()
+ .collect();
+ if frags.is_empty() {
+ "0".to_string()
+ } else {
+ frags.join("\n + ")
+ }
+ }
+
+ /// The chunk-level expression must reproduce the two terms
+ /// `blobs_consistency` inlines today: legacy-only files (guarded by
+ /// NOT EXISTS) plus manifests citing the chunk.
+ #[test]
+ fn chunk_level_reproduces_todays_two_terms() {
+ let expr = summed(RefLevel::Chunk, "b.hash");
+ assert!(expr.contains("storage.files"), "{expr}");
+ assert!(
+ expr.contains("NOT EXISTS"),
+ "legacy term must keep the CDC guard: {expr}"
+ );
+ assert!(
+ expr.contains("= ANY(cnt_m.chunk_hashes)"),
+ "chunk term missing: {expr}"
+ );
+ assert!(expr.contains("b.hash"), "must correlate on the outer row");
+ assert!(expr.contains('+'), "both terms must be summed: {expr}");
+ }
+
+ /// Only `storage.files` references a manifest, so the manifest-level
+ /// expression is the single files term with no `NOT EXISTS` guard — the
+ /// guard exists to keep CDC rows *out* of the chunk level, and applying
+ /// it here would count nothing at all.
+ #[test]
+ fn manifest_level_is_files_only_and_unguarded() {
+ let expr = summed(RefLevel::Manifest, "m.file_hash");
+ assert!(expr.contains("storage.files"), "{expr}");
+ assert!(!expr.contains("NOT EXISTS"), "{expr}");
+ assert!(
+ !expr.contains("chunk_hashes"),
+ "chunks must not contribute at manifest level: {expr}"
+ );
+ assert!(!expr.contains('+'), "only one source contributes: {expr}");
+ assert!(expr.contains("m.file_hash"));
+ }
+
+ /// Fragments must not use the aliases the sweeps use for their outer row
+ /// (`b` for storage.blobs, `m` for chunk_manifests), or the manifest sweep
+ /// would shadow its own alias and silently correlate against itself.
+ #[test]
+ fn fragments_avoid_outer_row_aliases() {
+ for level in RefLevel::ALL {
+ let expr = summed(level, "m.file_hash");
+ for bad in [
+ "storage.files f",
+ "storage.files b",
+ "chunk_manifests m ",
+ "chunk_manifests b",
+ ] {
+ assert!(!expr.contains(bad), "alias collision at {level:?}: {expr}");
+ }
+ }
+ }
+
+ /// A source declining a level must drop out of the sum entirely, which is
+ /// what keeps manifest-only tables out of the chunk recompute where the
+ /// single-chunk hash alias would double-count them.
+ #[test]
+ fn chunks_source_declines_manifest_level() {
+ assert!(chunks_ref_sql(RefLevel::Manifest, "m.file_hash").is_none());
+ assert!(chunks_ref_sql(RefLevel::Chunk, "b.hash").is_some());
+ }
+
+ /// Guards the registry contract the sweeps rely on: an empty level still
+ /// yields a valid scalar expression.
+ #[test]
+ fn empty_registry_yields_zero_literal() {
+ let r = BlobReferenceRegistry::new();
+ assert_eq!(r.ref_count_expr(RefLevel::Manifest, "m.file_hash"), "0");
+ }
+}
diff --git a/src/infrastructure/repositories/pg/file_blob_write_repository.rs b/src/infrastructure/repositories/pg/file_blob_write_repository.rs
index b84a5378..0e57777c 100644
--- a/src/infrastructure/repositories/pg/file_blob_write_repository.rs
+++ b/src/infrastructure/repositories/pg/file_blob_write_repository.rs
@@ -1024,10 +1024,21 @@ impl FileWritePort for FileBlobWriteRepository {
DomainError::internal_error("FileBlobWrite", format!("fetch blob_hash: {e}"))
})?;
- // DELETE fires trg_files_decrement_blob_ref → storage.blobs.ref_count--
+ // DELETE fires `trg_files_decrement_blob_ref` — post-2026-08-23
+ // it dispatches manifest-first (see migration
+ // `20261017000000_file_delete_trigger_manifest_aware.sql`):
+ // decrements `chunk_manifests.ref_count` if the hash names a
+ // manifest (walking chunks on last-ref), else falls back to
+ // `storage.blobs.ref_count`. Counter state after this call is
+ // already correct.
self.delete_file(file_id).await?;
- // If the blob is now unreferenced, remove disk file + thumbnails.
+ // Physical cleanup only. `cleanup_if_orphaned` was previously
+ // manifest-aware and did counter compensation for the old
+ // trigger's over-decrement; after the trigger rewrite it's a
+ // legacy-blob-eager-reap helper — safe to keep calling
+ // unconditionally (no-op for CDC hashes; reaps legacy blobs
+ // that reached ref_count = 0).
if let Some(hash) = blob_hash {
self.dedup.cleanup_if_orphaned(&hash).await;
}
diff --git a/src/infrastructure/repositories/pg/mod.rs b/src/infrastructure/repositories/pg/mod.rs
index 313e183c..99483610 100644
--- a/src/infrastructure/repositories/pg/mod.rs
+++ b/src/infrastructure/repositories/pg/mod.rs
@@ -1,5 +1,6 @@
mod address_book_pg_repository;
mod app_password_pg_repository;
+pub mod blob_reference_sources;
mod calendar_event_pg_repository;
mod calendar_pg_repository;
mod contact_group_pg_repository;
diff --git a/src/infrastructure/scheduler/types.rs b/src/infrastructure/scheduler/types.rs
index 0cb22375..57f48f8c 100644
--- a/src/infrastructure/scheduler/types.rs
+++ b/src/infrastructure/scheduler/types.rs
@@ -45,11 +45,25 @@ use serde::{Deserialize, Serialize};
/// of the entry to probe instead of the currently-active backend.
/// `None` falls through to the live backend (today's behaviour).
/// - Others — ignored.
+///
+/// Semantics of `repair` (added 2026-10-17 for the refcount fix):
+/// - `blobs_consistency` / `manifests_consistency` — when `true`,
+/// after each `refcount_mismatch` / `manifest_refcount_mismatch`
+/// finding is recorded, apply the corrective UPDATE that sets the
+/// stored counter to the auditor's computed `actual_ref_count`.
+/// Content-safe: the row itself is fine, only the counter is
+/// wrong. Race-safe: each UPDATE recomputes the auditor formula
+/// in the same statement, so a concurrent write can't leave a
+/// stale value. Default `false` preserves discovery-only
+/// behaviour. Also propagates through `consistency_batch` to
+/// both tenants — one `?repair=true` call fixes both counters.
+/// - Others — ignored.
#[derive(Debug, Clone, Default)]
pub struct JobRunArgs {
pub force: bool,
pub deep: bool,
pub storage: Option,
+ pub repair: bool,
}
/// Uniform outcome the supervisor logs and stores for every job dispatch.
diff --git a/src/infrastructure/services/blobs_consistency_service.rs b/src/infrastructure/services/blobs_consistency_service.rs
index 5f0c3a0e..a2eae950 100644
--- a/src/infrastructure/services/blobs_consistency_service.rs
+++ b/src/infrastructure/services/blobs_consistency_service.rs
@@ -63,6 +63,7 @@ use async_trait::async_trait;
use chrono::{DateTime, Duration, Utc};
use sqlx::PgPool;
+use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::common::config::NamedStorageEntry;
use crate::infrastructure::scheduler::{
@@ -114,6 +115,64 @@ pub struct BlobsConsistencyCheck {
/// fallback for a Local target entry with no `_ROOT_DIR`. Same
/// fallback rule the boot path uses.
storage_path_fallback: PathBuf,
+ /// The chunk-level page query, assembled once from the blob-reference
+ /// registry so this recompute and `dedup_gc` agree on what "referenced"
+ /// means. Built at construction rather than per page so the sweep runs a
+ /// fixed statement — same reasoning as `DedupService::manifest_reap_sql`.
+ /// See `docs/plan/derived-blobs.md`.
+ chunk_page_sql: String,
+}
+
+/// The chunk-level page query, with `actual_ref_count` summed from the
+/// registered reference sources.
+///
+/// `storage.blobs.ref_count` semantics — the invariant `dedup_service`
+/// actually maintains:
+///
+/// ```text
+/// ref_count = (number of chunk_manifests whose chunk_hashes[] contains
+/// this hash)
+/// + (number of files.blob_hash pointing at this hash on the
+/// LEGACY whole-file path — files with NO manifest for their
+/// blob_hash)
+/// ```
+///
+/// Naively `COUNT(files) + COUNT(manifests referring)` double-counts
+/// single-chunk CDC files: where a file's whole-file hash equals its lone
+/// chunk's hash (anything under one CDC chunk), the file appears BOTH in
+/// `files.blob_hash` and in the manifest's `chunk_hashes[]`. The
+/// `NOT EXISTS` guard inside `FilesReferenceSource`'s chunk-level fragment
+/// excludes CDC-path files from the legacy term so the two don't overlap.
+///
+/// The GIN index on `chunk_hashes` (migration
+/// `20260628000000_delta_upload_gin_index`) keeps the `= ANY(chunk_hashes)`
+/// probe cheap.
+///
+/// # Panics
+///
+/// If no source contributes at [`RefLevel::Chunk`] — a wiring bug that
+/// would make every blob look unreferenced and flag the whole table as
+/// `refcount_mismatch`.
+fn chunk_page_sql(registry: &BlobReferenceRegistry) -> String {
+ let expected = registry.ref_count_expr(RefLevel::Chunk, "b.hash");
+ assert!(
+ expected != "0",
+ "no chunk-level blob reference source registered: every blob would \
+ appear unreferenced"
+ );
+
+ format!(
+ "SELECT
+ b.hash AS hash,
+ b.size AS size,
+ b.ref_count AS ref_count,
+ b.created_at AS created_at,
+ ({expected})::bigint AS actual_ref_count
+ FROM storage.blobs b
+ WHERE ($1::text IS NULL OR b.hash > $1)
+ ORDER BY b.hash
+ LIMIT $2"
+ )
}
impl BlobsConsistencyCheck {
@@ -122,12 +181,14 @@ impl BlobsConsistencyCheck {
backend: Arc,
storage_entries: Vec,
storage_path_fallback: PathBuf,
+ reference_registry: Arc,
) -> Self {
Self {
pool,
backend,
storage_entries,
storage_path_fallback,
+ chunk_page_sql: chunk_page_sql(&reference_registry),
}
}
@@ -286,6 +347,10 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
// stats.finding_count — actual persistence happens in
// `record_finding` on each emission).
let mut finding_count = 0u64;
+ // Only touched when `args.repair == true`. Symmetric with
+ // `manifests_consistency`; reported in completion log +
+ // `extra_stats` so operators see "found N, fixed M" in one line.
+ let mut repaired_count = 0u64;
// Deep mode is a per-run flag with two consumers:
// 1. This handler — decides whether to re-hash bytes.
@@ -338,6 +403,41 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
);
}
+ // Repair mode: same shape as `deep` above so the admin run-
+ // detail view can display `params.repair = "true"` alongside
+ // `params.deep`. Fresh persists what the trigger asked for;
+ // Resume reads back so a paused repair scan stays a repair
+ // scan (a mid-scan crash mustn't silently downgrade to
+ // discovery-only for the remaining rows).
+ let repair = if is_fresh {
+ let v = if args.repair { "true" } else { "false" };
+ if let Err(e) = store.set_string_param("repair", v).await {
+ return RunOutcome::Failed {
+ message: format!("failed to persist repair flag to params: {e}"),
+ };
+ }
+ args.repair
+ } else {
+ match store.get_string_param("repair").await {
+ Ok(Some(v)) => v == "true",
+ Ok(None) => false,
+ Err(e) => {
+ return RunOutcome::Failed {
+ message: format!("read `repair` from params: {e}"),
+ };
+ }
+ }
+ };
+
+ if repair {
+ tracing::info!(
+ target: "oxicloud::consistency",
+ event = "blobs_consistency.repair_mode_active",
+ run_id = %store.run_id(),
+ "repair mode: refcount_mismatch findings will trigger corrective UPDATE"
+ );
+ }
+
loop {
// Cooperative cancel poll between batches.
match store.status().await {
@@ -364,58 +464,14 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
}
}
- // Fetch the next batch. Per-row `actual_ref_count`
- // computed inline via correlated subqueries — one for
- // legacy whole-file references (`files.blob_hash`), one
- // for CDC chunk references (`chunk_manifests.chunk_hashes`).
- // GIN index on `chunk_hashes` (migration
- // 20260628000000_delta_upload_gin_index) makes the
- // `= ANY(chunk_hashes)` probe cheap.
- // `storage.blobs.ref_count` semantics — what the invariant
- // dedup_service maintains actually is:
- //
- // ref_count = (number of chunk_manifests whose
- // chunk_hashes[] contains this hash)
- // + (number of files.blob_hash pointing at
- // this hash on the LEGACY whole-file path
- // — i.e. files with NO manifest for their
- // blob_hash)
- //
- // Naively `COUNT(files) + COUNT(manifests referring)`
- // double-counts single-chunk CDC files: for a file whose
- // whole-file hash == its single chunk's hash (any file
- // small enough to fit in one CDC chunk — under ~256 KB
- // average), the file appears BOTH in `files.blob_hash`
- // AND in the manifest's `chunk_hashes[]`. The `NOT
- // EXISTS` clause below excludes CDC-path files from the
- // legacy count so the two terms don't overlap.
- let rows: Vec = match sqlx::query_as(
- r#"
- SELECT
- b.hash AS hash,
- b.size AS size,
- b.ref_count AS ref_count,
- b.created_at AS created_at,
- (
- (SELECT COUNT(*) FROM storage.files f
- WHERE f.blob_hash = b.hash
- AND NOT EXISTS (
- SELECT 1 FROM storage.chunk_manifests m
- WHERE m.file_hash = f.blob_hash
- ))
- + (SELECT COUNT(*) FROM storage.chunk_manifests m
- WHERE b.hash = ANY(m.chunk_hashes))
- )::bigint AS actual_ref_count
- FROM storage.blobs b
- WHERE ($1::text IS NULL OR b.hash > $1)
- ORDER BY b.hash
- LIMIT $2
- "#,
- )
- .bind(cursor.as_deref())
- .bind(BATCH_SIZE)
- .fetch_all(self.pool.as_ref())
- .await
+ // Fetch the next batch. `actual_ref_count` is summed from the
+ // registered reference sources — see `chunk_page_sql`, which
+ // documents the invariant and the single-chunk double-count trap.
+ let rows: Vec = match sqlx::query_as(&self.chunk_page_sql)
+ .bind(cursor.as_deref())
+ .bind(BATCH_SIZE)
+ .fetch_all(self.pool.as_ref())
+ .await
{
Ok(r) => r,
Err(e) => {
@@ -431,11 +487,17 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
event = "blobs_consistency.completed",
run_id = %store.run_id(),
finding_count = finding_count,
+ repaired_count = repaired_count,
+ repair_requested = repair,
deep = deep,
- "blobs_consistency completed with {} finding(s)",
- finding_count
+ "blobs_consistency completed with {} finding(s), {} repaired",
+ finding_count,
+ repaired_count
);
- return RunOutcome::completed();
+ return RunOutcome::completed_with(serde_json::json!({
+ "repair_requested": repair,
+ "repaired_count": repaired_count,
+ }));
}
let grace_cutoff = Utc::now() - CREATE_GRACE;
@@ -463,6 +525,67 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
}),
)
.await;
+
+ // Repair pass — content-safe corrective UPDATE. Sets
+ // `stored` to the value the auditor's two-term formula
+ // would compute at UPDATE time (subquery mirrors
+ // `chunk_page_sql`'s `actual_ref_count`), so a
+ // concurrent write between our page fetch and this
+ // UPDATE can't leave a stale value — the subquery
+ // re-reads inside the same statement. The
+ // `<> (subquery)` guard makes the UPDATE a no-op if
+ // the drift has healed, making this idempotent under
+ // retry.
+ if repair {
+ let expected = "( \
+ (SELECT COUNT(*) FROM storage.files f \
+ WHERE f.blob_hash = b.hash \
+ AND NOT EXISTS ( \
+ SELECT 1 FROM storage.chunk_manifests m \
+ WHERE m.file_hash = f.blob_hash \
+ )) \
+ + (SELECT COUNT(*) FROM storage.chunk_manifests m \
+ WHERE b.hash = ANY(m.chunk_hashes)) \
+ )";
+ let update_sql = format!(
+ "UPDATE storage.blobs b \
+ SET ref_count = {expected} \
+ WHERE b.hash = $1 \
+ AND b.ref_count <> {expected}",
+ );
+ match sqlx::query(&update_sql)
+ .bind(&row.hash)
+ .execute(self.pool.as_ref())
+ .await
+ {
+ Ok(res) if res.rows_affected() > 0 => {
+ repaired_count += 1;
+ tracing::info!(
+ target: "audit",
+ event = "blobs_consistency.repaired",
+ run_id = %store.run_id(),
+ hash = %row.hash,
+ stored_was = row.ref_count,
+ actual = row.actual_ref_count,
+ "🩹 blob ref_count repaired"
+ );
+ }
+ Ok(_) => {
+ // No row touched — concurrent repair or
+ // self-healing drift. Silent no-op.
+ }
+ Err(e) => {
+ tracing::warn!(
+ target: "oxicloud::consistency",
+ event = "blobs_consistency.repair_failed",
+ run_id = %store.run_id(),
+ hash = %row.hash,
+ error = %e,
+ "blob ref_count repair UPDATE failed — finding stays"
+ );
+ }
+ }
+ }
}
// Skip physical probes for rows within the write
@@ -611,11 +734,17 @@ impl RecoverableJobHandler for BlobsConsistencyCheck {
event = "blobs_consistency.completed",
run_id = %store.run_id(),
finding_count = finding_count,
+ repaired_count = repaired_count,
+ repair_requested = repair,
deep = deep,
- "blobs_consistency completed with {} finding(s)",
- finding_count
+ "blobs_consistency completed with {} finding(s), {} repaired",
+ finding_count,
+ repaired_count
);
- return RunOutcome::completed();
+ return RunOutcome::completed_with(serde_json::json!({
+ "repair_requested": repair,
+ "repaired_count": repaired_count,
+ }));
}
}
}
@@ -683,3 +812,64 @@ async fn recompute_hash(
Ok(hasher.finalize().to_hex().to_string())
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::infrastructure::repositories::pg::blob_reference_sources::{
+ ChunksReferenceSource, FilesReferenceSource,
+ };
+
+ fn default_registry() -> BlobReferenceRegistry {
+ let pool = Arc::new(
+ sqlx::pool::PoolOptions::::new()
+ .connect_lazy("postgres://invalid/invalid")
+ .expect("lazy pool never connects"),
+ );
+ let mut registry = BlobReferenceRegistry::new();
+ registry.register(Arc::new(FilesReferenceSource::new(pool.clone())));
+ registry.register(Arc::new(ChunksReferenceSource::new(pool)));
+ registry
+ }
+
+ /// Golden test for the chunk-level recompute. Pins the statement
+ /// byte-for-byte because it is assembled from the registry rather than
+ /// written as a literal — the reviewer should read the SQL here.
+ ///
+ /// This expression must stay equal to what the query computed before the
+ /// registry existed: the legacy-files term guarded by `NOT EXISTS`, plus
+ /// the manifests-citing-this-chunk term. If a change makes those two
+ /// overlap, every single-chunk CDC file is counted twice and the whole
+ /// table reports `refcount_mismatch`.
+ #[tokio::test]
+ async fn chunk_page_statement_is_stable() {
+ let sql = chunk_page_sql(&default_registry());
+ let expected = r#"SELECT
+ b.hash AS hash,
+ b.size AS size,
+ b.ref_count AS ref_count,
+ b.created_at AS created_at,
+ ((SELECT COUNT(*) FROM storage.files cnt_f
+ WHERE cnt_f.blob_hash = b.hash
+ AND NOT EXISTS (
+ SELECT 1 FROM storage.chunk_manifests cnt_m
+ WHERE cnt_m.file_hash = cnt_f.blob_hash
+ ))
+ + (SELECT COUNT(*) FROM storage.chunk_manifests cnt_m
+ WHERE b.hash = ANY(cnt_m.chunk_hashes)))::bigint AS actual_ref_count
+ FROM storage.blobs b
+ WHERE ($1::text IS NULL OR b.hash > $1)
+ ORDER BY b.hash
+ LIMIT $2"#;
+ assert_eq!(sql, expected, "chunk page statement changed:\n{sql}");
+ }
+
+ /// With no chunk-level source every blob would look unreferenced and the
+ /// sweep would report the entire table as `refcount_mismatch`. Refuse to
+ /// build the statement instead.
+ #[test]
+ #[should_panic(expected = "no chunk-level blob reference source")]
+ fn empty_registry_refuses_to_build_page_statement() {
+ let _ = chunk_page_sql(&BlobReferenceRegistry::new());
+ }
+}
diff --git a/src/infrastructure/services/consistency_batch_service.rs b/src/infrastructure/services/consistency_batch_service.rs
index ad174da1..f167615a 100644
--- a/src/infrastructure/services/consistency_batch_service.rs
+++ b/src/infrastructure/services/consistency_batch_service.rs
@@ -178,6 +178,7 @@ impl JobHandler for ConsistencyBatch {
"per_check": per_check,
"deep": args.deep,
"force": args.force,
+ "repair": args.repair,
"ok": ok_count,
"err": err_count,
}),
diff --git a/src/infrastructure/services/dedup_service.rs b/src/infrastructure/services/dedup_service.rs
index c8b84fa3..243f037e 100644
--- a/src/infrastructure/services/dedup_service.rs
+++ b/src/infrastructure/services/dedup_service.rs
@@ -55,6 +55,7 @@ use std::sync::Arc;
use tokio_util::io::StreamReader;
use crate::application::ports::blob_lifecycle::BlobLifecycleHook;
+use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
use crate::application::ports::blob_storage_ports::BlobStorageBackend;
use crate::application::ports::dedup_ports::{
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
@@ -424,6 +425,51 @@ async fn populate_integrity_blob_sizes<'a>(
IntegrityBlobSizes { hashes, sizes }
}
+/// Build the manifest reap statement from the registered reference sources.
+///
+/// A manifest is collectible when either:
+/// * `ref_count` reached 0 via `cleanup_if_orphaned` on the single-file
+/// delete path, **or**
+/// * nothing references it any more — the bulk-delete path (user cascade,
+/// `empty_trash`), where the PG trigger only touches `storage.blobs` and
+/// the per-file `cleanup_if_orphaned` call is skipped, so `ref_count` is
+/// never decremented and the second clause is the only thing that reaps.
+///
+/// The second clause used to name `storage.files` directly, which hardcoded
+/// "files is the only thing that can reference a manifest". Any new referring
+/// table — thumbnails via `storage.content_derived_blobs`, previews via
+/// `storage.file_attached_blobs` — would then have its manifests reaped on the
+/// next sweep *despite a correct `ref_count`*: clause one false, clause two
+/// true, `OR` fires, bytes gone. See `docs/plan/derived-blobs.md`.
+///
+/// # Panics
+///
+/// If no source contributes at [`RefLevel::Manifest`]. That is a wiring bug,
+/// and it must be loud: with no source, "nothing references it" is vacuously
+/// true for every row and this statement would delete every manifest in the
+/// database. `DedupService::new` always registers `FilesReferenceSource`, so
+/// the only way to reach this is to pass a deliberately empty registry.
+fn manifest_reap_sql(registry: &BlobReferenceRegistry) -> String {
+ let orphaned = registry
+ .no_reference_predicate(RefLevel::Manifest, "m.file_hash")
+ .expect(
+ "no manifest-level blob reference source registered: the reap \
+ predicate would match every manifest",
+ );
+
+ format!(
+ "DELETE FROM storage.chunk_manifests
+ WHERE ctid = ANY(
+ SELECT ctid
+ FROM storage.chunk_manifests m
+ WHERE m.ref_count <= 0
+ OR {orphaned}
+ LIMIT $1
+ )
+ RETURNING file_hash, chunk_hashes, total_size"
+ )
+}
+
pub struct DedupService {
/// Pluggable blob storage backend (local FS, S3, …).
backend: Arc,
@@ -442,6 +488,16 @@ pub struct DedupService {
/// seen immediately), weight-bounded (a manifest is ~72 B per chunk),
/// short TTL so GC'd manifests age out fast (benches/MANIFEST-CACHE.md).
manifest_cache: moka::future::Cache>,
+ /// Every table that holds blob references, so GC agrees with the
+ /// consistency jobs on what "referenced" means. Defaults to the two
+ /// built-in sources; DI replaces it once more tables exist. Never
+ /// optional — an empty registry would make "nothing references it"
+ /// vacuously true and the manifest sweep would reap everything.
+ reference_registry: Arc,
+ /// The manifest reap statement, built once from `reference_registry`.
+ /// Kept as a field so `garbage_collect` runs a fixed statement rather
+ /// than assembling SQL inside a delete loop — see `manifest_reap_sql`.
+ manifest_reap_sql: String,
}
impl DedupService {
@@ -455,15 +511,32 @@ impl DedupService {
pool: Arc,
maintenance_pool: Arc,
) -> Self {
+ let registry = Arc::new(Self::default_reference_registry(pool.clone()));
Self {
backend,
pool,
maintenance_pool,
blob_lifecycle: None,
manifest_cache: Self::build_manifest_cache(),
+ reference_registry: registry.clone(),
+ manifest_reap_sql: manifest_reap_sql(®istry),
}
}
+ /// The two sources that were implicit before the registry existed.
+ /// Keeping this as the default means every construction path — including
+ /// tests — has a manifest-level source, so the reap predicate can never
+ /// degenerate to "nothing references anything".
+ fn default_reference_registry(pool: Arc) -> BlobReferenceRegistry {
+ use crate::infrastructure::repositories::pg::blob_reference_sources::{
+ ChunksReferenceSource, FilesReferenceSource,
+ };
+ let mut registry = BlobReferenceRegistry::new();
+ registry.register(Arc::new(FilesReferenceSource::new(pool.clone())));
+ registry.register(Arc::new(ChunksReferenceSource::new(pool)));
+ registry
+ }
+
/// See the `manifest_cache` field docs. Weight ≈ real heap bytes of one
/// entry; 32 MiB cap ≈ tens of thousands of typical (sub-1 GB) files.
fn build_manifest_cache() -> moka::future::Cache> {
@@ -476,6 +549,25 @@ impl DedupService {
.build()
}
+ /// Registers the blob-reference registry used by the manifest reap
+ /// predicate. Without it `garbage_collect` skips manifest collection
+ /// entirely — see `docs/plan/derived-blobs.md`.
+ pub fn with_reference_registry(mut self, registry: Arc) -> Self {
+ self.manifest_reap_sql = manifest_reap_sql(®istry);
+ self.reference_registry = registry;
+ self
+ }
+
+ /// The registry backing the reap predicate.
+ ///
+ /// Exposed so `blobs_consistency` recomputes refcounts from the *same*
+ /// source set GC reaps from. If the two ever diverged, the sweep would
+ /// bless counts the collector disagrees with — and the collector wins,
+ /// destructively.
+ pub fn reference_registry(&self) -> Arc {
+ self.reference_registry.clone()
+ }
+
/// Registers the blob lifecycle dispatcher (thumbnail cleanup, …).
pub fn with_blob_lifecycle(mut self, lifecycle: Arc) -> Self {
self.blob_lifecycle = Some(lifecycle);
@@ -510,12 +602,15 @@ impl DedupService {
.connect_lazy("postgres://invalid:5432/none")
.unwrap(),
);
+ let stub_registry = Arc::new(Self::default_reference_registry(stub_pool.clone()));
Self {
backend: Arc::new(LocalBlobBackend::new(Path::new("/tmp/oxicloud_stub_blobs"))),
pool: stub_pool.clone(),
- maintenance_pool: stub_pool,
+ maintenance_pool: stub_pool.clone(),
blob_lifecycle: None,
manifest_cache: Self::build_manifest_cache(),
+ reference_registry: stub_registry.clone(),
+ manifest_reap_sql: manifest_reap_sql(&stub_registry),
}
}
@@ -523,6 +618,32 @@ impl DedupService {
pub async fn initialize(&self) -> Result<(), DomainError> {
self.backend.initialize().await?;
+ // The reap statement is assembled from the registered reference
+ // sources, so it is not greppable in the source tree. It DELETES
+ // manifests, so log it unconditionally at info rather than hiding it
+ // behind a filter an operator has to know to enable — if what GC
+ // considers "referenced" ever changes, that must be visible on the
+ // next boot without anyone going looking.
+ //
+ // Whitespace-collapsed to a single field so a multi-line query does
+ // not sprawl across the boot log; expand it with
+ // `sed 's/ AND / AND\n /g'` or just paste it into psql.
+ tracing::info!(
+ target: "oxicloud::dedup",
+ sources = ?self
+ .reference_registry
+ .sources()
+ .iter()
+ .map(|s| s.source_name())
+ .collect::>(),
+ statement = %self
+ .manifest_reap_sql
+ .split_whitespace()
+ .collect::>()
+ .join(" "),
+ "🧹 manifest reap predicate registered"
+ );
+
let blob_count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.blobs")
.fetch_one(self.pool.as_ref())
.await
@@ -1838,66 +1959,48 @@ impl DedupService {
pub async fn cleanup_if_orphaned(&self, hash: &str) {
let short = &hash[..hash.len().min(12)];
- // ── CDC manifest path (must run FIRST) ───────────────────
- // For single-chunk CDC files file_hash == chunk_hash, so the PG
- // trigger on storage.files already decremented storage.blobs.ref_count
- // when this function is called. try_dedup_hit increments
- // chunk_manifests.ref_count but NOT storage.blobs.ref_count, so
- // blobs.ref_count can reach 0 while the manifest still has ref_count > 1
- // (other files sharing the same blob). Checking the manifest first
- // prevents premature blob + manifest deletion.
- let manifest = sqlx::query_as::<_, (i32, Vec)>(
- "SELECT ref_count, chunk_hashes \
- FROM storage.chunk_manifests WHERE file_hash = $1",
- )
- .bind(hash)
- .fetch_optional(self.pool.as_ref())
- .await
- .unwrap_or(None);
-
- if let Some((ref_count, chunk_hashes)) = manifest {
- if ref_count <= 1 {
- // Last reference — remove manifest and all its chunks.
- if let Err(e) = self
- .remove_manifest_reference(hash, ref_count, &chunk_hashes)
- .await
- {
- tracing::warn!("cleanup_if_orphaned: manifest cleanup failed for {short}: {e}");
- }
- } else {
- // Other files still share this blob: just decrement the manifest
- // counter and undo the PG trigger's premature chunk ref_count
- // decrement (blobs.ref_count is chunk-level; the manifest is the
- // authoritative file-level counter).
- sqlx::query(
- "UPDATE storage.chunk_manifests \
- SET ref_count = ref_count - 1 WHERE file_hash = $1",
- )
- .bind(hash)
- .execute(self.pool.as_ref())
- .await
- .ok();
- // Undo the PG trigger's decrement of storage.blobs.ref_count.
- // The trigger fired with blob_hash = file_hash, so only the row
- // WHERE hash = file_hash is affected. For single-chunk files
- // file_hash == chunk_hash and that row exists; for multi-chunk
- // files file_hash is not in storage.blobs, making this a no-op.
- sqlx::query("UPDATE storage.blobs SET ref_count = ref_count + 1 WHERE hash = $1")
- .bind(hash)
- .execute(self.pool.as_ref())
- .await
- .ok();
- tracing::debug!(
- "cleanup_if_orphaned: manifest {short} ref_count {ref_count}→{}",
- ref_count - 1
- );
- }
- return;
- }
-
- // ── Legacy blob path (no manifest) ───────────────────────
+ // 2026-08-23 refactor: this function used to compensate for the
+ // OLD PG trigger `trg_files_decrement_blob_ref` unconditionally
+ // decrementing `storage.blobs.ref_count`, which was wrong for
+ // CDC files (their `blob_hash` names a `chunk_manifests.file_hash`,
+ // not a chunk-in-a-manifest). The compensation branches would:
+ // * Decrement `chunk_manifests.ref_count` a SECOND time (the
+ // trigger having wrongly touched blobs, not the manifest);
+ // * Undo the trigger's blob decrement (rc > 1 branch);
+ // * Call `remove_manifest_reference` (rc <= 1 branch), which
+ // deletes manifest + dereferences chunks — again duplicating
+ // work the trigger should own.
+ //
+ // Migration `20261017000000_file_delete_trigger_manifest_aware.sql`
+ // rewrote the trigger to be manifest-aware, so it now correctly
+ // decrements EITHER the manifest OR the blob depending on which
+ // one the hash names, walks chunks on last-ref manifest delete,
+ // and leaves the counters in a consistent state without any
+ // compensation call. Running the old compensation ON TOP of the
+ // new trigger causes double-decrement / double-delete and is
+ // exactly what broke `dedup_blob_cleanup.hurl` step 7
+ // (`ref_count == 1` observed 0 after purging one of two dedup
+ // uploads).
+ //
+ // What remains here: **physical cleanup only**. If the trigger
+ // brought a LEGACY whole-file blob to ref_count = 0 and no
+ // manifest still references it (either directly via file_hash or
+ // indirectly as a chunk in another manifest's chunk_hashes[]),
+ // reap the DB row and the backend file eagerly. For CDC chunks
+ // whose ref_count reached 0 via the trigger's last-ref manifest
+ // path, `dedup_gc` handles physical reap with a grace window
+ // against re-upload races.
+ //
+ // Callers can keep invoking `cleanup_if_orphaned` unconditionally
+ // — for CDC paths it's a cheap no-op (manifest still exists OR
+ // the hash never had a blob row), for legacy paths it reaps.
let deleted_blob = sqlx::query_scalar::<_, String>(
- "DELETE FROM storage.blobs WHERE hash = $1 AND ref_count <= 0 RETURNING hash",
+ "DELETE FROM storage.blobs \
+ WHERE hash = $1 \
+ AND ref_count <= 0 \
+ AND NOT EXISTS (SELECT 1 FROM storage.chunk_manifests \
+ WHERE $1 = ANY(chunk_hashes)) \
+ RETURNING hash",
)
.bind(hash)
.fetch_optional(self.pool.as_ref())
@@ -1909,7 +2012,7 @@ impl DedupService {
tracing::warn!("cleanup_if_orphaned: disk delete failed for {short}: {e}");
}
self.fire_blob_hooks(hash);
- tracing::info!("cleanup_if_orphaned: removed orphaned blob {short}");
+ tracing::info!("cleanup_if_orphaned: removed orphaned legacy blob {short}");
}
}
@@ -2558,10 +2661,18 @@ impl DedupService {
// A manifest is collectible when:
// • ref_count has been decremented to 0 by cleanup_if_orphaned
// on the single-file-delete service path, OR
- // • no `storage.files.blob_hash` references its file_hash
+ // • NO registered reference source references its file_hash
// (covers bulk-delete paths: user cascade, empty_trash —
// where the PG trigger only touches storage.blobs and the
// per-file cleanup_if_orphaned call is skipped).
+ //
+ // The second clause used to name `storage.files` directly. That
+ // hardcoded "files is the only thing that can reference a manifest",
+ // so any new referring table (thumbnails via
+ // storage.content_derived_blobs, …) would see its manifests reaped
+ // on the next sweep despite a correct ref_count — the first clause
+ // is false, the second true, and the OR fires. It is now the union
+ // of every registered source; see docs/plan/derived-blobs.md.
loop {
// Keep the historically cheap DELETE-only shape for the dominant
// no-work sweep. Embedding it in the delete/aggregate/update CTE
@@ -2570,23 +2681,14 @@ impl DedupService {
// update. From two onward, aggregate in-process and issue one UPDATE:
// the measured crossover is already positive at two, while 500 and
// 1,000 manifests improve by 60.03x and 51.16x respectively.
- let batch: Vec<(String, Vec, i64)> = sqlx::query_as(
- "DELETE FROM storage.chunk_manifests
- WHERE ctid = ANY(
- SELECT ctid FROM storage.chunk_manifests m
- WHERE m.ref_count <= 0
- OR NOT EXISTS (
- SELECT 1 FROM storage.files f
- WHERE f.blob_hash = m.file_hash
- )
- LIMIT $1
- )
- RETURNING file_hash, chunk_hashes, total_size",
- )
- .bind(BATCH_SIZE)
- .fetch_all(self.maintenance_pool.as_ref())
- .await
- .map_err(|e| DomainError::internal_error("Dedup", format!("GC manifests: {e}")))?;
+ // Assembled once at construction (see `manifest_reap_sql`), not
+ // per sweep: no string work in the hot path, a stable statement for
+ // prepared-statement caching, and a byte-for-byte golden test.
+ let batch: Vec<(String, Vec, i64)> = sqlx::query_as(&self.manifest_reap_sql)
+ .bind(BATCH_SIZE)
+ .fetch_all(self.maintenance_pool.as_ref())
+ .await
+ .map_err(|e| DomainError::internal_error("Dedup", format!("GC manifests: {e}")))?;
if batch.is_empty() {
break;
@@ -3267,6 +3369,41 @@ impl crate::infrastructure::scheduler::JobHandler for DedupService {
#[cfg(test)]
mod tests {
use super::*;
+
+ /// Golden test for the statement `garbage_collect` runs against production
+ /// data. It is assembled from the registered reference sources rather than
+ /// written as a literal, so this pins the whole thing byte-for-byte — the
+ /// point being that a reviewer reads the SQL *here* instead of mentally
+ /// evaluating the registry.
+ ///
+ /// If this fails after adding a source, read the diff carefully: the new
+ /// branch must appear inside the `NOT (...)` group, ORed with the others.
+ /// A branch landing outside that group inverts the predicate for every
+ /// other source and reaps live manifests.
+ #[tokio::test]
+ async fn manifest_reap_statement_is_stable() {
+ let sql = DedupService::new_stub().manifest_reap_sql;
+ let expected = r#"DELETE FROM storage.chunk_manifests
+ WHERE ctid = ANY(
+ SELECT ctid
+ FROM storage.chunk_manifests m
+ WHERE m.ref_count <= 0
+ OR NOT (EXISTS (SELECT 1 FROM storage.files cnt_f WHERE cnt_f.blob_hash = m.file_hash))
+ LIMIT $1
+ )
+ RETURNING file_hash, chunk_hashes, total_size"#;
+ assert_eq!(sql, expected, "reap statement changed:\n{sql}");
+ }
+
+ /// The reap predicate must never match a manifest that some source still
+ /// references. With an empty registry `NOT (...)` would have no operands,
+ /// so the builder refuses rather than emitting a statement that deletes
+ /// every manifest in the database.
+ #[test]
+ #[should_panic(expected = "no manifest-level blob reference source")]
+ fn empty_registry_refuses_to_build_reap_statement() {
+ let _ = manifest_reap_sql(&BlobReferenceRegistry::new());
+ }
use std::collections::HashSet;
use tempfile::NamedTempFile;
diff --git a/src/infrastructure/services/manifests_consistency_service.rs b/src/infrastructure/services/manifests_consistency_service.rs
new file mode 100644
index 00000000..3ee5b016
--- /dev/null
+++ b/src/infrastructure/services/manifests_consistency_service.rs
@@ -0,0 +1,462 @@
+//! Reconciles `storage.chunk_manifests.ref_count` against its actual
+//! referrers.
+//!
+//! ### Why this exists
+//!
+//! There are **two** reference counters, and only one of them was ever
+//! verified. `DedupService::add_reference` bumps
+//! `chunk_manifests.ref_count` first and only falls back to
+//! `storage.blobs.ref_count`, so a reference lands on whichever counter
+//! its hash names:
+//!
+//! * a **chunk** reference → `storage.blobs.ref_count`, reconciled by
+//! `blobs_consistency::refcount_mismatch`;
+//! * a **Blob** reference (a CDC file, and now every derived artifact) →
+//! `chunk_manifests.ref_count`, reconciled by **nothing** before this
+//! job existed.
+//!
+//! That gap was survivable only because `dedup_gc`'s reap predicate had a
+//! second clause — "no `storage.files` row references this manifest" —
+//! which quietly compensated for drift on the bulk-delete paths where
+//! `ref_count` is never decremented. Generalising that clause to the
+//! reference registry (so thumbnails stop being reaped) removes the
+//! compensation, which is exactly why the manifest counter now has to be
+//! checked directly. See `docs/plan/derived-blobs.md`.
+//!
+//! ### The check
+//!
+//! * `manifest_refcount_mismatch` (severity `inconsistent`) —
+//! `chunk_manifests.ref_count` disagrees with the number of registered
+//! referrers. An **under**-count is the dangerous direction: GC reaps a
+//! manifest whose content is still reachable, taking its chunks with it.
+//! An over-count merely pins storage. Content-safe to report either way
+//! — the manifest row and its chunks are intact, the counter is wrong.
+//!
+//! ### Why a separate job rather than a phase of `blobs_consistency`
+//!
+//! One subject per job, per the subject-iteration principle the other five
+//! consistency tenants follow. It also avoids changing the cursor format of
+//! an existing *recoverable* job, which would strand any run paused across
+//! the deploy.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+use sqlx::PgPool;
+
+use crate::application::ports::blob_reference_ports::{BlobReferenceRegistry, RefLevel};
+use crate::infrastructure::scheduler::{
+ JobRegistry, JobRunArgs, JobStore, JobStoreProvider, RecoverableJobHandler, RunOutcome,
+ RunStatus, record_or_log,
+};
+
+pub const MANIFESTS_CONSISTENCY_JOB_NAME: &str = "manifests_consistency";
+
+/// Rows per batch. Each row costs one indexed subquery per registered
+/// source; 200 matches `blobs_consistency` so the cancel-poll cadence is
+/// the same for an operator watching either job.
+const BATCH_SIZE: i64 = 200;
+
+/// The page query, with `actual_ref_count` summed from the registered
+/// reference sources at [`RefLevel::Manifest`].
+///
+/// Only sources that reference a **Blob** contribute — `storage.files`
+/// today, plus `storage.content_derived_blobs` and
+/// `storage.file_attached_blobs` once they exist.
+/// `ChunksReferenceSource` returns `None` here: a manifest is never
+/// referenced by another manifest, and including it would count this
+/// manifest's own chunks as referrers of itself.
+///
+/// # Panics
+///
+/// If no source contributes at [`RefLevel::Manifest`] — a wiring bug that
+/// would report every manifest as mismatched.
+fn manifest_page_sql(registry: &BlobReferenceRegistry) -> String {
+ let expected = registry.ref_count_expr(RefLevel::Manifest, "m.file_hash");
+ assert!(
+ expected != "0",
+ "no manifest-level blob reference source registered: every manifest \
+ would appear unreferenced"
+ );
+
+ format!(
+ "SELECT
+ m.file_hash AS file_hash,
+ m.ref_count AS ref_count,
+ m.total_size AS total_size,
+ m.chunk_count AS chunk_count,
+ ({expected})::bigint AS actual_ref_count
+ FROM storage.chunk_manifests m
+ WHERE ($1::text IS NULL OR m.file_hash > $1)
+ ORDER BY m.file_hash
+ LIMIT $2"
+ )
+}
+
+pub struct ManifestsConsistencyCheck {
+ pool: Arc,
+ /// Built once from the blob-reference registry so this recompute and
+ /// `dedup_gc`'s reap predicate answer "what references this manifest"
+ /// identically. Assembled at construction rather than per page so the
+ /// sweep runs a fixed statement.
+ page_sql: String,
+}
+
+impl ManifestsConsistencyCheck {
+ pub fn new(pool: Arc, reference_registry: Arc) -> Self {
+ Self {
+ pool,
+ page_sql: manifest_page_sql(&reference_registry),
+ }
+ }
+
+ /// Chainable self-registration. On-demand only — operators fire it
+ /// from `POST /api/admin/jobs/manifests_consistency/trigger`.
+ pub async fn register_recoverable_job(
+ self: Arc,
+ registry: &JobRegistry,
+ provider: &Arc,
+ ) -> Arc {
+ registry
+ .register_recoverable_job(self.clone(), provider.clone(), None)
+ .await;
+ self
+ }
+}
+
+#[derive(Debug, sqlx::FromRow)]
+struct ManifestRow {
+ file_hash: String,
+ ref_count: i32,
+ total_size: i64,
+ chunk_count: i32,
+ actual_ref_count: i64,
+}
+
+#[async_trait]
+impl RecoverableJobHandler for ManifestsConsistencyCheck {
+ fn name(&self) -> &str {
+ MANIFESTS_CONSISTENCY_JOB_NAME
+ }
+
+ async fn count_total(&self) -> Option {
+ let row: Result<(i64,), sqlx::Error> =
+ sqlx::query_as("SELECT COUNT(*) FROM storage.chunk_manifests")
+ .fetch_one(self.pool.as_ref())
+ .await;
+ match row {
+ Ok((n,)) => Some(n.max(0) as u64),
+ Err(e) => {
+ tracing::debug!(
+ target: "oxicloud::consistency",
+ event = "manifests_consistency.count_total_failed",
+ error = %e,
+ "count_total failed — run will not surface a progress bar"
+ );
+ None
+ }
+ }
+ }
+
+ async fn run_resumable(
+ &self,
+ store: &dyn JobStore,
+ args: &JobRunArgs,
+ resume_cursor: Option>,
+ ) -> RunOutcome {
+ let is_fresh = resume_cursor.is_none();
+
+ // Cursor: the last `file_hash` as UTF-8. Same convention as
+ // `blobs_consistency`, which also pages a hash-keyed table.
+ let mut cursor: Option = match resume_cursor {
+ None => None,
+ Some(bytes) if bytes.is_empty() => None,
+ Some(bytes) => match String::from_utf8(bytes) {
+ Ok(s) => Some(s),
+ Err(e) => {
+ return RunOutcome::Failed {
+ message: format!("invalid cursor: not valid UTF-8: {e}"),
+ };
+ }
+ },
+ };
+
+ // Persist the repair flag into `params.repair` so the admin
+ // run-detail view can display whether the run was a discovery
+ // scan or an active repair. Fresh takes it from args; Resume
+ // reads back so a paused repair scan stays a repair scan (a
+ // mid-scan crash mustn't silently downgrade the remaining
+ // rows to discovery-only). Same shape as
+ // `blobs_consistency_service.rs`'s `deep` handling — see the
+ // reasoning documented there.
+ let repair = if is_fresh {
+ let v = if args.repair { "true" } else { "false" };
+ if let Err(e) = store.set_string_param("repair", v).await {
+ return RunOutcome::Failed {
+ message: format!("failed to persist repair flag to params: {e}"),
+ };
+ }
+ args.repair
+ } else {
+ match store.get_string_param("repair").await {
+ Ok(Some(v)) => v == "true",
+ Ok(None) => false,
+ Err(e) => {
+ return RunOutcome::Failed {
+ message: format!("read `repair` from params: {e}"),
+ };
+ }
+ }
+ };
+
+ if repair {
+ tracing::info!(
+ target: "oxicloud::consistency",
+ event = "manifests_consistency.repair_mode_active",
+ run_id = %store.run_id(),
+ "repair mode: manifest_refcount_mismatch findings will trigger corrective UPDATE"
+ );
+ }
+
+ let mut finding_count = 0u64;
+ // Only relevant when `repair == true`. Reported inline in
+ // the completion log + the `extra_stats` payload so operators
+ // can see "we found N and fixed M" in one line.
+ let mut repaired_count = 0u64;
+
+ loop {
+ // Cooperative cancel poll between batches.
+ match store.status().await {
+ Ok(RunStatus::CancelRequested) => {
+ tracing::info!(
+ target: "oxicloud::consistency",
+ event = "manifests_consistency.cancelled",
+ run_id = %store.run_id(),
+ finding_count = finding_count,
+ "manifests_consistency cancelled cooperatively, pausing"
+ );
+ return RunOutcome::Paused {
+ cursor: cursor
+ .as_ref()
+ .map(|s| s.as_bytes().to_vec())
+ .unwrap_or_default(),
+ };
+ }
+ Ok(_) => {}
+ Err(e) => {
+ return RunOutcome::Failed {
+ message: format!("status poll: {e}"),
+ };
+ }
+ }
+
+ let rows: Vec = match sqlx::query_as(&self.page_sql)
+ .bind(cursor.as_deref())
+ .bind(BATCH_SIZE)
+ .fetch_all(self.pool.as_ref())
+ .await
+ {
+ Ok(r) => r,
+ Err(e) => {
+ return RunOutcome::Failed {
+ message: format!("batch fetch: {e}"),
+ };
+ }
+ };
+
+ if rows.is_empty() {
+ tracing::info!(
+ target: "oxicloud::consistency",
+ event = "manifests_consistency.completed",
+ run_id = %store.run_id(),
+ finding_count = finding_count,
+ repaired_count = repaired_count,
+ repair_requested = repair,
+ "manifests_consistency completed with {} finding(s), {} repaired",
+ finding_count,
+ repaired_count
+ );
+ return RunOutcome::completed_with(serde_json::json!({
+ "repair_requested": repair,
+ "repaired_count": repaired_count,
+ }));
+ }
+
+ for row in &rows {
+ if row.ref_count as i64 == row.actual_ref_count {
+ continue;
+ }
+ finding_count += 1;
+ let delta = row.actual_ref_count - row.ref_count as i64;
+ record_or_log(
+ store,
+ MANIFESTS_CONSISTENCY_JOB_NAME,
+ "manifest_refcount_mismatch",
+ "inconsistent",
+ None, // a hash isn't a UUID; the identifier lives in detail
+ serde_json::json!({
+ "file_hash": row.file_hash,
+ "stored": row.ref_count,
+ "actual": row.actual_ref_count,
+ "delta": delta,
+ "total_size": row.total_size,
+ "chunk_count": row.chunk_count,
+ // Under-count is the dangerous direction: GC reaps a
+ // manifest whose content is still reachable.
+ "reap_risk": delta > 0,
+ }),
+ )
+ .await;
+
+ // Repair pass — content-safe corrective UPDATE. The
+ // stored counter is set to what the auditor formula
+ // would compute at UPDATE time (subquery matches
+ // `manifest_page_sql`'s `actual_ref_count` predicate),
+ // so a concurrent file insert/delete between our page
+ // fetch and this UPDATE can't leave a stale value —
+ // the subquery re-reads inside the same statement.
+ // The `<> (subquery)` guard makes the UPDATE a no-op
+ // if the value is already correct, so this is
+ // idempotent under retry.
+ if repair {
+ match sqlx::query(
+ "UPDATE storage.chunk_manifests m \
+ SET ref_count = ( \
+ SELECT COUNT(*) FROM storage.files \
+ WHERE blob_hash = m.file_hash \
+ ) \
+ WHERE m.file_hash = $1 \
+ AND m.ref_count <> ( \
+ SELECT COUNT(*) FROM storage.files \
+ WHERE blob_hash = m.file_hash \
+ )",
+ )
+ .bind(&row.file_hash)
+ .execute(self.pool.as_ref())
+ .await
+ {
+ Ok(res) if res.rows_affected() > 0 => {
+ repaired_count += 1;
+ tracing::info!(
+ target: "audit",
+ event = "manifests_consistency.repaired",
+ run_id = %store.run_id(),
+ file_hash = %row.file_hash,
+ stored_was = row.ref_count,
+ actual = row.actual_ref_count,
+ "🩹 manifest ref_count repaired"
+ );
+ }
+ Ok(_) => {
+ // Row not touched — either another concurrent
+ // repair fixed it first, or the drift healed
+ // itself between page fetch and UPDATE.
+ // Silent no-op.
+ }
+ Err(e) => {
+ tracing::warn!(
+ target: "oxicloud::consistency",
+ event = "manifests_consistency.repair_failed",
+ run_id = %store.run_id(),
+ file_hash = %row.file_hash,
+ error = %e,
+ "manifest ref_count repair UPDATE failed — finding stays"
+ );
+ }
+ }
+ }
+ }
+
+ // Advance cursor + checkpoint.
+ let last_hash = rows
+ .last()
+ .map(|r| r.file_hash.clone())
+ .expect("non-empty rows");
+ cursor = Some(last_hash.clone());
+ let batch_len = rows.len() as u64;
+ if let Err(e) = store.checkpoint(last_hash.into_bytes(), batch_len).await {
+ return RunOutcome::Failed {
+ message: format!("checkpoint: {e}"),
+ };
+ }
+
+ if (rows.len() as i64) < BATCH_SIZE {
+ tracing::info!(
+ target: "oxicloud::consistency",
+ event = "manifests_consistency.completed",
+ run_id = %store.run_id(),
+ finding_count = finding_count,
+ repaired_count = repaired_count,
+ repair_requested = repair,
+ "manifests_consistency completed with {} finding(s), {} repaired",
+ finding_count,
+ repaired_count
+ );
+ return RunOutcome::completed_with(serde_json::json!({
+ "repair_requested": repair,
+ "repaired_count": repaired_count,
+ }));
+ }
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::infrastructure::repositories::pg::blob_reference_sources::{
+ ChunksReferenceSource, FilesReferenceSource,
+ };
+
+ fn default_registry() -> BlobReferenceRegistry {
+ let pool = Arc::new(
+ sqlx::pool::PoolOptions::::new()
+ .connect_lazy("postgres://invalid/invalid")
+ .expect("lazy pool never connects"),
+ );
+ let mut registry = BlobReferenceRegistry::new();
+ registry.register(Arc::new(FilesReferenceSource::new(pool.clone())));
+ registry.register(Arc::new(ChunksReferenceSource::new(pool)));
+ registry
+ }
+
+ /// Golden test — the statement is assembled from the registry, so pin it
+ /// byte-for-byte and read the SQL here rather than deriving it mentally.
+ ///
+ /// Two invariants a future source must not break: the files term carries
+ /// **no** `NOT EXISTS` guard (that guard exists to keep CDC rows out of
+ /// the *chunk* level; applying it here would count nothing), and
+ /// `chunk_hashes` appears nowhere — a manifest citing its own chunks is
+ /// not a referrer of itself.
+ #[tokio::test]
+ async fn manifest_page_statement_is_stable() {
+ let sql = manifest_page_sql(&default_registry());
+ let expected = r#"SELECT
+ m.file_hash AS file_hash,
+ m.ref_count AS ref_count,
+ m.total_size AS total_size,
+ m.chunk_count AS chunk_count,
+ ((SELECT COUNT(*) FROM storage.files cnt_f
+ WHERE cnt_f.blob_hash = m.file_hash))::bigint AS actual_ref_count
+ FROM storage.chunk_manifests m
+ WHERE ($1::text IS NULL OR m.file_hash > $1)
+ ORDER BY m.file_hash
+ LIMIT $2"#;
+ assert_eq!(sql, expected, "manifest page statement changed:\n{sql}");
+ }
+
+ #[tokio::test]
+ async fn chunks_source_contributes_nothing_at_manifest_level() {
+ let sql = manifest_page_sql(&default_registry());
+ assert!(
+ !sql.contains("chunk_hashes"),
+ "a manifest must not count its own chunks as referrers: {sql}"
+ );
+ }
+
+ #[test]
+ #[should_panic(expected = "no manifest-level blob reference source")]
+ fn empty_registry_refuses_to_build_page_statement() {
+ let _ = manifest_page_sql(&BlobReferenceRegistry::new());
+ }
+}
diff --git a/src/infrastructure/services/mod.rs b/src/infrastructure/services/mod.rs
index 7eeab9c5..2fbb00ae 100644
--- a/src/infrastructure/services/mod.rs
+++ b/src/infrastructure/services/mod.rs
@@ -31,6 +31,7 @@ pub mod last_seen_tracker;
pub mod local_blob_backend;
pub mod local_fs_mount_provider;
pub mod login_lockout_service;
+pub mod manifests_consistency_service;
pub mod media_metadata_service;
pub mod mock_email_sender;
pub mod mount_provider_factory;
diff --git a/src/interfaces/api/handlers/admin_handler.rs b/src/interfaces/api/handlers/admin_handler.rs
index e1985f77..43f75068 100644
--- a/src/interfaces/api/handlers/admin_handler.rs
+++ b/src/interfaces/api/handlers/admin_handler.rs
@@ -2575,6 +2575,12 @@ pub async fn list_jobs(State(state): State>) -> impl IntoResponse
/// `deep=true` opts into slow variants — `consistency_batch` fans it
/// out to sub-jobs; `storage_consistency` (when implemented) will
/// re-BLAKE3 each blob for bitrot detection. See `JobRunArgs.deep`.
+///
+/// `repair=true` opts into corrective action on the refcount
+/// consistency tenants (`blobs_consistency`, `manifests_consistency`,
+/// and `consistency_batch` which fans out to both). Default `false`
+/// preserves discovery-only. See `JobRunArgs.repair` for the
+/// content-safety and race-safety guarantees.
#[derive(serde::Deserialize)]
pub struct TriggerJobQuery {
#[serde(default)]
@@ -2591,6 +2597,8 @@ pub struct TriggerJobQuery {
/// `AppConfig.storage_entries`.
#[serde(default)]
pub storage: Option,
+ #[serde(default)]
+ pub repair: bool,
}
/// `POST /api/admin/jobs/{name}/trigger` — dispatch one run off-schedule.
@@ -2629,15 +2637,18 @@ pub async fn trigger_job(
job = %name,
force = query.force,
deep = query.deep,
- "👮🏻♂️ Admin triggered job {} (force={}, deep={})",
+ repair = query.repair,
+ "👮🏻♂️ Admin triggered job {} (force={}, deep={}, repair={})",
name,
query.force,
query.deep,
+ query.repair,
);
let args = JobRunArgs {
force: query.force,
deep: query.deep,
storage: query.storage.clone(),
+ repair: query.repair,
};
// Jobs that can run for hours (backend_migration, future
diff --git a/tests/api/admin_jobs.hurl b/tests/api/admin_jobs.hurl
index 13915939..92464c9a 100644
--- a/tests/api/admin_jobs.hurl
+++ b/tests/api/admin_jobs.hurl
@@ -225,9 +225,10 @@ jsonpath "$.outcome.count" exists
# Step 4c — Trigger `consistency_batch`. Coordinator (plain
# JobHandler) — snapshots the registry, filters names
# ending `_consistency`, sequentially triggers each.
-# `outcome.count` = number of children dispatched (5 as
-# of Slice 10: drives + folders + files + blobs +
-# backend). `extra.per_check` carries a per-child outcome
+# `outcome.count` = number of children dispatched (6 as
+# of the refcount_cascade fix: drives + folders +
+# files + blobs + manifests + backend). `extra.per_check`
+# carries a per-child outcome
# map. Batch itself always returns ok — child failures
# live inside per_check. `?deep=true` propagates as
# `extra.deep`.
@@ -239,16 +240,21 @@ HTTP 200
[Asserts]
jsonpath "$.ok" == true
jsonpath "$.outcome.outcome" == "ok"
-jsonpath "$.outcome.count" == 5
+jsonpath "$.outcome.count" == 6
jsonpath "$.outcome.extra.deep" == true
-jsonpath "$.outcome.extra.ok" == 5
+jsonpath "$.outcome.extra.ok" == 6
jsonpath "$.outcome.extra.err" == 0
-# per_check is keyed by child job name.
-jsonpath "$.outcome.extra.per_check.drives_consistency.outcome" == "ok"
-jsonpath "$.outcome.extra.per_check.folders_consistency.outcome" == "ok"
-jsonpath "$.outcome.extra.per_check.files_consistency.outcome" == "ok"
-jsonpath "$.outcome.extra.per_check.blobs_consistency.outcome" == "ok"
-jsonpath "$.outcome.extra.per_check.backend_consistency.outcome" == "ok"
+# per_check is keyed by child job name. `manifests_consistency` was added
+# by the refcount_cascade fix — see docs/plan/derived-blobs.md and
+# `[[bug_dual_refcount_divergence]]` for why the second counter needed
+# its own tenant. Auto-picked by `consistency_batch` via `.ends_with(
+# "_consistency")` (no explicit list in the batch service).
+jsonpath "$.outcome.extra.per_check.drives_consistency.outcome" == "ok"
+jsonpath "$.outcome.extra.per_check.folders_consistency.outcome" == "ok"
+jsonpath "$.outcome.extra.per_check.files_consistency.outcome" == "ok"
+jsonpath "$.outcome.extra.per_check.blobs_consistency.outcome" == "ok"
+jsonpath "$.outcome.extra.per_check.manifests_consistency.outcome" == "ok"
+jsonpath "$.outcome.extra.per_check.backend_consistency.outcome" == "ok"
# ─────────────────────────────────────────────────────────────
diff --git a/tests/api/refcount_cascade.hurl b/tests/api/refcount_cascade.hurl
new file mode 100644
index 00000000..89c00905
--- /dev/null
+++ b/tests/api/refcount_cascade.hurl
@@ -0,0 +1,614 @@
+# =============================================================
+# OxiCloud – Copy-folder ref_count regression
+# =============================================================
+# Regression test for the ref_count drift found on Ed's sandbox
+# (2026-08-22) and traced to the copy-folder path. When a folder
+# is copied, every file inside gets duplicated as a NEW file row
+# pointing at the SAME blob(s) — dedup wins bytes-on-disk, but
+# `storage.blobs.ref_count` MUST bump by the number of new refs.
+# If it doesn't, dedup GC will reap a blob that a live file row
+# still references → dangling reference → user gets 404 on
+# download of the copied file.
+#
+# Covers two paths so the CDC boundary can't hide a regression:
+#
+# 1. Small file (`refcount-cascade-small.txt`) → single legacy whole-
+# file blob. `storage.blobs.ref_count` counted directly on
+# the file's content hash.
+# 2. 2 MB file (`refcount-cascade-cdc.bin`) → FastCDC produces multiple
+# distinct chunks. Whole-file `content_hash` still resolves
+# through the dedup API.
+#
+# Both fixtures are DEDICATED — unique content so ref_count
+# assertions are absolute (== 1, == 2). Do NOT reuse these
+# fixtures in other hurl files or absolute assertions here will
+# flake.
+#
+# Deletion is TWO STEPS in OxiCloud:
+# `DELETE /api/folders/{id}` → moves to trash (ref_count
+# unchanged; children still
+# reference the blob).
+# `DELETE /api/trash/{id}` → permanent purge; NOW
+# ref_count decrements. If it
+# hits 0, blob row is deleted
+# synchronously (`exists=false`).
+# So the test purges trash after every folder-delete step —
+# skipping that would make the assertions wrong regardless of
+# whether the copy-side bug is present.
+#
+# `/api/dedup/check/{hash}` returns `ref_count` only for admin
+# callers (regular users get `null` for anti-enumeration). This
+# suite requires the admin token; setup.hurl seeds it.
+#
+# Run:
+# hurl --variables-file tests/api/test.env --test \
+# tests/api/refcount_cascade.hurl
+# =============================================================
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 1 — Login (admin, required for ref_count in dedup API)
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/auth/login
+Content-Type: application/json
+{
+ "username": "{{username}}",
+ "password": "{{password}}"
+}
+
+HTTP 200
+[Captures]
+token: jsonpath "$.access_token"
+
+
+# ─────────────────────────────────────────────────────────────
+# Step 2 — Baseline sweeps: run BOTH consistency tenants that
+# check ref_count invariants and capture their finding
+# counts. Later checkpoints assert equality with these
+# baselines instead of `== 0` — so stale findings from
+# previous tests don't flunk this one; only NEW drift
+# introduced by our copy/delete does.
+#
+# Two tenants because there are two counters (see
+# `[[bug_dual_refcount_divergence]]`):
+#
+# - `blobs_consistency` — reconciles
+# `storage.blobs.ref_count` against its auditor
+# formula. Catches chunk-level drift.
+# - `manifests_consistency` — reconciles
+# `storage.chunk_manifests.ref_count` against its
+# auditor formula. Catches whole-file drift (the
+# path the FE + `/api/dedup/check` surface reads).
+#
+# Trigger returns `outcome.count = stats.finding_count`
+# for recoverable tenants (see
+# `scheduler/recoverable.rs::JobOutcome::ok_with` in
+# the Completed branch). `outcome.outcome == "ok"`
+# means the run walked the whole subject; it does NOT
+# mean zero findings.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Captures]
+baseline_blobs_findings: jsonpath "$.outcome.count"
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+
+
+POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Captures]
+baseline_manifests_findings: jsonpath "$.outcome.count"
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+
+
+# =============================================================
+# Scenario A — small file (single legacy whole-file blob)
+# =============================================================
+
+# ─────────────────────────────────────────────────────────────
+# A1 — Create source folder
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/folders
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+ "name": "hurl-ref-source-small"
+}
+
+HTTP 201
+[Captures]
+src_small_id: jsonpath "$.id"
+
+
+# ─────────────────────────────────────────────────────────────
+# A2 — Upload the small fixture. Content is unique to this
+# test, so ref_count starts at exactly 1 (no dedup
+# collision with any other fixture in the suite).
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/files/upload
+Authorization: Bearer {{token}}
+[MultipartFormData]
+folder_id: {{src_small_id}}
+file: file,fixtures/refcount-cascade-small.txt; text/plain
+
+HTTP 201
+[Captures]
+small_file_id: jsonpath "$.id"
+[Asserts]
+jsonpath "$.content_hash" == "2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3"
+
+
+# ─────────────────────────────────────────────────────────────
+# A3 — Baseline: exactly one live reference.
+# ─────────────────────────────────────────────────────────────
+GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 1
+
+
+# ─────────────────────────────────────────────────────────────
+# A4 — Create target folder + copy source into it. The batch
+# endpoint is the single entry point every FE / WebDAV
+# code path funnels through.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/folders
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+ "name": "hurl-ref-target-small"
+}
+
+HTTP 201
+[Captures]
+tgt_small_id: jsonpath "$.id"
+
+POST {{base_url}}/api/batch/folders/copy
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+ "folder_ids": ["{{src_small_id}}"],
+ "target_folder_id": "{{tgt_small_id}}"
+}
+
+HTTP 200
+[Captures]
+copy_small_root: jsonpath "$.successful[0].new_root_folder_id"
+[Asserts]
+jsonpath "$.stats.successful" == 1
+jsonpath "$.stats.failed" == 0
+jsonpath "$.successful[0].files_copied" == 1
+
+
+# ─────────────────────────────────────────────────────────────
+# A5 — COPY-SIDE ASSERTION: ref_count must be exactly 2. The
+# pre-fix bug left this at 1 — a subsequent GC would then
+# reap the blob out from under the copied file.
+# ─────────────────────────────────────────────────────────────
+GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 2
+
+
+# ─────────────────────────────────────────────────────────────
+# A5b — Sweep both consistency tenants that check ref_count.
+# Complements the single-hash probe above: if the copy
+# path miscounted some OTHER blob or manifest, the single-
+# hash probe wouldn't catch it. Delta vs the baselines
+# isolates NEW drift from ambient.
+#
+# Both tenants required — one counter each; see
+# [[bug_dual_refcount_divergence]] for why the copy path
+# must maintain both symmetrically.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
+
+
+POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
+
+
+# ─────────────────────────────────────────────────────────────
+# A6 — Soft-delete the copied folder tree (moves to trash).
+# ref_count is EXPECTED to stay at 2 — trashed files
+# still reference the blob per the `NOT is_trashed` gap
+# the auditor deliberately closed (see
+# `[[project_by_hash_drop_is_trashed_filter]]`). Asserting
+# == 2 here makes the trash-vs-permanent boundary explicit
+# so a future refactor that changed the semantics would
+# surface at THIS line, not several steps downstream.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/folders/{{copy_small_root}}
+Authorization: Bearer {{token}}
+
+HTTP 204
+
+GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 2
+
+
+# ─────────────────────────────────────────────────────────────
+# A7 — Purge the copy folder permanently.
+# `DELETE /api/trash/{id}` accepts the ORIGINAL resource
+# id as the path param (verified in `trash_handler.rs::
+# delete_permanently`) — no need to GET+filter the trash
+# listing to translate. If soft-delete silently failed,
+# the ref_count assertion two lines below catches it.
+# ref_count must drop to exactly 1 (the source folder
+# still holds its file). Guards the DECREMENT half of
+# the invariant: a double-decrement here would go to 0
+# and the next GC wipes the still-live original's blob.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/trash/{{copy_small_root}}
+Authorization: Bearer {{token}}
+
+HTTP 200
+
+
+GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 1
+
+
+# ─────────────────────────────────────────────────────────────
+# A8 — Soft-delete + purge the SOURCE folder (last holder).
+# ref_count hits 0 → blob row deleted synchronously →
+# `exists == false` on the next probe. Mirror pattern to
+# `dedup_blob_cleanup.hurl` step 10, applied to folder-
+# scoped delete.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/folders/{{src_small_id}}
+Authorization: Bearer {{token}}
+
+HTTP 204
+
+
+DELETE {{base_url}}/api/trash/{{src_small_id}}
+Authorization: Bearer {{token}}
+
+HTTP 200
+
+
+GET {{base_url}}/api/dedup/check/2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == false
+
+
+# ─────────────────────────────────────────────────────────────
+# A9 — End-of-Scenario-A sweep on BOTH tenants. Scenario A
+# introduced two file rows (source + copy), then deleted
+# both. Net effect on the DB is zero — so the drift count
+# on each counter must be exactly the baseline.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
+
+
+POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
+
+
+# =============================================================
+# Scenario B — 2 MB multi-chunk file (CDC manifest path)
+#
+# Coverage: this scenario now sweeps BOTH `blobs_consistency`
+# (chunk-level ref_counts on `storage.blobs.ref_count`) and
+# `manifests_consistency` (whole-file ref_counts on
+# `storage.chunk_manifests.ref_count`). Same-shape assertions
+# as Scenario A — see the baseline capture block near the top
+# of this file and [[bug_dual_refcount_divergence]] for why
+# both are needed.
+# =============================================================
+
+# ─────────────────────────────────────────────────────────────
+# B1 — Source folder for the multi-chunk scenario. Isolated
+# from Scenario A so deletion order can't mask a bug (e.g.
+# a shared blob whose counter goes negative).
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/folders
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+ "name": "hurl-ref-source-cdc"
+}
+
+HTTP 201
+[Captures]
+src_cdc_id: jsonpath "$.id"
+
+
+# ─────────────────────────────────────────────────────────────
+# B2 — Upload the 2 MB dedicated fixture. Deterministic per-
+# position content → FastCDC produces multiple distinct
+# chunks (no chunk-level dedup with any other fixture).
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/files/upload
+Authorization: Bearer {{token}}
+[MultipartFormData]
+folder_id: {{src_cdc_id}}
+file: file,fixtures/refcount-cascade-cdc.bin; application/octet-stream
+
+HTTP 201
+[Captures]
+cdc_file_id: jsonpath "$.id"
+[Asserts]
+jsonpath "$.content_hash" == "fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83"
+
+
+# ─────────────────────────────────────────────────────────────
+# B3 — Baseline. If exists=false here, the whole-file hash
+# isn't registered in `storage.blobs` for CDC uploads on
+# this build — swap to a chunk-hash probe or a
+# blobs_consistency-driven assertion. See
+# `docs/plan/recovery.md`.
+# ─────────────────────────────────────────────────────────────
+GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 1
+
+
+# ─────────────────────────────────────────────────────────────
+# B4 — Target folder + folder copy.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/folders
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+ "name": "hurl-ref-target-cdc"
+}
+
+HTTP 201
+[Captures]
+tgt_cdc_id: jsonpath "$.id"
+
+POST {{base_url}}/api/batch/folders/copy
+Authorization: Bearer {{token}}
+Content-Type: application/json
+{
+ "folder_ids": ["{{src_cdc_id}}"],
+ "target_folder_id": "{{tgt_cdc_id}}"
+}
+
+HTTP 200
+[Captures]
+copy_cdc_root: jsonpath "$.successful[0].new_root_folder_id"
+[Asserts]
+jsonpath "$.stats.successful" == 1
+jsonpath "$.successful[0].files_copied" == 1
+
+
+# ─────────────────────────────────────────────────────────────
+# B5 — CDC copy-side assertion: ref_count == 2.
+# ─────────────────────────────────────────────────────────────
+GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 2
+
+
+# ─────────────────────────────────────────────────────────────
+# B5b — Full-DB sweep on BOTH tenants after CDC copy. Multi-
+# chunk path exercises the manifest side of the invariant
+# — a bug that skips one chunk out of N would leak that
+# chunk without touching the whole-file assertion above.
+# `blobs_consistency` catches chunk-level drift;
+# `manifests_consistency` catches whole-file drift.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
+
+
+POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
+
+
+# ─────────────────────────────────────────────────────────────
+# B6 — Soft-delete copy: ref_count stays at 2.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/folders/{{copy_cdc_root}}
+Authorization: Bearer {{token}}
+
+HTTP 204
+
+GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 2
+
+
+# ─────────────────────────────────────────────────────────────
+# B7 — Purge copy from trash directly by folder id: ref_count → 1.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/trash/{{copy_cdc_root}}
+Authorization: Bearer {{token}}
+
+HTTP 200
+
+
+GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == true
+jsonpath "$.ref_count" == 1
+
+
+# ─────────────────────────────────────────────────────────────
+# B8 — Soft-delete + purge SOURCE: ref_count → 0, blob purged.
+# ─────────────────────────────────────────────────────────────
+DELETE {{base_url}}/api/folders/{{src_cdc_id}}
+Authorization: Bearer {{token}}
+
+HTTP 204
+
+
+DELETE {{base_url}}/api/trash/{{src_cdc_id}}
+Authorization: Bearer {{token}}
+
+HTTP 200
+
+
+GET {{base_url}}/api/dedup/check/fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.exists" == false
+
+
+# ─────────────────────────────────────────────────────────────
+# B9 — End-of-Scenario-B sweep on BOTH tenants. All Scenario B
+# rows gone; both counter drift counts must be back at
+# baseline.
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/blobs_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_blobs_findings}}
+
+
+POST {{base_url}}/api/admin/jobs/manifests_consistency/trigger
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
+jsonpath "$.outcome.count" == {{baseline_manifests_findings}}
+
+
+# =============================================================
+# Cleanup — soft-delete + purge the two empty target folders so
+# the run leaves nothing behind. Same direct-by-id pattern as
+# above; no trash-listing filter needed.
+# =============================================================
+DELETE {{base_url}}/api/folders/{{tgt_small_id}}
+Authorization: Bearer {{token}}
+HTTP 204
+
+DELETE {{base_url}}/api/trash/{{tgt_small_id}}
+Authorization: Bearer {{token}}
+HTTP 200
+
+
+DELETE {{base_url}}/api/folders/{{tgt_cdc_id}}
+Authorization: Bearer {{token}}
+HTTP 204
+
+DELETE {{base_url}}/api/trash/{{tgt_cdc_id}}
+Authorization: Bearer {{token}}
+HTTP 200
+
+
+# ─────────────────────────────────────────────────────────────
+# Final — force `dedup_gc` synchronously so orphaned manifests +
+# blobs actually get reaped and their Rust blob-lifecycle hooks
+# fire (which is what deletes disk thumbnails / face embeddings
+# / audio tags keyed by the whole-file hash).
+#
+# The trigger `trg_files_decrement_blob_ref` deliberately only
+# adjusts counters (see migration `20261017000000_file_delete_
+# trigger_manifest_aware.sql`) — a SQL trigger can't invoke Rust
+# callbacks. Physical cleanup + hook firing lives in `dedup_gc`
+# Phase 1 (see `dedup_service.rs:2660-2772`), which picks up
+# manifests at ref_count <= 0 and calls
+# `fire_blob_hooks(file_hash)` per reap.
+#
+# Without this trigger the test would technically pass (the
+# ref_count assertions all hold; the `exists == false` checks
+# are user-scoped and don't need the DB row gone), but
+# `storage_cleanup_check.sh` running after us would then find
+# 6 orphan thumbnails on disk and fail the whole api-test run.
+# Making the test self-contained keeps the diagnostic tight —
+# if orphans remain after this trigger, the bug is in GC or
+# hooks, not in our cleanup order.
+#
+# `?force=true` bypasses the orphan-grace window (safe: this
+# test has no concurrent uploader that could race the reap).
+# ─────────────────────────────────────────────────────────────
+POST {{base_url}}/api/admin/jobs/dedup_gc/trigger?force=true
+Authorization: Bearer {{token}}
+
+HTTP 200
+[Asserts]
+jsonpath "$.ok" == true
+jsonpath "$.outcome.outcome" == "ok"
diff --git a/tests/api/refcount_cascade_diag.sh b/tests/api/refcount_cascade_diag.sh
new file mode 100755
index 00000000..df7918df
--- /dev/null
+++ b/tests/api/refcount_cascade_diag.sh
@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+# =============================================================
+# refcount_cascade.hurl — post-failure diagnostic
+# =============================================================
+# When `refcount_cascade.hurl` asserts a specific
+# `ref_count` value and the API returns something else, this
+# script inspects the two DB tables the API surface consults
+# to distinguish which side is broken:
+#
+# 1. `storage.chunk_manifests.ref_count` — queried FIRST by
+# `dedup_service::get_blob_metadata` (`dedup_service.rs`
+# :1543-1552). If a manifest row exists for the hash,
+# the API returns THIS ref_count.
+# 2. `storage.blobs.ref_count` — legacy whole-file fallback,
+# returned only when NO manifest row exists.
+#
+# Small files (< CDC min chunk size) still get a manifest row
+# — one degenerate chunk with `chunk_hashes = [file_hash]` —
+# so BOTH tables carry a ref_count for the same hash. When the
+# copy or purge path only updates one of the two, the counters
+# diverge.
+#
+# The `actual_auditor` column is the source of truth: the
+# `blobs_consistency` auditor formula counting live references
+# from `storage.files` + `chunk_manifests.chunk_hashes[]`
+# (`blobs_consistency_service.rs:395-408`). Both stored values
+# should equal this.
+#
+# Bug interpretation matrix (S=stored, M=manifest, A=auditor):
+#
+# S == M == A → consistent (test bug, unlikely)
+# S < A and M == A → blob decrement over-fires
+# S == A and M > A → manifest decrement missed
+# S > A and M > A → decrement missed both sides
+# S < A and M < A → double-decrement both sides
+# S > A and M == A → increment missed on blob side
+# S == A and M < A → increment missed on manifest
+# (S=0, M=2, A=1) [seen 8/23]→ blob double-decrement +
+# manifest never decremented
+#
+# The 2026-08-22 sandbox drift showed the raw shape
+# (`stored=0, actual=1`) that this diagnostic now separates
+# per-table.
+#
+# Called from `run.sh` immediately on hurl failure — see the
+# dedicated `if ! hurl …; then bash …_diag.sh; fi` block.
+# =============================================================
+
+set -uo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+
+# Same connection string every other test-time script uses.
+export PGPASSWORD=oxicloud_test
+PSQL=(psql -h 127.0.0.1 -p 5433 -U oxicloud_test -d oxicloud_test
+ --set ON_ERROR_STOP=1 --pset pager=off)
+
+SMALL_HASH='2d8eb13178cff0036a22e0c3c42446061e86579f9d59ea73c8343bccc2df0fd3'
+CDC_HASH='fb1e63c28bb792e0f69cd16cd7595989f83c218cf70894e07d1f811ab1dc6f83'
+
+log() { echo "[ref_count-diag] $*"; }
+
+log "─────────────────────────────────────────────────────────"
+log "refcount_cascade.hurl failed — running diagnostic."
+log "Shows blob.ref_count AND manifest.ref_count side by side —"
+log "the API queries manifest first (dedup_service.rs:1543), so"
+log "if the two diverge, the API surface + auditor + on-disk"
+log "state all report different numbers. See script header."
+log "─────────────────────────────────────────────────────────"
+
+# Full picture for each fixture hash — one row per hash.
+# LEFT JOINs so a hash present in only one table still surfaces
+# (the other column comes back NULL, which is itself diagnostic).
+"${PSQL[@]}" <A → manifest decrement missed"
+log " blobA → double-decrement on blob +"
+log " no-decrement on manifest"
+log " (matches the 2026-08-23 case:"
+log " blob=0, manifest=2, actual=1)"
+log " blob>A, manifest=A → blob increment missed"
+log " blob=A, manifest