feat(consistency): derived_consistency — the last coverage-matrix gap

Finds derived mappings whose Blob is gone on either side. Nothing else
can, and that is the point rather than an oversight: every other job
reasons from a Blob outwards, so a row whose SOURCE was reaped breaks
none of their invariants — valid reference, exactly correct refcount,
bytes present on the backend. Every check agrees the system is healthy
while the artifact is pinned forever. A leak that looks like
correctness, which is why it took four suite runs to name.

Two findings:

  derived_orphan_mapping (inconsistent) — source_hash has neither a
  manifest nor a blob row, so purge_derived_blobs can never fire for it.
  Storage that grows and never reclaims.

  derived_dangling_blob (data_loss) — blob_hash has no Blob behind it.
  The mapping promises an artifact that is gone, so a read finds the row
  and then fails.

Existence means EITHER table on both sides, since source_hash and
blob_hash each name a Blob: a manifest for CDC content, a bare blob row
for legacy whole-file content. Checking one would report every legacy
blob as missing.

Paged on the full primary key with a row-value comparison rather than
source_hash alone — a source has several variants, so a page boundary
can fall inside one and advancing by source would skip the rest. Both
existence probes fold into the page query, so a page is one round-trip
rather than 2xN. Cursor round-trip is tested, including that a malformed
one fails loudly: silently restarting would make a paged audit
under-report, which is the worst failure available to a job whose
purpose is finding what is missing.

e4c78ae0 stops new orphans at the write side; this finds the ones
already on disk, which that fix cannot reach. Added to the end-of-suite
sweep so it runs against real state every time.
This commit is contained in:
Edouard Vanbelle
2026-08-28 19:33:02 +02:00
parent de0f625d4c
commit 4fef34b230
4 changed files with 348 additions and 0 deletions
+6
View File
@@ -532,6 +532,12 @@ CONSISTENCY_JOBS=(
blobs_consistency
manifests_consistency
backend_consistency
# Catches what the others structurally cannot: a derived mapping whose
# source Blob is gone looks healthy to every refcount-based check — valid
# reference, correct count, bytes present — while pinning its artifact
# forever. That leak reached this suite as three unreclaimable blobs and
# took four runs to identify.
derived_consistency
)
CONSISTENCY_FAILED=0