fix(storage): skip sidecars whose source is gone, before writing anything

Running the import on a real install produced a store-then-discard loop:
NEW BLOB (CDC) immediately followed by MANIFEST DELETED, once per
sidecar. store_derived_blob wrote the bytes, the source-exists guard
refused the row, and `inserted == 0` released the reference again.

The refusal is right — `.thumbnails/` outlives years of deleted files,
and importing those would recreate exactly the orphan rows e4c78ae0
eliminated. The mistake was deciding it AFTER the write.

Now checked before the read and the store, via blob_exists (manifest
first, blob as fallback). Two costs it removes: a blob write plus a
manifest delete per dead sidecar on EVERY run, and a tail that never
empties — unimportable files are rediscovered forever, so the job never
reports zero and step 10e's gate never opens.

Reported as `sidecar_source_gone` so the scale is visible before
anything is removed, and deleted under `repair`. That is the one unlink
in this job needing no readback: there is nothing to read back and
nothing to regenerate from.

Counted separately in the completion log, because "skipped, source gone"
and "already present" mean different things to an operator deciding
whether the migration has converged.

Worth noting for anyone reading the raw logs: NEW BLOB names the hash of
the STORED BYTES, while the sidecar filename is the SOURCE hash. They
are different values, so grepping the log hash against .thumbnails finds
nothing. The new finding carries both.
This commit is contained in:
Edouard Vanbelle
2026-08-28 22:52:21 +02:00
parent b3221e265d
commit 1a3d7d201a
@@ -236,6 +236,7 @@ impl RecoverableJobHandler for ThumbDerivedImport {
let mut failed = 0u64; let mut failed = 0u64;
let mut deleted = 0u64; let mut deleted = 0u64;
let mut unverified = 0u64; let mut unverified = 0u64;
let mut dead_source = 0u64;
let mut since_checkpoint = 0usize; let mut since_checkpoint = 0usize;
// The DIRECTORY is `{size}` on disk; the VARIANT is `{size}.{ext}` // The DIRECTORY is `{size}` on disk; the VARIANT is `{size}.{ext}`
// since migration `20261022000000`. Conflating them is a real trap: // since migration `20261022000000`. Conflating them is a real trap:
@@ -317,6 +318,50 @@ impl RecoverableJobHandler for ThumbDerivedImport {
.await; .await;
} }
} }
} else if !self.dedup.blob_exists(hash).await {
// The source is gone, so this sidecar cannot be imported:
// a mapping to a dead source is precisely the orphan row
// `store_derived_blob` now refuses, because nothing would
// ever reap that hash again and the row would pin its
// artifact forever.
//
// Checked BEFORE the read and the blob write, not after.
// Without this the refusal still happens, but only once
// the bytes have been stored — so every run writes a blob
// and immediately deletes its manifest again, per dead
// sidecar, forever. On a real install where `.thumbnails/`
// has outlived years of deleted files, that is most of
// them.
//
// It also matters for the tail: these files are
// unimportable by definition, so a run that keeps
// rediscovering them never reports zero and step 10e's
// gate never opens. Under `repair` they are deleted —
// safe, and the only unlink here that needs no readback,
// since there is nothing to read back and nothing to
// regenerate from.
dead_source += 1;
if delete_imported {
let path = self.thumbnails_root.join(dir_name).join(&name);
if fs::remove_file(&path).await.is_ok() {
deleted += 1;
}
} else {
record_or_log(
store,
THUMB_DERIVED_IMPORT_JOB_NAME,
"sidecar_source_gone",
"anomaly",
None,
serde_json::json!({
"path": position,
"source_hash": hash,
"note": "source Blob no longer exists; the thumbnail is \
unimportable and is deleted on a repair run",
}),
)
.await;
}
} else { } else {
let path = self.thumbnails_root.join(dir_name).join(&name); let path = self.thumbnails_root.join(dir_name).join(&name);
match fs::read(&path).await { match fs::read(&path).await {
@@ -441,8 +486,10 @@ impl RecoverableJobHandler for ThumbDerivedImport {
failed = failed, failed = failed,
deleted = deleted, deleted = deleted,
unverified = unverified, unverified = unverified,
dead_source = dead_source,
"thumb_derived_import: {imported} imported, {already} already present, \ "thumb_derived_import: {imported} imported, {already} already present, \
{failed} failed, {deleted} sidecar(s) deleted, {unverified} kept unverified" {failed} failed, {deleted} sidecar(s) deleted, {unverified} kept unverified, \
{dead_source} skipped (source gone)"
); );
RunOutcome::completed() RunOutcome::completed()