Step 5, read path — Option 2 of the two shapes discussed: the derived
blob is consulted LAST, after the sidecar, not first.
Read order is now
moka -> ext-{file_id}.jpg -> {blob_hash}.webp on disk -> derived blob
For every thumbnail already on disk the new branch is never reached, so
the database stays off the hot path and a fault in it cannot break a
working gallery. It answers only what disk cannot: a thumbnail rendered
by another instance, or a box whose sidecar was never populated. Legacy
content keeps serving from disk until `derived_import` migrates it.
That inverts the plan's stated order deliberately. Derived-blob-first is
right for the END state, because it is what lets the sidecar be deleted;
sidecar-first is right transitionally, because the risky reordering
should happen after the table has been seen serving real reads. The flip
belongs in the release that removes the sidecar, and the comment at the
branch says so.
The existing precedence is preserved and now documented: the file-keyed
client upload (ext-) is checked BEFORE the content-keyed server render.
That ordering is a security property, not a preference — content-keyed
artifacts are shared across every file with that content, so checking
the file-keyed one first is what keeps one user's uploaded preview from
ever being served for another user's identical file.
Shape notes:
* `find_derived_blob` lands on DedupPort/DedupService as the read
counterpart of `store_derived_blob`, so ThumbnailService needs no pool
field — and therefore ThumbnailService::new, DI and three tests are
untouched.
* It carries `content_type`, which is what will retire the byte-sniffing
in the handlers once reads are table-primary.
* The parameter is `Option<&DedupService>`, concrete rather than
`&dyn DedupPort`: DedupPort uses native `async fn` and so is not
dyn-compatible, and ThumbnailPort is never used as a trait object
(checked) — both handlers hold the concrete Arc. `None` means
sidecar-only, which is exactly today's behaviour and what the abstract
port impl passes.
fmt, clippy --all-features --all-targets, 35 unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5, write path only. Every eagerly-rendered thumbnail is now ALSO
stored through DedupService and recorded in
storage.content_derived_blobs. The sidecar write stays and reads are
untouched, so nothing user-visible changes.
That split is deliberate. This is the first commit in the plan that
changes runtime behaviour on a hot path, so it fills the table while
reads still come from disk: the rows can be inspected against real data
before anything depends on them, and a rollback at any point leaves
working thumbnails. The read path and sidecar removal follow separately.
DedupService::store_derived_blob does the whole contract in one place,
so no caller has to remember the accounting:
* writes the bytes through the normal CDC path, so derived blobs
inherit the backend, encryption, migration and key rotation that
source content already gets;
* records (source_hash, kind, variant) -> blob_hash;
* releases the reference store_from_stream took IF the mapping
already existed. Two instances racing to render the same thumbnail
must leave ref_count at 1, not 2 — otherwise every re-render
inflates it and pins the blob forever.
ThumbnailService deliberately does NOT gain a DedupService field: it
implements BlobLifecycleHook, and holding one would close the cycle
DedupService -> BlobLifecycleService -> hook -> DedupService that the
existing comment warns about. The handle is passed per call instead,
which every eager path already has.
The tier-3 write is best-effort and logged. A failure must not cost the
user a thumbnail that is already on disk and in the moka cache;
`derived_import` sweeps anything missed. The sidecar write keeps its
existing failure behaviour and now `continue`s, so a disk failure no
longer falls through to the cache insert.
Nothing reads these rows yet, so the only observable effect is rows
appearing in the table and the manifest ref_count they hold — which
`manifests_consistency` will now count, since
ContentDerivedReferenceSource was registered in 8d4052e1 before any
writer existed.
fmt, clippy --all-features --all-targets, and 35 unit tests across the
touched modules clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 5 foundation of docs/plan/derived-blobs.md. Creates the mapping
table for server-derived artifacts and registers it as a blob-reference
source — deliberately BEFORE anything writes to it, which is the
ordering the plan requires: dedup_gc's reap predicate has to know the
table exists, or the first sweep after the first thumbnail deletes it.
No writer yet, so this is inert: the table is empty and every added SQL
term counts zero. The point is that the machinery is in place first.
storage.content_derived_blobs maps (source_hash, kind, variant) to the
derived blob_hash. The two hash columns mean different things and the
migration says so at length: source_hash is a DEPENDENT pointer holding
no reference (the file keeps the source alive), while blob_hash is a
reference HOLDER bumping chunk_manifests.ref_count. Counting source_hash
would pin every source Blob for as long as a thumbnail existed.
ContentDerivedReferenceSource contributes at the manifest level only.
A derived artifact's blob_hash names a Blob, never a chunk, and
contributing at the chunk level would double-count — a thumbnail is
almost always single-chunk, so its manifest hash equals its lone chunk's
hash, the same aliasing trap the legacy-files term guards against with
NOT EXISTS. There is a test for the invariant, and the chunk-level
golden test passing UNCHANGED is independent confirmation.
Collapses three definitions of "what references a blob" into one.
Adding the source revealed that DI assembled its own registry while
DedupService::new built a different default, and the two consistency
test helpers built a third — so the golden tests would have pinned SQL
production never runs. There is now a single `built_in_registry(pool)`;
DI reads it back via DedupService::reference_registry() rather than
assembling its own.
The reap-predicate golden test caught the change exactly as designed,
and the new branch landed inside the NOT (...) group ORed with files —
so a manifest is reaped only when NEITHER source references it. A branch
landing outside that group would have inverted the predicate for every
other source; that is why the test pins the whole statement rather than
asserting substrings.
fmt, clippy --all-features --all-targets, and 15 unit tests clean.
fix(migrations): order content_derived_blobs after the refcount fixes
Renames 20261015000000_content_derived_blobs.sql to
20261018000000_content_derived_blobs.sql.
The file was authored before the rebase onto fix/copy_folder_ref_count_issue,
so its version sorted BEFORE migrations that now precede it in history:
20261016000000_copy_folder_tree_manifest_refcount.sql
20261017000000_file_delete_trigger_manifest_aware.sql
20261017000002_repair_existing_refcount_drift.sql
Filename order and commit order disagreeing is the problem, not any
dependency — the table is standalone and creates nothing those
migrations touch. But an installation that has already applied through
…17000002 would then be offered a LOWER unapplied version, which sqlx
either applies out of order or rejects on its version check, and a
fresh install would get an ordering no upgrade path ever produces.
Reproducibility between the two is the whole point of the version
prefix.
Kept as its own commit rather than amending 01d90524, since interactive
rebase isn't available here and rewriting mid-branch while the ref_count
work is still being rebased elsewhere would churn hashes again. Worth
squashing into 01d90524 at merge.
No content change — pure rename, verified nothing references the old
filename.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Follows from the blob/chunk taxonomy already in this section: the job
iterates storage.blobs, which post-CDC holds chunks, so it inherits
whatever that table ends up called.
Two rules attached, because a job name is not an internal identifier —
it appears in POST /api/admin/jobs/<name>/trigger, in
background_runs.job_name, and in whatever dashboards operators built:
* Travel with the schema rename, never ahead of it. A job called
chunks_consistency iterating a table still called storage.blobs is
more confusing than today's mismatch.
* Never recycle `blobs_consistency`. Under the corrected taxonomy the
manifest job IS the blob-level job, so the freed name looks
available — and a name that survives a release while changing
meaning silently breaks admin URLs and orphans run history.
manifests_consistency is unambiguous either way, so exactly one job
gets renamed rather than two swapping.
Also records what is explicitly NOT renamed: the `.blob` on-disk suffix,
where correcting it to `.chunk` would mean renaming every file in every
deployment's blob store — a migration that can fail halfway, for
clarity no consumer benefits from since nothing parses the suffix. And
file.blob_hash, whose semantics are unchanged.
Docs only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
macos-13 runner tier is being phased out by GitHub — queues persistently
exceeded 1 h during v0.9.0-rc1 build. Intel Mac users fall back to
'cargo install --features bundled-assets' from source, Docker
--platform linux/amd64, or a Linux VM.
JS-based GH Actions (checkout, artifact steps, setup-node) can't run
inside Alpine on ARM64 — Node.js binary shipped by the actions
requires glibc, and the x64-Alpine workaround doesn't extend to
arm64. Cross-compile natively via 'rustup target add' + musl-tools
instead.
The 'broken pipe' error under set -euo pipefail was masking the real
answer. Capture status + content-type + first-char in one curl, dump
first 200 bytes of body on failure so we can see what the server
actually served.
Documents OXICLOUD_ENABLE_VIDEO_THUMBNAILS (+ OXICLOUD_FFMPEG_PATH) in
example.env and docs/config/env.md — closes the discoverability gap
where the env var was only visible in Rust docstrings.
Also lands docs/plan/bundled-binary.md — the design record referenced
from code comments in src/cli/mod.rs, src/interfaces/web/embedded.rs,
and the Dockerfile.
Adds the tag-triggered workflow that builds 4 musl-linux + macOS
tarballs and attaches them to the tag's GitHub Release. Ships a
matching install guide (docs/install/binary.md) with SHA256SUMS
verify, systemd unit, upgrade flow, and hardware notes. Adds
[package.metadata.binstall] so 'cargo binstall oxicloud' works
automatically once the first release lands.
Also re-enables incremental compilation in the dev profile — the
'modest single-crate savings' rationale from when the crate was small
has been outgrown; full rebuild ~10 min is now the dev-loop bottleneck.
execute once via SQL migration to fix old entries
next: will be catch by manifests_consistency job, if new are found it means new bug discovered
no auto repair to prevent hidding bugs, operator can still invoke repair while waiting a fix
Silent data loss on an ordinary UI folder copy.
The function bumped only storage.blobs:
UPDATE storage.blobs b SET ref_count = ref_count + hc.cnt
FROM (...) hc WHERE b.hash = hc.blob_hash;
but a CDC file's blob_hash names a MANIFEST, not a chunk. For any
multi-chunk file that predicate matches zero rows, so the copy took no
reference at all. Delete the original afterwards and remove_reference
walks the manifest to 0, dedup_gc reaps the manifest and every chunk
behind it, and the copy is unreadable.
Single-chunk files escaped by accident: their whole-file hash equals
their lone chunk's hash, so the UPDATE did match — bumping the wrong
counter, which surfaces as a manifest under-count plus a blob
over-count rather than as loss. That asymmetry is why the bug survived:
small files, which dominate most test corpora, look fine.
Reproduced through the UI on a 5 MiB / 18-chunk file:
chunk_manifests.ref_count stayed at 1 while two storage.files rows
referenced it, and manifests_consistency reported
manifest_refcount_mismatch with delta 1, reap_risk true.
The fix mirrors DedupService::add_reference — manifest first, blobs only
as fallback, with a NOT EXISTS guard so a single-chunk file is not
counted at both levels (which would turn the under-count into an
over-count). orphaned_at is cleared on the blobs branch, as
add_reference does when resurrecting a blob inside its GC grace window.
Only the reference-counting block changes; the rest of the function is
20260902000001 verbatim.
Existing drift is deliberately NOT repaired here — a schema migration
cannot know which counter is authoritative. manifests_consistency
reports it; repair belongs with the recovery framework.
NOT executed against a database: the test instance was down and the dev
instance is read-only by convention. A parse error would fail at boot,
before any data is touched. Verify by re-running the reproduction — a
folder copy of a >1 MiB file should now leave ref_count at 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reap statement is assembled from the registered reference sources,
so it cannot be grepped out of the source tree — and it DELETES
manifests. Hiding it behind a debug filter an operator has to know to
enable was the wrong default: if what GC considers "referenced" ever
changes, that has to be visible on the next boot without anyone going
looking for it.
Reported in testing: `RUST_LOG=info,oxicloud::dedup=debug` did not
surface it, while a global `RUST_LOG=debug` did — at the cost of an
unusably noisy boot. Rather than have operators carry a special filter
for a line describing a destructive statement, promote it.
The statement is whitespace-collapsed into a single `statement` field
so a multi-line query does not sprawl across the boot log, and the
registered `sources` are logged alongside it — that list is what
actually determines the predicate, so a change to it is the thing worth
noticing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 3 (prerequisite 2) of docs/plan/derived-blobs.md, and the last one
before the thumbnail slice.
There are two reference counters and only one was ever verified.
add_reference bumps chunk_manifests.ref_count first and only falls back
to storage.blobs.ref_count, so a reference lands on whichever counter
its hash names: chunk references feed storage.blobs and are reconciled
by blobs_consistency::refcount_mismatch, while Blob references — every
CDC file, and every derived artifact once those exist — feed
chunk_manifests.ref_count, which nothing reconciled.
That gap was survivable only because dedup_gc's reap predicate carried a
second clause ("no storage.files row references this manifest") that
quietly compensated for drift on the bulk-delete paths where ref_count
is never decremented. Generalising that clause to the reference registry
in 1c8ead49 — so thumbnails stop being reaped — removed the
compensation, which is precisely why the counter now needs checking
directly. The two changes have to ship together.
Adds manifests_consistency, a recoverable job reporting
manifest_refcount_mismatch (severity inconsistent). The finding carries
reap_risk so an operator can triage: an under-count means GC reaps a
manifest whose content is still reachable, taking its chunks with it,
while an over-count merely pins storage.
A separate job rather than a second phase of blobs_consistency: one
subject per job, as the other five consistency tenants do, and it avoids
changing the cursor format of an existing recoverable job — which would
strand any run paused across the deploy.
The page query is assembled from the same registry dedup_gc reaps from
(via DedupService::reference_registry), built once at construction, and
pinned by a golden test. Two invariants the test guards: the files term
carries no NOT EXISTS guard — that guard keeps CDC rows out of the
*chunk* level and here would count nothing — and chunk_hashes appears
nowhere, since a manifest citing its own chunks is not a referrer of
itself.
fmt, clippy --all-features --all-targets, and 3 new unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Completes step 1 of docs/plan/derived-blobs.md. The chunk-level
`actual_ref_count` recompute was two correlated subqueries written
inline; it now sums the registered reference sources instead, so
`blobs_consistency` and `dedup_gc` answer "what references this hash"
from one place. If they ever diverged the sweep would bless counts the
collector disagrees with — and the collector wins, destructively.
No behaviour change: the generated expression is the same legacy-files
term (guarded by NOT EXISTS) plus the same manifests-citing-this-chunk
term, and a golden test pins the whole statement byte-for-byte.
Built once at construction, like the reap statement, so the sweep runs
a fixed query per page rather than assembling SQL inside the loop. The
builder refuses an empty registry rather than emitting a query where
every blob looks unreferenced and the entire table reports
refcount_mismatch; there is a test.
DI now constructs one registry and hands the same instance to both
consumers — `DedupService::reference_registry()` is what
`BlobsConsistencyCheck` receives, so agreement is structural rather
than a convention someone has to maintain.
The long comment explaining the single-chunk double-count trap moved
from the query site to the builder's doc comment, where the NOT EXISTS
guard it describes actually lives.
fmt, clippy --all-features --all-targets and the 17 affected unit tests
all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prerequisite 0 of docs/plan/derived-blobs.md. The zero-ref manifest
sweep read:
WHERE m.ref_count <= 0
OR NOT EXISTS (SELECT 1 FROM storage.files f
WHERE f.blob_hash = m.file_hash)
That OR hardcodes "storage.files is the only thing that can reference a
manifest". A thumbnail manifest held by storage.content_derived_blobs
has ref_count = 1, so the first clause is false — but no files row names
a thumbnail's Blob hash, so NOT EXISTS is true, the OR fires, and the
manifest is deleted, its chunks dereferenced and the bytes reaped on the
next sweep. Landing content_derived_blobs before this fix would destroy
the derived tier on the first GC run.
The second clause is not merely defensive: it is the ONLY reap path for
bulk deletes (user cascade, empty_trash), where the PG trigger touches
storage.blobs but never decrements the manifest and the per-file
cleanup_if_orphaned call is skipped. So the fix has to preserve that
role, not just add tables to the NOT EXISTS. It is now the union of
every registered manifest-level source.
Assembled once, not per sweep. An earlier cut of this change put a
format! inside the DELETE, which made the most dangerous statement in
the file unreadable, un-pasteable into psql, and injection-shaped even
though every input is &'static str. The statement is now built at
construction and stored on DedupService, so:
* the reap loop runs a fixed statement with no string work,
* the SQL string is stable, so prepared-statement cache keys are too,
* a golden test pins it byte-for-byte — a reviewer reads the SQL in
the test rather than mentally evaluating the registry,
* initialize() logs it at debug with the contributing source names,
recovering the "paste it into psql" property the literal had.
The registry is mandatory rather than Option. An empty registry makes
"nothing references it" vacuously true for every row, so the builder
panics instead of emitting a statement that would delete every manifest
in the database; DedupService::new always registers the two built-in
sources, so that panic is unreachable by construction. There is a test
for it.
Adds ref_exists_sql to the port, defaulting to (count) > 0 and
overridden by FilesReferenceSource with a real EXISTS. Without it the
reap predicate would have traded today's short-circuiting NOT EXISTS
for a COUNT(*) = 0 that scans every referrer — a regression precisely
on heavily-deduplicated blobs, which is what GC walks most.
fmt, clippy --all-features --all-targets, and the 11 affected unit
tests all clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 1 of docs/plan/derived-blobs.md. Makes "who references this blob
hash" an extension point instead of SQL hardcoded in two places
(dedup_gc's reap predicate and blobs_consistency's refcount recompute,
both naming storage.files and storage.chunk_manifests directly). Adding
a blob-owning table without teaching those two risks silent orphaning:
GC sees ref_count = 0 and reaps live content.
Behaviour is unchanged — this commit only introduces the port and the
two sources that reproduce today's SQL. Wiring follows.
Two levels, not one. add_reference bumps chunk_manifests.ref_count first
and only falls back to storage.blobs.ref_count, so a reference lands on
whichever counter its hash names and the two must be recomputed
separately. RefLevel is a parameter rather than a property of a source,
because storage.files legitimately contributes at both: a manifest-less
legacy row references a chunk, a CDC row references a Blob. The
NOT EXISTS guard on the chunk-level files term is load-bearing — for a
single-chunk file the whole-file hash equals its lone chunk's hash, so
without it the row is counted at both levels.
SQL fragments rather than a per-hash count. blobs_consistency recomputes
with one query per page, the expected count inlined as correlated
subqueries; asking each source for a count per hash would turn that into
sources x rows round-trips. So sources emit a fragment the registry sums
into the existing page query, and count_references exists only for the
on-demand path where the candidate set is already filtered to
ref_count = 0.
Fragments use their own aliases (cnt_f, cnt_m) rather than the sweeps'
outer-row aliases (b, m). A fragment reusing `m` would shadow the outer
alias in the manifest sweep and silently correlate against itself;
there is a test for it.
The SQL builders are free functions so the shape can be asserted without
constructing a pool — sqlx's connect_lazy still needs a Tokio context,
and the fragments are pure string assembly anyway.
9 unit tests. fmt, clippy --all-features --all-targets, build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
derived-blobs.md — consolidates several review rounds.
BLOCKER found while building the coverage matrix: the zero-ref manifest
sweep in dedup_gc (dedup_service.rs:2574) deletes a manifest when
`ref_count <= 0 OR NOT EXISTS (SELECT 1 FROM storage.files ...)`. That
OR hardcodes "storage.files is the only thing that can reference a
manifest", so a thumbnail manifest held only by content_derived_blobs
is deleted on the next GC run, its chunks dereferenced and the bytes
reaped. Promoted to prerequisite 0 and delivery step 2. It is also the
missing half of the unreconciled chunk_manifests.ref_count: that OR is
the hack that made the drift survivable.
Adds the 13-edge consistency coverage matrix (rows 1-6 and 13 covered,
7-11 not), and records that backend_consistency needs NO change — the
backend holds chunks, which neither new table references.
BlobReferenceSource correction: `ref_level()` was wrong because
FilesReferenceSource spans both levels (chunk for legacy manifest-less
rows, Blob for CDC rows). Replaced with
`ref_count_sql(level, alias) -> Option<String>`.
Also: migration of existing sidecar content; schema trim to the columns
nothing else owns (no size/format/codec/renderer; content_type kept as
non-key since it removes today's byte-sniffing); ext-{file_id}.jpg
corrected — the client generator ships and covers PDF, which has no
server-side rasteriser, so file_attached_blobs is required rather than
deferred; uploaded_by on the shares NOT NULL/no-FK convention; the
mermaid relation map; copy and version semantics with the
copy_file_satellites consolidation; the storage.files vs file_metadata
table-identity fix; DedupService -> BlobHandler recorded as decided.
NEW hidden-system.md — retires auth.users.image TEXT (inline base64
avatar, up to 512 KiB, already worked around with a narrow projection
after it was measured detoasting M avatars per group fan-out) in favour
of *_file_id pointers at ordinary storage.files rows in one shared
hidden system drive. Because storage.files is already a
BlobReferenceSource, a file pointer costs zero new reference sources
and zero new consistency edges. Records why the alternatives lose,
the drive's required properties (hidden at enumeration, trash off,
quota exempt, boot fail-fast), per-kind visibility in code, the
secrets exclusion rule, the avatar migration, and the future object
catalogue.
Docs only; no code or schema changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Enrich OxiCloud to maximise the use of `dedup` Engine
2 cases will be covered:
- blobs issues from other blobs (thumbnail automatic generation from blob)
- by filename (ex: thumbnail uploaded from users)
A local cache will be added when blobs are remote (S3 or similar)
parse_ical_datetime rejected any datetime without the trailing 'Z',
so events created without a timezone in calendar apps — which DAVx5
syncs as floating time per RFC 5545 3.3.5 form 2 — failed with
'Invalid DTSTART: Invalid datetime format: expected YYYYMMDDTHHMMSSZ'
and HTTP 400, breaking the whole event upload.
Accept the 15-char floating form and interpret the wall-clock time as
UTC. TZID-anchored forms remain unsupported until VTIMEZONE handling
lands.
Fixes#682
Edouard Vanbelle