The thumbnail ETag was "thumb-{file_id}-{size}-{format}", sent with
Cache-Control: public, max-age=31536000, immutable. Replacing a file's
content preserves its id — file_upload_service rebuilds the entity with
parts.id and a new hash, then fires on_file_updated, which deletes and
regenerates the thumbnails — so the server produced a new thumbnail while
still advertising the old ETag. Because `immutable` tells a conforming
browser not to revalidate at all inside the freshness window, clients kept
rendering the previous image for up to a year, unfixably.
Keyed on the content hash the directive becomes honest: a thumbnail is a
pure function of (source bytes, size, format), so that triple identifies
the response. New content yields a new ETag.
The same change fixes the opposite direction. A copy, or any dedup twin,
had a different id and therefore a different ETag, so clients refetched
bytes they already held even though both are served from the same derived
blob. Now identical content agrees on an ETag and revalidates to 304
across files, users and copies.
Both thumbnail endpoints were affected: the REST handler and the
NextCloud preview handler.
Cost is one PK lookup ahead of the 304 decision, where the id-keyed
version needed none — paid for by no longer serving stale images. It is
partly recovered: both handlers already resolved the same hash further
down for the render path, and that second lookup is now gone, so the
cache-miss path is unchanged and only the 304 path pays. The resolved
hash is also handed to get_cached_thumbnail instead of None, saving the
service its own lookup.
No new disclosure: content_hash is already on FileDto and returned by
GET /api/files/{id}.
Tests: thumbnail_etag_content_keyed.hurl covers invalidation — overwrite
in place via WebDAV PUT, assert the ETag changed, assert a client holding
the stale one gets 200 rather than 304. derived_blob_copy.hurl gains the
sharing direction: a copy answers with the SAME ETag and revalidates to
304, which is the one externally observable consequence of content-keying
and was not previously testable.
The byte-identity assertions were documented as proving that a copy
shares the original's content_derived_blobs row. They prove no such
thing: rendering is deterministic in the source bytes and the variant,
so a copy that re-rendered from scratch returns identical bytes. The
copy is in fact a moka hit — that cache is keyed on
(source_hash, size, format), which the copy shares — so it never
reaches the derived tier here at all.
Nor is there an assertion that would fix it. Duplication is impossible
by construction: the PK is (source_hash, kind, variant), a copy carries
the same source_hash, and store_derived_blob is ON CONFLICT DO NOTHING.
The schema enforces the property, so no runtime behaviour can violate
it and there is nothing to catch.
Same limitation narrows step 11: it proves the SOURCE content survived
GC, not the derived blob — a reaped derived blob is re-rendered
transparently from the live source.
What the file does prove is unchanged and is the part that was broken:
both copy paths take a real blob reference (ref_count 1 -> 2 -> 3), and
purging the original does not destroy the copies. No assertions changed.
Step 8 of docs/plan/derived-blobs.md. "What follows a file on copy" was
written twice — the copy_file CTE and storage.copy_folder_tree — and had
already drifted: the tree path bumped storage.blobs only, missing
manifests, which was silent data loss on any multi-chunk file. Fixing it
meant writing the same logic a second time. Step 9 adds a file-keyed
satellite table, which would mean a third and fourth.
Two SQL functions:
storage.add_blob_references(TEXT[]) — the manifest-first reference
contract for SQL callers, returning hashes that matched no registry
row. Set-based so the tree path keeps its single-statement cost; a
per-row helper would have made a 10k-file copy 10k calls.
storage.copy_file_satellites(UUID[], UUID[]) — dead properties plus
the blob reference. The body is the copy-semantics declaration: what
is absent (comments, favorites, content-keyed derived rows) is listed
with its reason, so the taxonomy is executable rather than documented
elsewhere and drifting.
Both copy paths now call it. The single-file path becomes a real
transaction, which also fixes the reference being best-effort: a failed
add_reference used to log a warning and leave a copy holding no
reference at all — the exact shape that gets its content reaped. It
cannot be a CTE arm, because data-modifying CTEs share one snapshot and
the function must read the row the INSERT just wrote.
Verified against a scratch PG with all migrations applied: multi-chunk
manifest 1→2, single-chunk alias bumped at manifest level only (the
NOT EXISTS guard), chunks behind a manifest untouched, dead properties
duplicated, length mismatch rejected, repeats counted.
tests/api/derived_blob_copy.hurl covers it end-to-end and answers the
question the copy raises: content_derived_blobs is NOT copied. A copy
carries the same blob_hash, so it resolves the same derived row — the
test asserts byte-identical thumbnails from both copy paths, then
deletes the original, runs GC, and requires both copies to still serve.
That last step only passes if the references are real.
One dedup_gc pass cannot fully drain now that thumbnails are derived
blobs. Reaping a source releases the references its derived artifacts
hold (each content_derived_blobs row pins a manifest), and those
releases happen mid-sweep — the derived chunks are stamped orphaned as
the pass is already walking past them, because
remove_manifest_reference deliberately does not unlink, to avoid racing
a concurrent upload re-referencing the same chunk. They are collectible
only on the NEXT sweep, which is why the check saw 15 leftover blobs.
Loops until a pass reclaims nothing rather than hardcoding two. Two is
correct only while the derivation graph is one level deep — a thumbnail
is derived from a file, nothing is derived from a thumbnail. That is a
property of the data, not an invariant the code enforces, so a fixed
count would silently under-drain the day transcodes-of-thumbnails or
E2E-wrapped derivatives exist, and the failure would surface as a
confusing leftover-file assertion rather than the design change it is.
Bounded at 3 with a warning if it does not settle.
Sleeps between passes. The JobRegistry serialises runs of the same job,
so a back-to-back trigger risks rejection as already-running — which
returns 0 reaped and would exit the loop early, declaring success with
blobs still on disk. A false pass is worse than a slow one. It also
gives the previous pass's detached unlink tasks (spawned by
on_blob_deleted, awaited by nothing) time to land.
Deliberately NOT fixed in production code: derived chunks land inside
the 1-hour orphan grace, so a second immediate sweep would collect
nothing there and the next scheduled run picks them up. A fixpoint loop
in garbage_collect would be dead code outside force=true, which is only
this test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- POST /api/auth/opaque/login/ke1 (public) — takes {userIdentifier, startLoginRequest}, resolves the identifier via the same @-dispatch as legacy login (AuthApplicationService::lookup_user_for_login, factored out), fetches envelope, runs ServerLogin::start (real branch for known users, dummy branch for anti-enum on unknown/unregistered), stashes state under a random exchange_id in the moka cache, returns {exchangeId, loginResponse}.
- POST /api/auth/opaque/login/ke3 (public) — atomic take from the cache FIRST (anti-enum + anti-replay), then decodes the payload, runs ServerLogin::finish, stamps opaque_migrated_at (Phase 3 signal), and mints a session via the new AuthApplicationService::mint_session_for_authenticated_user helper — returns the same AuthResponseDto shape as legacy login so the SPA has one downstream handler.
- Session mint factored: mint_session_for_authenticated_user(User) extracted from login() so both the legacy password path and OPAQUE KE3 converge through one implementation.
- OpaqueRepositoryPort::mark_migrated with COALESCE-preserving idempotent stamp of opaque_migrated_at.
- opaque-setup CLI + Dockerfile wiring already shipped (Step 0 hygiene).
- Routing fix: sub-prefix split (/api/auth/opaque/register vs /api/auth/opaque/login) — axum composes middleware between sibling nests at the same prefix, which was cross-applying auth+CSRF to my public login routes. Distinct prefixes side-step that cleanly. Documented in both main.rs and the router builder doc.
- Rate-limit sharing: login KE1/KE3 layered with the SAME login_limiter instance as legacy POST /api/auth/login, so an attacker can't halve the per-IP budget by spraying both endpoints.
Anti-enum + anti-replay hardening in KE3: take runs BEFORE payload parse so:
- Unknown / expired / already-consumed exchange_id → 401 InvalidCredentials (same shape as wrong-passphrase, no payload-shape leak)
- Consumed handle can't be re-used to spam parse attempts
OPAQUE (RFC 9807) implementation (using `opaque-ke` crate)
with opaque authentfication, server will never receive the password (in the auth=password mode)
this is a must have to create trust with users to permit end to end encryption in the future
(we cannot know if user use the same password/passphrase for his asymetric key or his oxicloud auth,
this is why server must never have the password)
pass1: prepare server
a job can be paused/resumed
a job as an exclusibity by it's name
if you want to run another job with same name:
either cancel the first one, or wait of it's terminaison
pause does not permit to run the other job, this can create
race conditions
add /api/folders/{id}/ancestors
this API to iterate parent up to the drive root or the shared folder
this will help UI to build the breadcrumb in 1 API call
and to identify the root element (is it a drive users has access to or
a shared folder ?)
ui: now only 1 API call is now required to build the breadcrumb
Close the two coverage gaps in the external-mounts feature — the admin
REST surface and the admin UI tab both previously had no automated tests.
Backend (hurl):
- tests/api/external_mounts.hurl — black-box coverage of the admin mount
endpoints (list / create / delete): input validation (empty name,
non-existent host path → 400), create → 201 with the full view, the
mount appears in the list, delete → 204 then 404, and the /api/admin
middleware denies a non-admin on every verb → 403.
- server.env: enable OXICLOUD_ENABLE_EXTERNAL_MOUNTS for the test server;
run.sh: register the new scenario.
Frontend (vitest):
- admin/page.test.ts — opening the mounts tab loads + lists mounts,
the create form calls createExternalMount, and delete goes through the
confirm modal to deleteExternalMount.
Tooling:
- devenv.nix: add b3sum (used by tests/api/run.sh's storage-integrity check).
- provide is_shared and is_favorite information in DTO, information propagated as badge/buttons per items
regarding performances I try to be minimalis on SQL to prevent any perf regression
- remove old set of sharedids and favoriteids (was not functionnal anymore)
- fix date picker (no past selection) in grant
- fix contextmenu close on /files section
Adds application/adapters/param_encoding.rs: encode/decode for the three
RFC 6868 caret-escape sequences (^n newline, ^^ literal caret, ^' literal
DQUOTE) plus render_param_value, which quotes and encodes a parameter
value only when the vCard/iCalendar param-value grammar requires it
(comma/semicolon/colon triggers quoting; an embedded DQUOTE/caret/newline
needs escaping even once quoted).
Wires render_param_value into the two vCard generators' TYPE parameter
emission (EMAIL/TEL/ADR) in carddav_adapter.rs::contact_to_vcard and
contact_service.rs::generate_vcard — contact `type` fields are free text,
not a fixed enum, so a value containing a comma or quote previously
produced outright broken vCard syntax (no quoting/escaping was applied
at all).
Scope note: this branch only fixes the generation side. The existing
vCard parser is a naive line-by-line substring scanner (`line.contains
("TYPE=WORK")`) with no real per-parameter tokenizing, so there's no
integration point yet for the decode half without a larger parser
rewrite — that's part of the vCard 4.0 work (RFC 6350), which depends on
this branch for the shared encoding utility.
webdav_patch.hurl and nc_webdav_patch.hurl existed with real coverage
since the original PATCH commits but were never added to
tests/api/run.sh, so just api-test/CI silently skipped them. Wire both
in, fix nc_webdav_patch.hurl's header-after-[BasicAuth] ordering bug
that meant it had never actually passed, and add two new
consistency-focused files chaining PATCH operations with
cross-protocol/cross-surface verification:
- webdav_patch_consistency.hurl: chained overwrites with ETag-change
checks, GET/HEAD/PROPFIND cross-protocol agreement, quota-507
leaving the file byte-for-byte unchanged, direct_put_max_bytes
prefix/suffix regression coverage.
- nc_webdav_patch_consistency.hurl: Editor/Viewer/Outsider permission
matrix, cross-surface lock interop, quota-507 via the NC surface.
Running these surfaced two real bugs in the NC PATCH handler, both
fixed here:
- The write step mapped every error (including a legitimate anti-enum
permission denial) to a raw 500 instead of AppError::from(e), unlike
the plain surface. A Viewer without Update permission got a 500
leak instead of the expected 404.
- nc_to_internal_path() didn't strip the leading '/' that chroot.path
carries from StoragePath::to_string(), so a LOCK taken via /webdav/
silently failed to block PATCH via /remote.php/dav/ on the same
file — the exact-string lock-store lookup never matched. Added a
regression unit test.
Edouard Vanbelle