Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):
DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
folders/files UNION-ALL branches as sargable row-value comparisons with
per-branch ORDER/LIMIT + two partial expression indexes
(folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
(19.5x); other sort modes at parity or better. New migration
20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
on the timeline index, joins moved above the top-N. 50k-photo library:
97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
(5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).
Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
concurrent cold readers = 16 full remote downloads racing truncating
writes on ONE deterministic .tmp (corruptible cache); now 1 download
(16x less egress, 2.8x wall on a shared link) and torn files can never
be renamed into the cache.
I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
(OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
nodes, three SDK stacks gone from every build). tokio "process" is now
an explicit feature (was enabled transitively by aws-config).
Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
asserts output identity across locales and a 3x floor.
Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):
DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
of one sequential query per child, and indexable `=` predicates instead
of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
(folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
invalidation on drive mutations) for find_default_for_user and the
markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
[benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
2-query resolution entirely (wants_quota()); the remaining lookups read
2 columns instead of the full auth.users row with its <=512 KiB avatar
(11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
upload quota check. [benches/QUOTA-PATH.md]
CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
(verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
99x less CPU for ~15% more bytes. SPA assets are now precompressed at
build time (scripts/precompress.mjs, 77% smaller) and served via
ServeDir::precompressed_br/gzip: 2016x less per-request work, and
clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]
Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
AuthorizationEngine::check_files_read_batch (default = old loop;
PgAclEngine override batches drive resolution + reuses role cache).
200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
recording (2 writes/file) for subtree entries already authorized at the
root - mirrors the native folder-download path. ~6,000 statements
removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
(weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
- removes one manifest query (p50 0.44-4.4 ms) from every stream,
range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
[benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
SELECT + full-row write-back: racy, lost updates, clobbered concurrent
owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
state machine for the ~100% already-NFC case (every row loaded from PG).
Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.
New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
resolve_quota only ever reported the caller's personal envelope,
ignoring the drive_id already resolved at every PROPFIND call site —
shared drives with their own quota showed the wrong numbers. Adds
AppState::resolve_webdav_quota, shared by both WebDAV surfaces:
nil drive_id or personal drive -> account envelope, shared drive ->
its own storage.drives quota/used_bytes.
Also adds quota-used-bytes/quota-available-bytes to the NextCloud-
compatible surface, which previously had no RFC 4331 support at all.
Registers webdav_quota_properties.hurl and the new
nc_webdav_quota_properties.hurl in tests/api/run.sh — neither was
wired into the suite before this change.
Threads the caller's account-wide (used, available) storage figures through
PROPFIND for the plain-file WebDAV surface, resolved once per request via
StorageUsagePort::get_user_storage_info and reused for every folder entry
in the response. Unlimited accounts (quota <= 0) omit quota-available-bytes
entirely per RFC 4331 §3, rather than disclosing a sentinel value.
Properties are only advertised as known when the quota subsystem is enabled
and the lookup succeeds; otherwise they fall through to the standard 404
propstat.
add env variable `OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX`
which is by default:
`OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX="@drive"`
so `/webdav/` -> points to user's personal drive (**backward compatibilit**y)
`/web/dav/@drive/{uuid|drive name}/` points to the respective drive
if admins want directly `/webdav/` pointing to list of drives they need to:
`OXICLOUD_WEBDAV_DRIVE_LISTING_PREFIX=""`
+ ensure lock is per user (RFC 4918 §9.11)
fix: #554
repl user_id by caller has read access in readonly functions
using CALLER_CAN_READ_DRIVE constant
ensure webdav preview is using the permission handler
DeadPropertyStore let PROPPATCH set any namespace/name verbatim,
incl. names the server itself emits as live state (DAV: entirely,
plus oc:/nc:/ocs: names used by write_file_response /
write_folder_response). That either forges a live prop or stores
dead rows nothing ever reads. is_protected_property() denylists
them; both PROPPATCH handlers (native + NC) now return per-property
403 instead of storing. oc:favorite stays writable via its existing
special-case, which runs before the protection check.
NC PROPPATCH only special-cased oc:favorite, silently dropped every
other prop while claiming 200. PROPFIND had no dead-prop lookup at
all. Reuse native /webdav/ dead-prop plumbing (now pub(crate)) so NC
surface gets real RFC 4918 §4.2 storage. Missing resource on
PROPPATCH is now 404 instead of a fake success no-op.
ensure that dead properties are correctly deleted on resource deletion
**IMPORTANT**: this is a partial fix:
dead properties are not deleted if resource is deleted from API !
code need to be reviewed to attach property directly to resource + use on delete cascade
Complete RFC 4918 litmus test suite compliance (basic 16/16,
copymove 13/13, props 30/30) by fixing six categories of failures:
- PropPatchOp: parse PROPPATCH in document order (RFC 4918 §9.2)
so that remove-then-set and set-then-remove yield different results
- Null namespace: resolve_name now correctly maps xmlns="" to empty-ns
QualifiedName instead of falling through to DAV: namespace
- Unicode chars: handle quick-xml 0.39's Event::GeneralRef for character
references (𐀀) — the parser emits these as GeneralRef, not Text
- MOVE preserves dead props: call rename_resource on MOVE, clearing
stale destination data even when source has no registered properties
- Malformed PROPFIND: return 400 when body lacks a complete <propfind>
element (test 2: bare <foo>, test 3: invalid xmlns:prefix="" binding)
- PROPPATCH document order: process ops via Vec<PropPatchOp> instead of
separate (sets, removes) to honour interleaved remove/set sequences
- COPY: return 403 on self-copy, 409 when destination parent missing,
409 (not 500) when overwriting a locked or conflicting resource
- MOVE: same parent-missing and conflict handling as COPY
- MKCOL: return 405 when collection already exists, 409 when parent
is missing (no auto-creation of ancestors)
- PUT: return 201 Created for new resources, 204 No Content for overwrites
- PROPFIND: use client-facing URI path (not internal home-folder path)
for DAV:href values so responses match the request URI
- PROPFIND: include dead properties from DeadPropertyStore in responses
- PROPPATCH: persist set/remove operations to DeadPropertyStore
Add an in-memory DeadPropertyStore backed by RwLock<HashMap> that
stores arbitrary client-supplied XML properties per resource path.
Wire it into AppState so PROPPATCH can persist dead props and PROPFIND
can retrieve them across requests.
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
now Drive is purely a metadata
each drive has always a root folder
this model minimize Oxicloud changes, and simplify
the Drive name is simply the folder's root's name
note: owner of Drive has more permission that an owner of the root folder
Threads `new_name: Option<&str>` through FileWritePort::copy_file and
FileManagementUseCase::copy_file_with_perms so a same-folder
COPY /a.txt → /b.txt picks up the destination name via a single
COALESCE($3::text, name) in the CTE. Without it the new row inherits
the source's filename and collides on the (folder_id, name, user_id)
unique index — the "Already Exists" 500 M8 was hitting.
handle_copy in the native WebDAV surface now passes
`(file.name != dest_name).then(|| dest_name.into())`, keeping the
"same name in a different folder" case at None so existing semantics
are preserved.
Extends the N2/PUT lock guard introduced earlier to the rest of the
native mutator surface. Same helper, same If: capture before body
consumption, same 423-on-reject shape:
- handle_delete : check source path
- handle_proppatch : check source path
- handle_move : check source AND destination paths
- handle_copy : check destination path only (source isn't
modified by a copy)
The class-2 DAV advertisement in OPTIONS is now honest across the
full surface, not just PUT.
New tests N2c-N2f run while n-locked.txt is still LOCKed (before the
existing N3 UNLOCK). Each asserts 423 without the token and verifies
the operation didn't half-apply: file present after DELETE-423,
source untouched + no destination after MOVE-423, locked destination's
content unchanged after COPY-423.
Positive (with-token) coverage is implicit via the M-series happy-
path tests that exercise each method on unlocked resources — a
regression that hard-rejected every call would fail there too.
Closes M5 / M7 / M8a / M8b.
The optimized PathResolver and the read-side find_*_by_path queries
disagree on what counts as 'a path that hits a row'. After the drive-
refactor migration rewrote the path column to drop the
"My Folder - <user>/" prefix, files PUT through the WebDAV surface
stayed reachable by GET (legacy lookup) but vanished from the
optimized resolver (strict path-match). MOVE/DELETE/COPY 404'd on
every root-level file as a result.
Introduces resolve_or_legacy: optimized resolver first, then the
GET-style legacy lookups as a strict superset. Ownership is enforced
in both branches. handle_delete / handle_move / handle_copy each
collapsed from two near-identical resolver-only + legacy-only branches
into a single match using the helper — fewer lines, identical
semantics, root-level paths now resolve.
handle_copy also fixes M8b: copy_file_with_perms takes no destination
name, so a copy to a different filename in the same folder collided
with the source. After the copy, rename the new file when dest_name
differs from source name. Mirrors what handle_move already does.
Closes N2. The native WebDAV PUT handler now consults the lock
store before accepting a write: if the target path is exclusively
locked, the request must carry the lock token in its If: header
or the server returns 423 Locked. Without a matching token, the
body is never consumed — a rejected PUT no longer wastes the
upload bandwidth or hits the CDC ingester.
Two helpers are introduced so the same enforcement plugs into
the other mutator methods (delete/move/copy/proppatch) when their
fixes land:
extract_if_header_tokens — angle-bracket-scoop view of If:
(sufficient for one-target writes;
full §10.4 tagged-list grammar would
only matter for multi-resource Ifs)
enforce_native_lock — Some(423) when locked + no/wrong
token, None otherwise
Test N2 flipped from pinned 204 to assert 423. Added N2b: same
PUT with the captured Lock-Token in If:(<...>) returns 204, so a
regression that hard-rejected every PUT would still fail loudly.
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Two DAV-surface fixes that replicate patterns the codebase already had:
NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.
WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
webdav encode_uri_path runs on every PROPFIND href and did
.map(...).collect::<Vec<_>>().join("/"), allocating a String per segment plus
a joined Vec. Write each utf8_percent_encode Display adapter straight into a
single preallocated String. Behavior is identical (split on '/', encode each
segment, join with '/'), including leading/trailing-slash edge cases.
subject_group list / list_with_counts each issued a second SELECT COUNT(*)
round-trip for the total. Fold it into the page query via COUNT(*) OVER() —
the pattern folder_db_repository already uses — halving the round-trips.
total_count is read from the first row and is 0 on an empty page, matching
folder_db_repository's documented convention.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:
- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
Now streams the body to a temp file with incremental BLAKE3 and goes
through update_file_streaming (shared spool helper with the native
WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.
- DedupService::store_chunks materialized every new chunk's data in a Vec
before uploading. Now reads each new chunk by positioned I/O
(read_exact_at, off the runtime via spawn_blocking) just before its
upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.
- The upload spool used the OS temp dir, often tmpfs/RAM in containers
where its page-cache counts against the cgroup memory limit. Add
OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.
Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.
Refs #404
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* make more coherent lifecycles
* remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
* reduce risk of orphean objects
* ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
* more details on docs/architecture/file-and-blob-lifecycle.md :
```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {
fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
fn on_file_updated(file_id, blob_hash, content_type);
fn on_file_copied(file_id, blob_hash, content_type, source_id)
fn on_file_deleted(file_id);
}
// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {
fn on_blob_created(blob_hash, content_type);
fn on_blob_deleted(blob_hash);
}
```
Dionisio Pozo