Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 0 of the delta-sync plan. Re-uploading a file the user already has
(another device, a restore, a duplicate) used to transfer every byte just
for the server to discard them as a dedup hit. The frontend now computes
the file's BLAKE3 locally and, on a hit, registers the file with a single
~150-byte metadata call.
Server — POST /api/files/by-hash:
- All checks live in the application service per the AuthZ rule:
Create permission on the target folder via the authorization engine,
hash ownership via the existing user-scoped query (a non-owned hash
returns 404 — same shape as "no such blob" — and emits an
instant_upload.rejected audit event), quota on the logical size.
- On success: one ref_count bump + the existing save_file_with_blob row
registration (compensation included); is_new_blob=false so lifecycle
hooks skip thumbnail regeneration. ~10 ms warm.
- The storage-usage service is now built before the application services
and injected, instead of only living on AppState.
Client — WASM BLAKE3 + worker:
- wasm/oxicloud-hash: the exact same blake3 crate the server uses,
compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match
server content addresses bit for bit. Built by scripts/build-wasm.sh;
the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no
npm dependencies, no wasm toolchain needed for regular builds.
- static/js/workers/hashWorker.js streams the File in 8 MiB slices off
the main thread (constant RAM at any file size).
- features/files/instantUpload.js orchestrates: threshold (8 MiB — below
it the round-trips cost more than the bytes), user-scoped
/api/dedup/check, by-hash registration, and silent fallback to the
normal byte upload on any miss, race or unsupported environment.
Wired into both uploadFiles and uploadFolderEntries.
- biome.json vendors exclusion fixed to cover nested directories
(previous vendors were .mjs and never matched the *.js include).
Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals
the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms
warm with a 151-byte request (vs 20,971,873 bytes for the byte upload);
the copy downloads byte-identical and the manifest ref_count goes 1→2;
a second user probing the same hash gets exists:false and 404 plus the
audit line; duplicate name → 409, malformed hash → 400; worker and wasm
are served with correct MIME (application/wasm).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Two follow-ups to the streaming-upload work:
The dedup garbage_collect() pass only ran when a user manually emptied
their trash. Zero-reference rows — chunks orphaned by aborted streaming
uploads (registered at ref_count 0 by the ingest rollback) and blobs
dereferenced by trash expiry itself — could linger indefinitely on
instances where nobody empties trash. The periodic TrashCleanupService
sweep now ends every run with garbage_collect() (maintenance pool,
batched), bounding orphan lifetime to the cleanup interval.
Folder-ZIP creation was strictly sequential: open blob stream, deflate,
close, repeat — every per-file blob-store round-trip (PG lookup + backend
open; a full HTTP round-trip on S3/Azure) added to the wall clock. It now
runs as a 2-stage pipeline: a prefetch task streams the planned files'
content ahead of the writer through a bounded channel (~4 MiB), so the
next file's read latency overlaps the current file's compression. ZIP
entries are still written strictly in order, peak RAM stays flat, and a
writer error hangs up the channel so the prefetcher stops on its own.
Verified end-to-end against PostgreSQL 16: an upload aborted at ~14 MB
left exactly 30 ref_count=0 chunk rows which the GC then reclaimed
(8.5 MB, rows + physical files); a chunked upload completed with a wrong
MD5 returned 400 with the tee-computed digest and the SAME session then
completed successfully with the right checksum (parts persist — the old
assembly deleted them, so the documented retry never actually worked);
a 4-file folder ZIP downloaded and extracted byte-identical.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
JWT validation cache now stores Arc<TokenClaims> and validate_token
returns Arc<TokenClaims>. On a cache hit — the 99% path for every
authenticated request — the moka lookup was deep-cloning the whole
claims struct (5 Strings: sub, jti, username, email, role) on every
call. It is now a refcount bump. Read-only callers (admin middleware)
go through Deref and allocate nothing; the auth middleware clones only
the three fields it moves into CurrentUser (was 5 clones, now 3), and
the admin paths clone only role (was 5, now 1). A new test asserts the
hit path returns a pointer-equal Arc.
TokenServicePort::validate_token is the single trait method touched;
its only implementor is JwtTokenService and the only production callers
are the auth and admin middleware (the WOPI handler uses a separate
WopiTokenService).
Dockerfile: rust:1.94.1-alpine3.23 -> rust:1.96-alpine3.24 and
alpine:3.23.3 -> alpine:3.24.0 for the runtime stage.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Files uploaded before chunk_manifests landed (20260414000000) are stored
as ONE whole-file blob with no manifest. Every legacy fallback in
DedupService exists to serve them, and the cost concentrates on Range
reads: with encryption enabled, seeking inside a legacy video decrypts
the ENTIRE blob (AES-GCM is all-or-nothing), where a CDC file decrypts
only the overlapping <=1 MiB chunks.
This adds a one-time, idempotent background migration (spawned from the
composition root after dedup init, maintenance pool) that converts each
legacy blob into a regular CDC file, indistinguishable from a native
upload:
1. Spool the blob through the normal read path (decrypts when
encryption is on) to a per-attempt-unique temp file, verifying
BLAKE3 == hash; sizes come from the verified spool, never from the
legacy storage.blobs.size column (the manifest's total_size drives
Range arithmetic).
2. CDC-chunk + store chunks via the existing store_chunks (one
manifest reference per distinct chunk).
3. One short accounting TX with the blob row locked: manifest INSERT
with ref_count = N current file references, blob ref_count -= N,
row deleted only at exactly 0 - so single-chunk files (chunk hash
== file hash) keep the physical blob, which IS the chunk; only
bookkeeping moves, no bytes are rewritten.
4. Physical whole-file blob deleted only when its row dropped.
Races lean on the row lock: a concurrent identical upload landing a
legacy reference after commit keeps the blob row alive and that file
readable via the fallback (bounded space leak, never data loss); a
crash between chunk store and the TX over-counts one file's chunk refs
(also a bounded leak). Corrupt blobs (content != hash) are logged,
counted, excluded from the sweep and left untouched, with a hard cap
before aborting.
Per-hash failures never block the sweep; manifests are the resumability
marker, so a restart continues where it left off. The legacy read/write
fallbacks stay in place as the safety net while a deployment converges;
they can be deleted once fleets report "legacy re-chunk: nothing to do".
Opt-out via OXICLOUD_LEGACY_RECHUNK=false (documented in example.env)
for metered remote backends where the one-time re-read should be
scheduled deliberately.
Covered by five integration tests against real PostgreSQL (multi-chunk
accounting + Range across a chunk boundary, single-chunk physical-blob
preservation, corrupt-blob isolation, empty blob, and the full
encrypted-backend roundtrip); they run concurrently, which also
exercises the cross-sweep race handling.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Two hot-path fixes for DAV sync clients, which poll continuously:
1. App-password Basic Auth cache TTL 30s → 300s. Every cache miss costs
a full Argon2id verification (~50-100ms CPU) plus two DB queries; at
30s a continuously-syncing client re-paid that every 30s. 5 min cuts
it ~10x. Safe because revoke() already invalidates the user's cached
entries immediately; expiry/deactivation are only re-checked on a
miss, so they now have a <=5 min grace window (comparable to a JWT
access-token lifetime) — documented on the constant.
2. WebDAV lock store: replace the tokio::spawn + sleep scheduled per
acquire/refresh with Moka's per-entry Expiry policy. Office clients
refresh locks constantly, leaving thousands of orphaned sleeping
timers pinned in the runtime that were never cancelled. by_path now
carries the exact per-lock TTL via LockExpiry (no background tasks);
by_token keeps a 24h backstop and resolves through by_path, so a
lingering reverse-index entry can never resurrect an expired lock.
Adds unit tests for the expiry policy and acquire/refresh/release
(the module had none).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.
- Add find_events_by_ical_uids / get_contacts_by_uids through every
layer (domain repo trait → PG repo → storage port/adapter → use case
→ handler) using one indexed `= ANY($2)` query, mirroring the
existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
uid_from_multiget_href helper (percent-decoding, case-insensitive
extension strip, collection hrefs rejected). Exact UID matching also
removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
GET /api/address-books/{id}/contacts: optional ListQuery fields
thread through ContactUseCase::list_contacts to a paginated repo
query. Omitted params keep returning the full book (frontend and
DAV listing/sync paths unchanged).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
The read repository's hash_cache assumed blob hashes were immutable
per file_id, but update_file_content_from_temp remaps the SAME row to
a new hash via swap_blob_hash. After a WebDAV/NC PUT overwrite,
streaming downloads (>=10 MB tier, Range requests, video playback)
kept resolving the OLD blob for the 30 s TTI window — and because
every read refreshes the TTI, a polling client could pin the stale
mapping indefinitely, eventually turning into 500s once the old
blob's refcount hit zero and GC removed it.
The write repository now holds a shared handle to the same moka cache
(clones share storage) and invalidates the entry right after every
content swap and hard delete commits — covering every present and
future caller of the write port rather than one service path. Stale
doc comments claiming the mapping was immutable (and SHA-256) fixed.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Request-path thumbnail generation (REST get_thumbnail and NC preview)
read the full source blob into RAM and decoded it with no concurrency
bound — a first-view gallery of K images stacked K full-size buffers
and K parallel decodes on the blocking pool. The background hook had
the inverse ordering problem: it read the blob eagerly and only then
queued on the decode semaphore, so N concurrent uploads held N
originals in memory while waiting.
All three paths now acquire the decode semaphore first and read the
blob under the permit, capping peak RAM at permits x image size:
- new ThumbnailService::get_thumbnail_from_blob defers the blob read
into the moka init closure (cache and disk hits never touch the
blob); both handlers use it and no longer pre-read.
- get_thumbnail_from_bytes (direct-bytes variant) now also takes a
permit before decoding; shared generate_and_persist core keeps the
two entrypoints duplicate-free.
- generate_all_sizes_background_from_blob (renamed from _from_bytes)
checks blob existence and disk-dedup state, then acquires the
permit, then reads; the redundant pre-read spawn wrapper in
ThumbnailRefreshHook is gone.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Replace the per-row tree_modified_at bump triggers with AFTER ... FOR
EACH STATEMENT triggers using transition tables: each DML statement now
pays one bump covering the distinct ancestor chains of all affected
rows (locked in id order so concurrent bumps over overlapping chains
cannot deadlock), instead of one chain UPDATE per affected row.
Value-based change detection replaces UPDATE OF column lists (PG
forbids those with transition tables), so EXIF media_sort_date syncs
and no-op updates no longer bump at all, and file moves now invalidate
the source chain as well as the destination.
Also fix a latent bug surfaced while testing this: the descendant
path/lpath cascade (trg_folders_cascade_path) was declared AFTER
UPDATE OF path, lpath, but rename/move statements SET name/parent_id,
and BEFORE-trigger rewrites do not count for UPDATE OF — the cascade
never fired, leaving every descendant folder with a stale path/lpath
after any rename or move. The migration canonically repairs existing
trees and re-creates the cascade on the columns the app actually
writes.
delete_expired_bulk now deletes in LIMIT-ed batches (1000 files / 100
folders per round, each its own implicit transaction) ordered by
trashed_at and served by new partial indexes over trashed rows, so
retention purges no longer hold one unbounded transaction.
Verified against a local PG16 cluster: all migrations apply from
scratch, the new one re-runs idempotently, and a 16-case behavioral
battery passes (chain bumps for insert/update/delete/move/trash, EXIF
invisibility, no-op invisibility, cascade repair, FK-cascade depth
guards, batched purge shape and partial-index plan).
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
The five FileLifecycleHook call sites handed dto.etag (format
"{blob_hash[..16]}-{modified_at}") to the blob_hash parameter. Blob
lookups keyed by that value can never resolve, so background thumbnail
generation and audio metadata extraction silently failed on every
upload/copy/update, pushing all thumbnail work onto the request path.
Pass dto.content_hash — the raw full BLAKE3 hash already carried by
FileDto — instead.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Same whitespace-only reformat as branch claude/jolly-johnson-yso7z7:
PR #326 landed three files that fail cargo fmt --check and its CI run
skipped the Rustfmt job, breaking the check for every later Rust PR.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Resolves the four-file conflict with PR #444, which landed a parallel
implementation of the same batch-fsync design (unsynced chunk writes plus
one sync_blobs durability barrier). Both sides converged on identical
trait signatures; main's version is kept for all overlapping code since
it is already merged and carries unit tests:
- blob_storage_ports.rs: doc-comment-only conflict, main's docs kept
- dedup_service.rs: main's pipeline kept. The naive textual merge would
have retained BOTH durability-barrier/INSERT blocks, double-counting
ref_count for every new chunk; the duplicated block from this branch
is dropped
- encrypted_blob_backend.rs, local_blob_backend.rs: main's
implementations kept (fsync_paths_parallel + tests); this branch's
equivalent helpers (fsync_dir, fsync_blob_file, SYNC_BLOBS_CONCURRENCY)
are referenced nowhere else and are dropped
Everything unique to this branch (streaming PROPFIND, range requests,
batch ID resolution, CalDAV/CardDAV indexed lookups, transcode and
breadcrumb changes) merges cleanly and is unchanged.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Creating a grant for a group dispatched the notification emails one at
a time inside the HTTP request — 30 members × ~500 ms of SMTP ≈ 15 s
holding the POST /api/grants response (the code carried a TODO
acknowledging it).
Dispatches are independent (coalescing and rate-limiting key on the
(granter, recipient) pair, distinct per member), so run them through
`buffered(6)`: ~6× less wall time for group fan-outs while capping
parallel SMTP sessions, with outcome order still matching member order.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The CDC chunk path paid `sync_all` + parent-dir fsync + one ref-count
upsert round-trip PER ~256 KB chunk: a new 1 GB file ≈ 8 000 fsyncs +
4 000 sequential INSERTs on the upload critical path (seconds on SSD,
tens of seconds on HDD).
- New `put_blob_from_bytes_unsynced` + `sync_blobs` on
BlobStorageBackend. Defaults delegate to the durable variants, so
remote backends (S3/Azure — durable on PUT) are untouched.
LocalBlobBackend writes chunks without fsync and `sync_blobs` then
fsyncs all files concurrently (kernel coalesces the writeback) plus
one fsync per DISTINCT shard directory — at most 256 — instead of one
per chunk. EncryptedBlobBackend delegates both so encryption over the
local backend keeps the optimization.
- store_chunks: uploads use the unsynced variant; one `sync_blobs`
barrier runs before anything references the chunks, and the per-chunk
ref-count upserts collapse into a single
`INSERT ... SELECT unnest(...) ON CONFLICT` statement.
Durability contract is unchanged: every chunk is on stable storage
before the manifest row that references it commits. A crash mid-upload
leaves chunk files without DB rows — the same orphan class the
per-chunk scheme already produced, just a wider window.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.
Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every GET of a JPEG ≤5 MB with `Accept: image/webp` ran a full decode +
lossless-WebP encode on the rayon pool (hundreds of ms of CPU), found
the result LARGER than the original — the `image` crate's WebP encoder
is lossless-only, which practically never beats an already-lossy photo —
and threw the work away. Nothing recorded the verdict, so the next GET
of the same file repeated it, forever. A 20-photo gallery saturated the
transcode pool on every visit.
- `can_transcode` no longer accepts `image/jpeg` — photos skip the
pipeline entirely. PNG/GIF stay (lossless WebP genuinely shrinks them).
- Negative verdicts for the remaining formats are now remembered: an
empty-Bytes sentinel in the moka cache (expires with the 10-min TTL)
plus a zero-byte `.skip` marker on disk that survives restarts.
`invalidate()` removes the marker alongside the cached transcode.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Two DAV-surface fixes that replicate patterns the codebase already had:
NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.
WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Closes#323.
LoginLockoutService cached failed-attempt counters keyed only on
the username, so any caller that could reach the auth endpoint and
guess (or enumerate) a username could lock that account out for the
entire lockout window — the rate limiter happily lets each IP make
its share of bad-password attempts before clamping, which is enough
to trip the per-account threshold in seconds. The reporter
demonstrated a complete DOS by spoofing X-Forwarded-For with
OXICLOUD_TRUST_PROXY_HEADERS=true.
Fix: change the lockout cache key from `username` to `username|ip`.
A flood from one IP locks that IP out of that account, but a
legitimate user coming from a different IP is unaffected.
Changes:
- LoginLockoutService::{check, record_failure, record_success} take
client_ip as a second argument; cache key is built via Self::key
(`format!("{username}|{ip}")`).
- middleware/rate_limit.rs: factor out extract_client_ip_from_parts
(HeaderMap + Option<&SocketAddr>) so handlers that don't take a
full Request<B> can still derive the same client identifier
extract_client_ip uses. extract_client_ip now delegates to it.
- auth_handler.rs login: derive client_ip from headers (the only
signal available without ConnectInfo) and pass it through to all
three lockout calls.
- nextcloud/basic_auth_middleware.rs: do the same with the full
Request via extract_client_ip.
Tests:
- Updated existing 4 unit tests to thread an IP arg.
- New does_not_lock_out_other_ips_for_same_account: lock from IP1,
assert IP2 still allowed (the #323 regression).
- New success_resets_only_the_acting_ip: a successful login from
IP2 must NOT clear an attacker's lockout from IP1.
Verification:
- `cargo build` ✅
- `cargo test login_lockout` → 6 passed (4 existing thread an IP
arg without behaviour change, 2 new pin the per-IP scoping).
Signed-off-by: SAY-5 <say.apm35@gmail.com>
The main search endpoint took `limit: params.limit.unwrap_or(100)` with no
ceiling, and that value flows straight into the SQL LIMIT of
search_files_paginated / search_files_in_subtree. A client passing
?limit=<huge> would make Postgres return that many rows into memory and into
the result cache. The suggestions endpoint already clamps (.min(20)); search
did not.
Cap at MAX_SEARCH_LIMIT (500). total_count still reflects the full match set
(COUNT(*) OVER()), so deeper results stay reachable via offset.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
store_chunks bumped storage.blobs.ref_count once per chunk *occurrence* (it
looped over the full chunk list, duplicates included), but
remove_manifest_reference decrements once per *distinct* chunk
(WHERE hash = ANY(chunk_hashes) matches each row a single time). For any file
that repeats a chunk -- zero-filled regions in disk/VM images, repeated
document structures, concatenated archives -- storing added +N while deleting
removed -1, so the blob's ref_count never returned to 0 and the chunk was
never garbage-collected: a permanent storage leak.
Count per distinct chunk on the store side too, matching deletion. This also
makes it faster:
- existing chunks: one batched `UPDATE ... WHERE hash = ANY($1)` instead of
one UPDATE per occurrence;
- a brand-new chunk repeated within a file is read, uploaded and INSERTed
once instead of once per occurrence.
The manifest still stores the full per-occurrence chunk sequence (needed to
reassemble the file). Forward fix: blobs already over-counted by the old path
stay over-counted (a reconcile/verify pass could recompute them), but the
bias is upward (leak), so no data is ever deleted early.
CDC tests pass (12); fmt + clippy clean.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
The /api router added its own predicate-less CompressionLayer (routes.rs),
nested inside the global predicate-aware one in main.rs. As the inner layer it
compressed responses first, so the global predicate that skips already-
compressed media was bypassed for every /api response: video/audio/image/zip
downloads got Brotli-compressed (CPU + first-byte latency for ~0 bytes saved)
and lost their Content-Length (forced to chunked -> no client progress bar).
- Remove the redundant /api CompressionLayer; /api now flows through the
single global layer in main.rs.
- Make that predicate smarter: compress by default so nothing shrinkable is
missed, and skip ONLY already-compressed types. It no longer blanket-excludes
image/*, so image/svg+xml (text, ~70% shrink) now compresses; raster formats
are listed individually. Added the previously-missed already-compressed
types: Office (docx/xlsx/pptx), ODF, epub, jar, apk, 7z/rar/bzip2/zstd/xz,
woff/woff2 fonts, icons.
Net: media downloads keep Content-Length and skip pointless compression, while
text/JSON/JS/CSS/SVG/XML/ttf/otf/wasm still compress. fmt + clippy clean.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
StoragePath::join deep-cloned the whole Vec<String> (every segment String)
just to append one element. Take self by value and push in place. All callers
pass owned values except PathService::create_file_path, which holds a borrow
and now clones explicitly — the same copy the old &self join already made.
File::with_name / with_folder / with_size took &self and rebuilt the struct,
cloning every carried-over field (id, mime_type, folder_id, blob_hash, ...).
Consume self and mutate only the fields that change. Behaviour is identical;
the fallible builders now drop the input on Err, which is fine for these
rename/move/resize transforms (all current callers replace the file).
Impact is small in practice — with_folder/with_size have no callers and
with_name is test-only, while the one hot join caller (create_file_path)
must copy segments regardless — but the consuming form is the idiomatic one.
Verified: cargo fmt + clippy --all-features --all-targets -D warnings clean;
domain tests (path_service::, entities::file::) pass — 30 + 6.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
webdav encode_uri_path runs on every PROPFIND href and did
.map(...).collect::<Vec<_>>().join("/"), allocating a String per segment plus
a joined Vec. Write each utf8_percent_encode Display adapter straight into a
single preallocated String. Behavior is identical (split on '/', encode each
segment, join with '/'), including leading/trailing-slash edge cases.
subject_group list / list_with_counts each issued a second SELECT COUNT(*)
round-trip for the total. Fold it into the page query via COUNT(*) OVER() —
the pattern folder_db_repository already uses — halving the round-trips.
total_count is read from the first row and is 0 on an empty page, matching
folder_db_repository's documented convention.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
list_incoming_grants and list_grants_on_resource ran fetch_all with no
LIMIT, so a pathological number of grants on one resource (or targeting one
subject) would be pulled fully into memory. These back the grant-management
endpoints ("Manage sharing", "Shared with me"), not the hot require() path.
A blind LIMIT is unsafe here: apply_role reads the full grant set to compute
an add/remove diff, so a silently truncated list would be acted on as if
complete (stale grants never revoked). Instead fetch MAX_GRANT_ROWS + 1 and
reject with an audit line (authz.grant_list_rejected / reason=over_row_cap)
when the cap is exceeded, bounding worst-case RAM without ever returning a
partial set. The check is shared via PgAclEngine::guard_grant_row_cap. Cap is
10_000 — orders of magnitude above any realistic single-resource/subject
grant count.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
CachedBlobBackend held its single tokio::Mutex<LruCache> across filesystem
syscalls, serializing every concurrent cache operation behind one lock:
- get_blob_stream / get_blob_range_stream: held across File::open()/seek()
- delete_blob: held across remove_file()
- initialize: held across the full cache-dir walk
- eviction (insert + fetch paths): held across remove_file() loops
Now the lock only guards the in-memory LRU. Presence checks bump recency
and release the guard before touching the filesystem (a vanished file falls
through to the existing fetch-and-cache path, covering the race), and
eviction selects victims under the lock then unlinks them after releasing
it. The duplicated eviction loop is extracted into
CachedRef::collect_evictions.
db: set test_before_acquire(false). With warm min_connections and a bounded
max_lifetime, the liveness ping sqlx issues on every acquire() costs more
than the rare dead connection it catches; stale sockets surface as a query
error and the pool recycles them either way.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
a prefix path as been add to avoid collision if OXICLOUD_UPLOAD_TMPDIR=OXICLOUD_CHUNK_DIR
upgrade to this version will convert previous upload chunked into the prefixed version at server start
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
to help administratorrs to defined correctly their storage architecture
purpose: avoid current scheme:
1. write to disk
2. reopen file to read data and digest it
now: digest is done while writing data to disk
all other implementation than Nextcloud are corrrect
add OXICLOUD_CHUNK_MAX_BYTES which correspond to the max upload chunk allowed
(differs from OXICLOUD_MAX_UPLOAD_SIZE which is the max total size of a file)
hurl test validate the change
Streams the request body straight to the chunk file with peak heap of
~one HTTP frame, regardless of chunk size or the configured cap. The
`storage.chunk_max_bytes` config (env `OXICLOUD_CHUNK_MAX_BYTES`,
default 100 MB) bounds a single PUT — separate from `max_upload_size`
which governs whole-file uploads. Without this separation, a client
could submit a chunk up to the whole-file cap (10 GB default) and
monopolise server memory.
- add /ocs/v1.php/cloud/user route (same as /ocs/v2.php/cloud/user )
- add also /remote.php/dav/avatars/{user}/{size}.png route (used by Nextcloud sync)
note: webp are transcoded in png, no cache but using pragma HTTP header
PROPFIND on upload chunk is necessary for application to resume the upload
it is used at least by Nextcloud Android app
issue has been raised via #415
unable to test, so I am using a playbook in test/webdav to simulate a
PROPIND on remote.php/dav/uploads/{user}/{session}
This solve issue with 2 migrations made the same day, due to merge on pull request, DB migration is blocking
the same version prefix:
- 20260625000000_files_user_size_index.sql (Dio)
- 20260625000000_folder_tree_modified_at.sql (Ed)
They were renamed to ...0001 and ...0002 (disjoint versions) + protection like "IF NOT EXISTS"
I have opt for an automated clean up of old entry:
`DELETE FROM _sqlx_migrations WHERE version = 20260625000000;`
runned on startup
affected users: Dio, myself and any dev that wanted to work on this project since eb0ba58158
GET /api/auth/me ran a synchronous O(N) SUM(size) over all the user's
files plus an unconditional UPDATE of auth.users on every call — one of
the most frequently hit endpoints — adding per-request latency, DB write
load, dead tuples and WAL even when nothing changed.
- /api/auth/me now serves the cached storage_used_bytes column instead of
recomputing it inline.
- New StorageUsageService::start_reconciliation_job runs a periodic sweep
on the maintenance pool that keeps the cached value current for every
mutation (uploads, deletes, trash), so freshness no longer depends on
hitting /me. Interval via OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default
600s, floored at 30s; first sweep deferred one interval to avoid boot load).
- update_storage_usage only writes when the value actually changes
(IS DISTINCT FROM), so the sweep produces no dead tuple / WAL on no-ops.
- New covering partial index idx_files_user_size_active makes the usage
SUM an index-only scan instead of a heap scan over all the user's files.
Also collapse the same pre-existing clippy collapsible_else_if in
carddav_handler that blocks the -D warnings gate on this base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:
- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
Now streams the body to a temp file with incremental BLAKE3 and goes
through update_file_streaming (shared spool helper with the native
WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.
- DedupService::store_chunks materialized every new chunk's data in a Vec
before uploading. Now reads each new chunk by positioned I/O
(read_exact_at, off the runtime via spawn_blocking) just before its
upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.
- The upload spool used the OS temp dir, often tmpfs/RAM in containers
where its page-cache counts against the cgroup memory limit. Add
OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.
Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.
Refs #404
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude