Commit Graph

258 Commits

Author SHA1 Message Date
Bradley Nelson 4427b1613b plugin logging 2026-06-16 23:00:23 -06:00
Bradley Nelson 803150635c add frontend 2026-06-16 21:26:36 -06:00
Edouard Vanbelle 4fc3746754 chore(logs): add explicit http logs
Default is now RUST_LOG=info,http=warn. Effect of each level on the access log:

  ┌────────────────────┬────────────────────────┐
  │   Level on http    │ Status classes emitted │
  ├────────────────────┼────────────────────────┤
  │ info               │ 2xx/3xx + 4xx + 5xx    │
  ├────────────────────┼────────────────────────┤
  │ warn (default)     │ 4xx + 5xx              │
  ├────────────────────┼────────────────────────┤
  │ error              │ 5xx only               │
  ├────────────────────┼────────────────────────┤
  │ off                │ nothing                │
  └────────────────────┴────────────────────────┘

  Target mapping:

  ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────┐
  │                                                       Routes                                                       │     Target      │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ health_routes                                                                                                      │ http::probe     │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ magic_link_router                                                                                                  │ http::web       │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ All /api/auth/* sub-routers (login, register, refresh, public, protected, app_pw, device_public, device_protected) │ http::api::auth │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ setup_router, public_api_routes, protected_api, wopi_api_protected                                                 │ http::api       │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ well_known_router, caldav_protected, carddav_protected, webdav_protected                                           │ http::dav       │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ nc_router                                                                                                          │ http::nextcloud │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ wopi_protocol                                                                                                      │ http::wopi      │
  ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
  │ web_routes (+ ServeDir fallback)                                                                                   │ http::web       │
  └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────┘

  # Default value:

  - **http=warn** if target http not specified
  - **http::web=error** if target http::web not specified

  Common operator overrides:

  # Server-error-only access logs (the new default)
  unset RUST_LOG

  # See login failures and other client errors on auth
  RUST_LOG=info,http=warn,http::api::auth=info

  # which is similar to
  RUST_LOG=info,http::api::auth=info

  # Full access log everywhere (heavy)
  RUST_LOG=info,http=info

  # Silence everything except errors
  RUST_LOG=warn
2026-06-15 21:56:59 +02:00
DioCrafts 81a93a489b feat: photo/video capture-date pipeline + premium UI/UX overhaul
Backend — Photos timeline now groups by real capture date instead of upload time. New MediaMetadataService (FileLifecycleHook) extracts EXIF DateTimeOriginal from images and container creation_time from videos (mov/mp4/mkv) via nom-exif, timezone-correct (OffsetTimeOriginal), persisting captured_at so the existing media_sort_date trigger takes over. Adds POST /admin/photos/metadata/reextract to backfill existing media. Falls back to upload date when no embedded date exists.

Frontend — premium grid cards: combined metadata line (relative date · size, owner avatar when shared), custom selection checkbox with a clear checked state, uniform full-width 4:3 thumbnail tiles independent of filename length, centered file-type icons, and a hit-test fix so checkbox/star/kebab clicks reach the controls (the decorative thumbnail no longer captures pointer events). Notification messages internationalised across all 16 locales. Broader polish: design tokens, a11y/focus-visible states, brand + PWA assets.

Chore — bump semver-compatible dependencies (cargo upgrade); add nom-exif 3.6.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 00:24:27 +02:00
Claude ed9a204e49 Delta download: file manifest + user-scoped chunk fetch for sync clients
Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:

- GET /api/files/{id}/manifest returns the file's chunk recipe
  ({file_hash, total_size, chunks}). Owner-scoped like the rest of the
  delta surface (Read permission through the authz engine first, then
  the chunk layer's possession standard; shared files use the regular
  download endpoints). A manifest is immutable for a given file_hash,
  so it is served with ETag = file_hash and If-None-Match answers 304 —
  polling sync clients pay one header round-trip per unchanged file.
  Legacy pre-CDC blobs are presented as a single-chunk manifest of
  themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
  [u32 BE length][bytes] frames in request order — the same wire format
  the upload direction uses. Entitlement is the same possession rule as
  negotiate/commit (chunks reachable through the caller's own files);
  anything else returns 404 {not_available} — deliberately
  indistinguishable from "never existed" — with a
  delta_download.rejected audit event. Batches are bounded by the
  chunk_max_bytes budget; Content-Length is exact (sizes come from the
  dedup index) and peak RAM is one backend read frame.

Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).

Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 16:38:02 +00:00
Claude 44967da7f1 Delta-upload protocol: negotiate chunks by hash, upload only what changed
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:

- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
  which ones the caller must upload. User-scoped and purely advisory.
- PUT  /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
  frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
  Every hash is recomputed server-side; chunks land as ref_count=0 orphans
  that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
  a single UPDATE…RETURNING restricted to chunks the caller is entitled
  to (reachable through their own non-trashed files, or unreferenced
  orphans); anything else returns 409 {still_missing} for the client to
  upload and retry. The pinned sequence is then RE-READ and the whole-file
  BLAKE3 recomputed before any manifest exists — a declared file_hash is
  never trusted, because a forged manifest would poison future whole-file
  dedup hits for other users. The manifest accounting is shared with the
  byte path (attach_manifest, extracted from store_from_stream); the file
  row is created (201) or its content swapped by file_id (200). Owners of
  the exact file_hash short-circuit to a pure reference bump.

Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.

Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 14:34:02 +00:00
Claude 0fab4ce17d Instant upload: register already-owned content by hash, zero bytes on the wire
Phase 0 of the delta-sync plan. Re-uploading a file the user already has
(another device, a restore, a duplicate) used to transfer every byte just
for the server to discard them as a dedup hit. The frontend now computes
the file's BLAKE3 locally and, on a hit, registers the file with a single
~150-byte metadata call.

Server — POST /api/files/by-hash:
- All checks live in the application service per the AuthZ rule:
  Create permission on the target folder via the authorization engine,
  hash ownership via the existing user-scoped query (a non-owned hash
  returns 404 — same shape as "no such blob" — and emits an
  instant_upload.rejected audit event), quota on the logical size.
- On success: one ref_count bump + the existing save_file_with_blob row
  registration (compensation included); is_new_blob=false so lifecycle
  hooks skip thumbnail regeneration. ~10 ms warm.
- The storage-usage service is now built before the application services
  and injected, instead of only living on AppState.

Client — WASM BLAKE3 + worker:
- wasm/oxicloud-hash: the exact same blake3 crate the server uses,
  compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match
  server content addresses bit for bit. Built by scripts/build-wasm.sh;
  the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no
  npm dependencies, no wasm toolchain needed for regular builds.
- static/js/workers/hashWorker.js streams the File in 8 MiB slices off
  the main thread (constant RAM at any file size).
- features/files/instantUpload.js orchestrates: threshold (8 MiB — below
  it the round-trips cost more than the bytes), user-scoped
  /api/dedup/check, by-hash registration, and silent fallback to the
  normal byte upload on any miss, race or unsupported environment.
  Wired into both uploadFiles and uploadFolderEntries.
- biome.json vendors exclusion fixed to cover nested directories
  (previous vendors were .mjs and never matched the *.js include).

Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals
the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms
warm with a 151-byte request (vs 20,971,873 bytes for the byte upload);
the copy downloads byte-identical and the manifest ref_count goes 1→2;
a second user probing the same hash gets exists:false and 404 plus the
audit line; duplicate name → 409, malformed hash → 400; worker and wasm
are served with correct MIME (application/wasm).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 13:54:32 +00:00
Claude e3f04d58aa Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 13:06:33 +00:00
Claude 6a27c742d4 Fix CalDAV/CardDAV multiget loading entire collections into memory
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.

- Add find_events_by_ical_uids / get_contacts_by_uids through every
  layer (domain repo trait → PG repo → storage port/adapter → use case
  → handler) using one indexed `= ANY($2)` query, mirroring the
  existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
  uid_from_multiget_href helper (percent-decoding, case-insensitive
  extension strip, collection hrefs rejected). Exact UID matching also
  removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
  get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
  GET /api/address-books/{id}/contacts: optional ListQuery fields
  thread through ContactUseCase::list_contacts to a paginated repo
  query. Omitted params keep returning the full book (frontend and
  DAV listing/sync paths unchanged).

https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
2026-06-11 09:21:35 +00:00
Claude b0f83cfa34 perf(thumbnails): hold a decode permit before reading source blobs
Request-path thumbnail generation (REST get_thumbnail and NC preview)
read the full source blob into RAM and decoded it with no concurrency
bound — a first-view gallery of K images stacked K full-size buffers
and K parallel decodes on the blocking pool. The background hook had
the inverse ordering problem: it read the blob eagerly and only then
queued on the decode semaphore, so N concurrent uploads held N
originals in memory while waiting.

All three paths now acquire the decode semaphore first and read the
blob under the permit, capping peak RAM at permits x image size:

- new ThumbnailService::get_thumbnail_from_blob defers the blob read
  into the moka init closure (cache and disk hits never touch the
  blob); both handlers use it and no longer pre-read.
- get_thumbnail_from_bytes (direct-bytes variant) now also takes a
  permit before decoding; shared generate_and_persist core keeps the
  two entrypoints duplicate-free.
- generate_all_sizes_background_from_blob (renamed from _from_bytes)
  checks blob existence and disk-dedup state, then acquires the
  permit, then reads; the redundant pre-read spawn wrapper in
  ThumbnailRefreshHook is gone.

https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
2026-06-10 13:36:29 +00:00
Claude d73065e06b style: apply rustfmt to login-lockout code merged in #326
Same whitespace-only reformat as branch claude/jolly-johnson-yso7z7:
PR #326 landed three files that fail cargo fmt --check and its CI run
skipped the Rustfmt job, breaking the check for every later Rust PR.

https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
2026-06-10 11:51:56 +00:00
Claude 7687766bf7 perf(dav-collections): indexed UID lookups for single-object operations
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.

Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.

https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
2026-06-10 09:52:28 +00:00
Claude 1912a17fa2 perf(dav): stream NC PROPFIND in batches; Range + 304 on WebDAV GETs
Two DAV-surface fixes that replicate patterns the codebase already had:

NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.

WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.

https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
2026-06-10 09:27:48 +00:00
SAY-5 b9af3092be chore: remove em-dashes from comments 2026-06-09 15:11:16 -07:00
SAY-5 9dfb29bdda fix(auth): scope lockout key to (account, IP) to prevent DOS by login flood
Closes #323.

LoginLockoutService cached failed-attempt counters keyed only on
the username, so any caller that could reach the auth endpoint and
guess (or enumerate) a username could lock that account out for the
entire lockout window — the rate limiter happily lets each IP make
its share of bad-password attempts before clamping, which is enough
to trip the per-account threshold in seconds. The reporter
demonstrated a complete DOS by spoofing X-Forwarded-For with
OXICLOUD_TRUST_PROXY_HEADERS=true.

Fix: change the lockout cache key from `username` to `username|ip`.
A flood from one IP locks that IP out of that account, but a
legitimate user coming from a different IP is unaffected.

Changes:
- LoginLockoutService::{check, record_failure, record_success} take
  client_ip as a second argument; cache key is built via Self::key
  (`format!("{username}|{ip}")`).
- middleware/rate_limit.rs: factor out extract_client_ip_from_parts
  (HeaderMap + Option<&SocketAddr>) so handlers that don't take a
  full Request<B> can still derive the same client identifier
  extract_client_ip uses. extract_client_ip now delegates to it.
- auth_handler.rs login: derive client_ip from headers (the only
  signal available without ConnectInfo) and pass it through to all
  three lockout calls.
- nextcloud/basic_auth_middleware.rs: do the same with the full
  Request via extract_client_ip.

Tests:
- Updated existing 4 unit tests to thread an IP arg.
- New does_not_lock_out_other_ips_for_same_account: lock from IP1,
  assert IP2 still allowed (the #323 regression).
- New success_resets_only_the_acting_ip: a successful login from
  IP2 must NOT clear an attacker's lockout from IP1.

Verification:
- `cargo build` ✅
- `cargo test login_lockout` → 6 passed (4 existing thread an IP
  arg without behaviour change, 2 new pin the per-IP scoping).

Signed-off-by: SAY-5 <say.apm35@gmail.com>
2026-06-09 14:47:47 -07:00
Dionisio Pozo 087cb44add Merge pull request #438 from EdouardVanbelle/feat/cap-chunk-size-and-use-stream 2026-06-09 18:19:49 +02:00
Claude 175a908e99 perf(search): clamp search page size to bound per-request memory
The main search endpoint took `limit: params.limit.unwrap_or(100)` with no
ceiling, and that value flows straight into the SQL LIMIT of
search_files_paginated / search_files_in_subtree. A client passing
?limit=<huge> would make Postgres return that many rows into memory and into
the result cache. The suggestions endpoint already clamps (.min(20)); search
did not.

Cap at MAX_SEARCH_LIMIT (500). total_count still reflects the full match set
(COUNT(*) OVER()), so deeper results stay reachable via offset.

https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
2026-06-09 14:59:01 +00:00
Claude 19b41af2e4 perf(http): single smart compression layer — stop compressing media on /api
The /api router added its own predicate-less CompressionLayer (routes.rs),
nested inside the global predicate-aware one in main.rs. As the inner layer it
compressed responses first, so the global predicate that skips already-
compressed media was bypassed for every /api response: video/audio/image/zip
downloads got Brotli-compressed (CPU + first-byte latency for ~0 bytes saved)
and lost their Content-Length (forced to chunked -> no client progress bar).

- Remove the redundant /api CompressionLayer; /api now flows through the
  single global layer in main.rs.
- Make that predicate smarter: compress by default so nothing shrinkable is
  missed, and skip ONLY already-compressed types. It no longer blanket-excludes
  image/*, so image/svg+xml (text, ~70% shrink) now compresses; raster formats
  are listed individually. Added the previously-missed already-compressed
  types: Office (docx/xlsx/pptx), ODF, epub, jar, apk, 7z/rar/bzip2/zstd/xz,
  woff/woff2 fonts, icons.

Net: media downloads keep Content-Length and skip pointless compression, while
text/JSON/JS/CSS/SVG/XML/ttf/otf/wasm still compress. fmt + clippy clean.

https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
2026-06-09 14:08:27 +00:00
Claude ec8ddebc30 perf: drop intermediate allocs in WebDAV href encoding; fold group-list COUNT into one query
webdav encode_uri_path runs on every PROPFIND href and did
.map(...).collect::<Vec<_>>().join("/"), allocating a String per segment plus
a joined Vec. Write each utf8_percent_encode Display adapter straight into a
single preallocated String. Behavior is identical (split on '/', encode each
segment, join with '/'), including leading/trailing-slash edge cases.

subject_group list / list_with_counts each issued a second SELECT COUNT(*)
round-trip for the total. Fold it into the page query via COUNT(*) OVER() —
the pattern folder_db_repository already uses — halving the round-trips.
total_count is read from the first row and is 0 on an empty page, matching
folder_db_repository's documented convention.

https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
2026-06-09 13:29:05 +00:00
Edouard Vanbelle 50ea406719 feat(OXICLOUD_DIRECT_PUT_MAX_BYTES): add a security limit on direct PUT
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
    add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
2026-06-09 11:06:44 +02:00
Edouard Vanbelle 964d66e7f8 featchukn-upload): client can provide full file hash completion 2026-06-09 09:55:27 +02:00
Edouard Vanbelle 41aad26702 feat(upload): cover chunk upload + add support of different digest hash
Prefer stream storage rather using buffered (in memory)

  note: on many unix like tmpfs are in-memory, sungle PUT are sized limited

  Storage map (NC stands for Nextcloud gateway)

  ┌───────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────┐
  │                   Streaming surface                   │                            Destination                             │                Configurable via                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST chunked PUT /api/uploads/{id} chunk              │ {storage_path}/.uploads/{upload_id}/chunk_{NNNNNN}                 │ OXICLOUD_STORAGE_PATH (the .uploads subdir is   │
  │                                                       │                                                                    │ hard-wired)                                     │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST chunked assemble (during /complete)              │ {storage_path}/.uploads/{upload_id}/assembled                      │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC chunked PUT /dav/uploads/.../{chunk}               │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}  │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC chunked assemble (during MOVE)                     │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/.assembled    │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC single-file PUT /dav/files/.../{path} (via         │ OXICLOUD_UPLOAD_TMPDIR if set, else OS default temp (/tmp on       │ OXICLOUD_UPLOAD_TMPDIR                          │
  │ spool_body_to_temp)                                   │ Linux)                                                             │                                                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST WebDAV PUT /webdav/{path} (via                   │ same as above                                                      │ OXICLOUD_UPLOAD_TMPDIR                          │
  │ spool_body_to_temp)                                   │                                                                    │                                                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST multipart upload /api/files/upload               │ {storage_path}/.dedup_temp/upload-{uuid}                           │ OXICLOUD_STORAGE_PATH (hard-wired subdir)       │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ WOPI PutFile                                          │ OS default temp via NamedTempFile::new() (no override)             │ (none — bug worth tracking)                     │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ Final blob storage (after fsync + rename)             │ {storage_path}/.blobs/{ab}/{abc…}.blob                             │ OXICLOUD_STORAGE_PATH                           │
  └───────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┘

  one caveat: a malicious user can create many chunked upload and saturate local storage
2026-06-09 09:52:09 +02:00
Edouard Vanbelle 5e638691ad security(upload): cap upload size to prevent memody/disk consumption
add OXICLOUD_CHUNK_MAX_BYTES which correspond to the max upload chunk allowed
    (differs from OXICLOUD_MAX_UPLOAD_SIZE which is the max total size of a file)
    hurl test validate the change

    Streams the request body straight to the chunk file with peak heap of
    ~one HTTP frame, regardless of chunk size or the configured cap. The
    `storage.chunk_max_bytes` config (env `OXICLOUD_CHUNK_MAX_BYTES`,
    default 100 MB) bounds a single PUT — separate from `max_upload_size`
    which governs whole-file uploads. Without this separation, a client
    could submit a chunk up to the whole-file cap (10 GB default) and
    monopolise server memory.
2026-06-09 09:26:33 +02:00
Dionisio Pozo eb0ba58158 Merge pull request #426 from EdouardVanbelle/refactor/etag-centralize
refactor & normalize etag for Nextcloud + fix NFC string (important fix)
2026-06-07 01:26:30 +02:00
DioCrafts 72715c66a0 Merge remote-tracking branch 'origin/main' into perf/auth-me-quota
# Conflicts:
#	example.env
#	src/common/config.rs
2026-06-07 01:25:14 +02:00
DioCrafts d7c6894c80 perf(quota): stop recomputing storage usage on every GET /api/auth/me
GET /api/auth/me ran a synchronous O(N) SUM(size) over all the user's
files plus an unconditional UPDATE of auth.users on every call — one of
the most frequently hit endpoints — adding per-request latency, DB write
load, dead tuples and WAL even when nothing changed.

- /api/auth/me now serves the cached storage_used_bytes column instead of
  recomputing it inline.
- New StorageUsageService::start_reconciliation_job runs a periodic sweep
  on the maintenance pool that keeps the cached value current for every
  mutation (uploads, deletes, trash), so freshness no longer depends on
  hitting /me. Interval via OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default
  600s, floored at 30s; first sweep deferred one interval to avoid boot load).
- update_storage_usage only writes when the value actually changes
  (IS DISTINCT FROM), so the sweep produces no dead tuple / WAL on no-ops.
- New covering partial index idx_files_user_size_active makes the usage
  SUM an index-only scan instead of a heap scan over all the user's files.

Also collapse the same pre-existing clippy collapsible_else_if in
carddav_handler that blocks the -D warnings gate on this base.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 01:13:58 +02:00
DioCrafts 061306cc84 fix(upload): stream WebDAV/NextCloud PUT to disk to prevent OOM on large files
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:

- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
  body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
  Now streams the body to a temp file with incremental BLAKE3 and goes
  through update_file_streaming (shared spool helper with the native
  WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.

- DedupService::store_chunks materialized every new chunk's data in a Vec
  before uploading. Now reads each new chunk by positioned I/O
  (read_exact_at, off the runtime via spawn_blocking) just before its
  upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.

- The upload spool used the OS temp dir, often tmpfs/RAM in containers
  where its page-cache counts against the cgroup memory limit. Add
  OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.

Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.

Refs #404

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 01:03:52 +02:00
Edouard Vanbelle bde7c83932 feat(content_hash): propagate etag and hash_content to */resources 2026-06-06 20:11:13 +02:00
Edouard Vanbelle eb895b0412 fix(webdav): fix trailing / on collection 2026-06-06 20:11:13 +02:00
Edouard Vanbelle c91515cb65 feat(etag): include mtime in file ETag formula 2026-06-06 18:49:14 +02:00
Edouard Vanbelle 0135930da9 refactor(file|folder): separate etag and blob_hash 2026-06-06 18:49:14 +02:00
Edouard Vanbelle 8cc21f17c5 feat(notify): add notif to internal users when granted
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
    - add resentd method in share menu item (work for both internal and external users)
    - user can disable email notification via his properties
    - add env variable from admin to disable notifications
2026-06-05 11:25:06 +02:00
Edouard Vanbelle 16c5e06e2b security(/magic): no cache + no referer + no index 2026-06-03 16:19:03 +02:00
Edouard Vanbelle 7db27af7a6 feat(user.prefered_locale): save user's locale + invited have same locale as inviters
- OIDC JIT define the locale only at user creation, user can so change his preference later
    - invited users will inherit inviter's locale
    - email will use prefered_locale
    - login to a new browser will use prefered_locale
2026-06-03 14:37:22 +02:00
Edouard Vanbelle 854f1d3a07 feat(templating): add and use templates for /magic and emails 2026-06-03 14:10:43 +02:00
Edouard Vanbelle 044bd76738 feat(i18n): add i18n on server side
- remove the hardcoded list of locales in favor of a discovry on start time
    - server will stop on badly formatted locale .json
    - add server.* entries for serer side translation

    server side translation will be used for templating and email
    note: no json in some embded html (like in /magic), amount of work was similar
2026-06-03 13:27:12 +02:00
Edouard Vanbelle 372b99890f feat(magic-link): user can resend email if magic link is expired 2026-06-03 11:39:07 +02:00
Edouard Vanbelle a9a2660576 feat(user edition): permit user without username to define one
- permit also edition of given_name & family_name
    - once username has been define it become immuable (due to Nextcloud implementation)
2026-06-03 00:35:25 +02:00
Edouard Vanbelle 8fc9a50681 feat(passwordless): add cookie challenge + low TTL
magic-link as now 2 modes:

        - invitation: long TTL (24), no challenge
        - passwordless login: short TTL (10min), cookie challenge to ensure that
        user goes back to same browser (no man in the middle capturing email)
2026-06-03 00:35:25 +02:00
Edouard Vanbelle 00af0e8a89 feat(registraton): add anti enumeration (cannot know if an account already exists)
Important: anti-enumeration is active only if SMTP is defined, welcome email can be used
    otherwise it is a classic registration with ok or conflic if account alrady exists
2026-06-03 00:35:25 +02:00
Edouard Vanbelle 130ff363dc feat(passwordless): pass4: add env variable to enable mgaiclink on account with password
OXICLOUD_MAGIC_LINK_OPEN_TO_PASSWORD_USERS (default false)
    For security I recommand to keep it false
    OIDC cannot be bypassed because OIDC may have MFA in place
2026-06-03 00:35:25 +02:00
Edouard Vanbelle 9a49ab44d8 feat(passwordless): pass3: passwordless account (via emailed magic-link)
Backend
  - RegisterDto — username and password both become Option<String> with #[serde(default)] so JSON can omit them entirely.
  - AuthApplicationService::register — username uniqueness check skipped when None (multiple NULLs OK under the UNIQUE index); password hashing skipped when None; User::new called with the actual Options instead of forcing Some(...).
  - auth_handler::register — branches on dto.password.is_none(). With password → existing 201 + UserDto. Without → triggers MagicLinkInviteService::send_login_link(&email) best-effort, then returns 200 + {"message": "Check your email…"}. The
  OIDC-mode-disables-password-registration gate now only fires for the password path (email-only signup is still allowed even in OIDC-only mode, because it doesn't store a password).
  - magic_link_handler::redirect_target — new 3-way decision tree:
    - Resource target (folder invitation) → /#/files/folder/{id} (existing)
    - NULL resource + is_external = false → /#/files (the welcome path for new internal users — they have a home folder)
    - NULL resource + is_external = true → /#/sharedwithme (the existing external-user landing)

  Tests
  - New tests/api/registration.hurl with 9 requests covering: classic (with-password) register → 201 + UserDto, email-only register → 200 + uniform message + welcome magic-link captured, redemption → 302 to /#/files + cookies set, profile read → username
  absent + is_external: false, resend magic-link works (eligible while passwordless), cleanup deletes both new users.
  - Wired into tests/api/run.sh right after auth_login.hurl.

  Plan additions
  - auth-simplification.md gained PR 22 at the bottom of the PR sequence — device-bound magic-link redemption via challenge cookie + asymmetric TTLs (login: 10 min, invitation: 24 h). Full design recap, schema migration, config knobs
  (OXICLOUD_MAGIC_LINK_LOGIN_TTL_MINUTES / _INVITE_TTL_HOURS), and Hurl coverage outline are in the plan. Slots in before PR 21's docs so the architecture page describes the final state from the start.

  Checks — cargo fmt, cargo clippy --all-features --all-targets -- -D warnings, cargo test --lib (297 passed), biome, stylelint, tsc, full Hurl suite (16 files) all green.
2026-06-03 00:35:25 +02:00
Edouard Vanbelle d57a50d056 feat(username|email): pass1: normalize auth.user data
username: now optional, if defined 2..64 chars
    password: now optional (no mode __NO_PASSWORD...__)
    oidc: now optional

    important: if need Nextcloud, username must be defined
2026-06-03 00:35:25 +02:00
Edouard Vanbelle 803849fe2e feat(user): map family/given name to OIDC & cardav 2026-06-03 00:32:00 +02:00
Edouard Vanbelle 6d70a7000e security(external_user): protect unnecessary route access to external users 2026-06-03 00:32:00 +02:00
Edouard Vanbelle 21c06da700 feat(magic-links): add rate limiting + archirecture documentation 2026-06-03 00:32:00 +02:00
Edouard Vanbelle 64d081ad0b feat(external): permit login via email (magic link) 2026-06-03 00:31:59 +02:00
Edouard Vanbelle ec72374651 feat(api): can grant external user (via email)
- add possibility to grant an external user.
    - route /api/users/{id} added (rate limited for security)
    - security: start route limitation for external users
        ex: they must not browse /api/users/{id} nor addressbook
2026-06-03 00:31:59 +02:00
Edouard Vanbelle 03f63ad103 feat(external users): email sanity + mock SMTP
- SMTP has a mock to enable end to end test and validate the whole path
     (via OXICLOUD_SMTP_MOCK)
    - add email normalisation ( including punicode)
    - api to share to external user
2026-06-03 00:31:59 +02:00
Edouard Vanbelle d03b9474c6 feat(smtp): add a precious SMTP test for admin only 2026-06-03 00:31:59 +02:00