Commit Graph

68 Commits

Author SHA1 Message Date
Edouard Vanbelle 3b31b8911b feat(breadcrumb): build breadcrumb in 1 API call
add /api/folders/{id}/ancestors

    this API to iterate parent up to the drive root or the shared folder
    this will help UI to build the breadcrumb in 1 API call
    and to identify the root element (is it a drive users has access to or
    a shared folder ?)

    ui: now only 1 API call is now required to build the breadcrumb
2026-07-27 01:33:09 +02:00
Edouard Vanbelle 53077e0fa3 fix(merge): fix merge of mount + is_favorite feature
fix the merge of moun feature and is_favorite + is_shared feature
    together it was raising an impossible build due to missing schema
2026-07-22 08:30:06 +02:00
Dionisio Pozo ef3e3f45b3 Merge pull request #521 from BCNelson/feat/external-file-mounts
feat(mounts): external file mounts — pluggable provider, read-write, WebDAV/NextCloud, admin UI
2026-07-22 06:52:43 +02:00
Edouard Vanbelle d9987782c4 feat(fileDto, folderDto): add is_favorite + is_shared
- provide is_shared and is_favorite information in DTO, information propagated as badge/buttons per items
      regarding performances I try to be minimalis on SQL to prevent any perf regression

    - remove old set of sharedids and favoriteids (was not functionnal anymore)
    - fix date picker (no past selection) in grant
    - fix contextmenu close on /files section
2026-07-22 01:55:10 +02:00
Bradley Nelson 9f4bc54113 Merge upstream/main into feat/external-file-mounts
Resolve conflicts between the external-file-mounts feature and upstream's
D5/D7 refactor (per-file provenance, keyset pagination, cross-drive move
gates, resource-access hook, folder-cascade lifecycle hook).

Key resolutions:
- FolderService::new now takes (repo, authz, file_lifecycle, mount_router);
  all callers + DI updated.
- FileRetrievalService / FileManagementService keep both the mount_router
  and the new resource_access_hook / drive_repo / storage_usage wiring.
- list_files_batch_with_perms: adapt the mount branch from offset- to
  keyset (after_name) pagination, mirroring paginate_mount_entries.
- download_file_impl: keep upstream's &HeaderMap + `impl IntoResponse + use<>`
  signature, retain the mount-download branch.
- Mount DTOs: the retired `owner_id` field maps onto created_by/updated_by
  (the mount owner) — the fields the frontend now uses for owner display.
- admin/+page.svelte: keep upstream's user-delete modal + the 'mounts' tab.
- Bump memmap2 0.9.10 -> 0.9.11 (RUSTSEC critical advisory fix) and
  regenerate Cargo.lock against the merged Cargo.toml.
2026-07-21 17:09:36 -06:00
Claude e8e4ef4b15 perf(round25): in-place encrypted decrypt, dedup hash move, dead folder-Query, playlist N+1 fold, contact vcard over-fetch
Five benchmark-gated optimizations from a fresh six-way audit (benches/ROUND25.md),
each with a BEFORE/AFTER gate that rolls back if AFTER does not beat BEFORE:

- M1 EncryptedBlobBackend::decrypt_bytes: replace split_off (a fresh Vec + full
  ciphertext memcpy on every decrypted chunk, contradicting its own "in place"
  doc) with in-place detached decrypt + a zero-copy Bytes::slice past the nonce.
  Peak RAM per read drops from ~2x to ~1x the payload (-262KB/op at 256KiB;
  scales with blob size). Plaintext byte-identical; tamper/wrong-key tests pass.
- M2 delta commit: move-unzip the owned chunk list instead of a third
  per-occurrence hash clone (-4000 allocs on a 4000-chunk commit).
- M3 folder ZIP download: drop the dead Query<HashMap> extractor it never read
  (byte-identical response; 5->0 allocs/request).
- Q1 public-playlist listing: fold the per-playlist COUNT(*) N+1 into one
  LEFT JOIN ... GROUP BY via a new inherent repo method (101 -> 1 round-trips,
  36x wall on a 100-playlist page).
- Q2 contact REST listings (paginated/search/by-group): stop over-fetching the
  multi-KB vcard TEXT the ContactDto discards, via a shared lite row mapper and
  narrowed SELECTs (6.4x wall on 1000 contacts with 8KiB vcards). The whole-book
  vCard export and CardDAV sync paths keep the column.

Adds bench_round25_micro (counting allocator tracking count+bytes) and
bench_round25_queries (live Postgres), both with equivalence gates and a
rollback exit(1). Verified: cargo fmt clean, cargo clippy -D warnings clean,
cargo test --lib --features bench = 529 passed / 0 failed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L8gs91AhmazoxMsDcNk3KT
2026-07-21 00:07:44 +00:00
Edouard Vanbelle 931e27d09c fix(resources): wire missing created_by and updated_by 2026-07-20 21:04:32 +02:00
Claude 50eca0627f perf: round 12 — auth write-path narrowing, fused quota gate, moka blob-cache index, media single-read, sized listing JSON
Benchmark-gated round (benches/ROUND12.md; every change ships with a
BEFORE/AFTER harness + equivalence gates, one candidate rejected by its
own bench):

DB / query shapes (bench_round12_queries):
- NC sharee search: username-only projection instead of the 21-column row
  (incl. the <=512 KiB avatar) per match, + gin_trgm_ops indexes on
  auth.users for the leading-wildcard ILIKE (4.98x; 54.7x with index).
- Password login: delete the redundant full-row update_user — create_session
  already stamps last_login_at in its own txn (4.45x per login).
- Email-verified stamp: narrow conditional UPDATE (8.9x); OIDC repeat login
  now compares profile state in memory and issues ZERO queries when nothing
  changed (was: full 17-column rewrite per login).
- Refresh rotation: revoke+insert+stamp fused into one transaction via new
  rotate_session port method (1.18x).
- WOPI CheckFileInfo / authorize_wopi_access: require(Read) + get_file +
  check(Update) overlapped with tokio::join!, original result precedence
  (cold 1.34x).
- Upload quota gate: user-envelope + drive-cap checks fused into ONE
  round-trip (check_upload_quotas) — the NC chunked PUT pays this per
  chunk (1.81x, 2 -> 1 queries/chunk); shared verdict evaluators keep
  error shapes byte-identical.

CPU / allocs (bench_round12_micro):
- sized_json: pre-sized listing serialization replacing axum Json's 128 B
  seed + doubling-realloc chain on files/folder-resources/photos/search
  responses (1.40x, 13 -> 2 allocs per 500-row page; byte-identical).
- Security headers: 4 SetResponseHeaderLayer folded into the CSP middleware
  pass (5 layers -> 1; 1.43x per request, -26 allocs; header set gated
  byte-identical incl. 304s).
- Media capture-metadata: single-read extraction — nom-exif now parses the
  buffer kamadak already read (zero-copy Bytes) and videos open once with a
  kind() dispatch; per-image opens 2-3 -> 1 (1.44x warm geomean, 1.6-3.2x
  cold cache; extraction outputs gated identical incl. the MIME-mislabel
  track fallback).
- Chunked-upload session ops: owner gate folded into the operation's own
  DashMap lookup + stack-encoded uuid compare (5 -> 3 lookups, -2 allocs,
  1.28x per chunk).

Blob cache (bench_blob_cache_index + round-3 regression guard):
- CachedBlobBackend index: tokio::sync::Mutex<LruCache> -> moka::sync::Cache
  with byte weigher. The mutex serialized every cached chunk read and scaled
  NEGATIVELY (2.08 -> 1.07 Mops/s from 1 -> 2 readers); moka probes are
  lock-free (2.17x at K=2). Byte budget now enforced by moka (manual
  current_size + collect_evictions machinery deleted); eviction listener
  unlinks size-evicted files only (Replaced entries keep their file —
  gated). Single-flight miss gate unchanged (16 concurrent misses -> 1
  fetch re-verified via the round-3 harness).
- put_blob now populates the cache BEFORE the inner backend consumes the
  source file (the old order failed 100% of the time — local renames,
  S3/Azure delete the source — so the first read after a whole-file put
  re-downloaded from the remote); inner-put failure invalidates the entry.

Frontend (vitest gates):
- List-view thumbnails request the 150px icon rendition instead of 400px
  preview into a 40px slot (~7.1x fewer pixels, ~4-5x fewer bytes per
  thumbnail across list views); grid keeps preview.

Rejected by its own bench (kept as evidence in bench_round12_micro §2):
- Single-pass compression predicate: the monomorphized And-chain already
  costs ~4.6 ns / 0 allocs total; the fused node measured within noise.

New migration: 20260719000000_users_search_trgm.sql (trgm indexes).
Deferred with prepared design: grouped file/grid view virtualization
(single-VirtualRows flatten, the photos pattern) — next round's headline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfidAJD5AHw23jtvBUNamB
2026-07-19 01:32:00 +00:00
Claude 221c1f31b0 perf: round 11 — StoragePath joined-only, classifier fusion, memoized bodies, query-shape pack, SPA fine-grained stars
Backend (each change benchmark-gated with BEFORE replicas + equivalence
gates; see examples/bench_round11_micro.rs, bench_round11_queries.rs,
bench_log_writer.rs and benches/ROUND11.md — final numbers land in the
follow-up doc commit):

- StoragePath re-representation: single canonical joined String, segments
  derived on demand; File/Folder drop the duplicated path_string field
  (4000→1000 allocs per 500-row listing page)
- Display classifier fusion: classify_display shares one stack-lowered
  extension across the three decision trees; call sites in FileDto,
  folder/favorites/recent handlers, trash, path-resolver (+ interning
  where Arc::from was still used)
- /status.php and /openapi.json memoized into OnceLock<Bytes> (openapi
  rebuilt a 171 KiB spec per request: 2.8 ms → 18 ns)
- NC upload-session PROPFIND: write! + pre-sized body + stack RFC2822
  dates (2.3-2.6x, 2582→772 allocs at 256 chunks)
- REST download: dead FileDto clone removed (capture mime/size + move)
- CalendarEventDto/TrashedItem into_parts moves (11 KiB ical_data memcpy
  gone per CalDAV row); CardDAV getlastmodified stack render
- 4xx path: borrowed ErrorResponse serialize, ErrorKind::as_str,
  not_found/already_exists clone kill
- vCard emit via write!; search page moved out with into_iter skip/take;
  content-hit UUIDs parsed once; group last-user check via HashSet
- RateLimiter: lock-free get + insert (and_upsert_with variant REJECTED
  by benchmark); CSRF token borrow-compare + borrowed cookie extraction
- Thumbnail/preview ETags built from as_str (Debug-identical bytes)
- Encrypted backend: encrypt_in_place_detached single-buffer write path,
  chunk-sized reserve in collect_stream; retry labels made lazy
- PG: deferred upload registration 3→1 round-trips (persist_file CTE
  template); direct_grant_cache for Calendar/AddressBook/Playlist authz
  (single-flight + set_role/clear_role invalidation); expand_user
  tokio::join!; geo clusters min(uuid)::text; recluster face assignment
  batched into one UNNEST update
- People recluster cosine: norms precomputed once (bit-identical gate)
- NC capabilities poll logs demoted to debug; tracing-appender dep added
  for the log-writer benchmark

Frontend:
- ResourceList.selectedEntries O(N)-per-toggle → id-index projection
  O(k log k); favorites/recent consume the batchToolbar snippet param and
  drop their duplicate filter + dead selectedIds mirror
- Recent: star state via new favoriteIds prop — a star click no longer
  rebuilds all N entries
- admin timeAgo >30d fallback uses the cached Intl.DateTimeFormat
- vitest gates in src/lib/components/round11.bench.test.ts

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABhTEHuGujvwoodh67Kga7
2026-07-18 22:02:00 +00:00
Claude fdf445d2b0 perf: round 9 — decorator PUT reactivation, session/search/dedup alloc purges, PROPFIND join!, folder-level cascade
Benchmark-gated round (benches/ROUND9.md): every change carries a
BEFORE/AFTER bench with equivalence/safety gates; verdicts below are from
the committed harnesses on 4 cores / local PG 16.

Backend:
- Blob decorators (Retry/Cached) now forward put_blob_from_bytes_unsynced
  + sync_blobs — the trait default had silently reinstated HEAD-before-PUT
  per chunk on decorated remote stacks, undoing ROUND3 §8. Full production
  stack: 500 probes -> 0, 1.9x wall at 10 ms RTT (bench_s3_put §3).
- NC PROPFIND per-page enrichment triple (favorites / oc:fileid / dead
  props) overlapped with tokio::join!: 2.07x local, 2.86x at 5 ms RTT
  (bench_nc_enrich_join, injected-latency decide-by-bench).
- Search enrichment consumes its DTOs and carries the interned Arc<str>
  display fields end-to-end (SearchFileResultDto type change, OpenAPI
  shape preserved): enrich_file 2.0x, 11.6 -> 2.2 allocs/row; the NC
  REPORT conversion stops re-running all three classifiers per row
  (bench_search_enrich).
- NC session Arc end-to-end: SharedNcSession extractor (8 -> 0 allocs),
  Arc<FolderDto> chroot cache (4 -> 0/hit), single shared Arc<CurrentUser>
  + lazy span render (11 -> 6/build) (bench_nc_session).
- Storage micro-pack: atomic create_new chunk writes (2.1x fresh),
  stream_chunks over the manifest Arc (4097 -> 0 allocs/read incl. the
  Range path), manifest single-flight (herd 64 -> 1 loads), hex_lower for
  chunk Content-MD5 (18 -> 1 allocs) (bench_storage_micro).
- OCS capabilities memoized into OnceLock<[Bytes;2]>: 237x, 102 -> 0
  allocs/poll, byte-identical (bench_capabilities_static).
- Drive::is_empty COUNT(*) sum -> EXISTS: 34.4x on a 100k-file drive
  (bench_drive_is_empty).
- favorites/recents row-map ROUND7 port: path/name/blob_hash moved,
  -2.75 allocs/row (bench_resource_row_map §2).
- Folder rows decode binary UUIDs (ROUND6 §10 port): 1.03-1.07x page
  fetch, honest verdict incl. one noise-band wash documented
  (bench_folder_uuid_decode).
- Authz: file cascade decision decomposed into memoized folder-level
  decision + direct-grant lookup (ROUND8 deferred item). Cold shared-album
  first view 592 -> 418 µs/thumb; warm path unchanged; safety gates incl.
  new direct-grant sibling isolation, revoke-flush re-verified, full
  integration authz suite green (bench_thumbnail_cascade_cache).

Frontend (vitest gates committed beside the code):
- resolveLabel/resolveRecipient O(directory) scan -> id-keyed Map: 13.9x
  (recipients.bench.test.ts).
- ResourceList selection-prune effect skips when nothing is selected
  (100 -> 0 Set builds per drain) and the photos timeline reads a
  listener-fed mobile flag instead of matchMedia per recompute
  (listDerives.bench.test.ts).

Verification: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 524 unit + 554 integration (--cfg integration_tests) tests pass;
frontend npm run check clean with 293 vitest tests green.

Deferred with rationale in ROUND9.md: CalDAV authz-before-fetch reorder
(maintainer sign-off), per-page batched parent resolution, JWT-claims
Arc<str>, batch_operations signature widening.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XDc9VtXvskJ6dnMRraSndn
2026-07-18 16:12:04 +00:00
Claude 7626dc95c1 perf: round 7 — photos timeline O(N²)→incremental, range-seek authz duplication, resources row-map clone
Benchmark-gated (equivalence + BEFORE/AFTER; results + reproduce commands in
benches/ROUND7.md):

- Photos timeline re-grouped + re-laid-out the whole accumulated library on
  every 60-item page (both `groups` and `photoRows` were $derived over the
  full list), Σ ≈ O(N²/60) main-thread work during a scroll. Pages arrive
  newest-first so grouping is append-only: the new PhotoTimeline
  (lib/utils/photoTimeline.ts) re-buckets only the fresh page and re-lays-out
  only changed groups, reusing untouched groups' cached rows, falling back to
  a full rebuild on any config/deletion/non-append change. The pure
  buildPhotoRows is the verbatim reference the gate holds it equal to at every
  page. 50×60 drain: 76 500 → 3 000 grouping ops (25.5x), 23.0 → 2.2 ms
  (10.6x).

- Range downloads paid authz + access-notify twice: download_file_impl
  resolves the file via get_file_with_perms, then the Range branch re-ran
  require_file + notify_file_accessed per request. Media/PDF viewers fetch
  exclusively via Range (one request per seek), so every seek in a scrub
  re-authorized an already-cleared file. Now routed through the non-perms
  get_file_range_preloaded (matching the share-landing + WebDAV range paths);
  the unused _with_perms range method is removed. The request-level gate still
  denies before the branch runs (bench asserts member granted, outsider
  denied). Per seek removed: WARM 0.67 µs, COLD 1362.66 µs — a grant-cascade
  drive-resolve query per seek for a shared-drive recipient on a cold cache.

- /api/folders/{id}/resources row→DTO mapping cloned row.name into the DTO
  though the row is owned; folders move it (fixed icons), files compute the
  name-derived icon/category classes first then move it. 500-row page:
  10.004 → 9.004 allocs/row (500 clones removed), output identical.

Deferred with rationale in ROUND7.md: thumbnail ACL-before-304 (security
posture — needs a security review, not a perf tweak), batch_operations
Arc<str>→String widening, list-view O(N²) on smaller lists, and the serial→
join! pairs (decide-by-bench with injected latency, per the round-6 rejection).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-18 13:11:41 +00:00
Claude 63cf6646d0 perf: round 5 — CalDAV cursor streaming, SPA interning gaps, NC href prefix, per-request micro-allocs
Seven benchmark-gated changes (benches/ROUND5.md; BEFORE/AFTER bench +
equivalence gate each, rollback rule as ROUND2-4 — two intermediate
CalDAV shapes measured worse and were themselves rolled back before
shipping):

- CalDAV whole-calendar responses (REPORT no-range/sync-collection,
  depth-1 collection PROPFIND, .ics GET): buffered double-residency →
  ONE window-ordered scan (MIN(start_time) OVER (PARTITION BY ical_uid))
  streamed through a PG cursor, pages cut at UID boundaries. TTFB
  23.3→11.0 ms (2.1x), peak heap 14.2→8.0 MiB at 4k events / 45→24 MiB
  at 12k, wall +9-15% (documented trade, ZIP-streaming class); both
  multistatus and ICS byte-identical to the buffered output. Rejected
  shapes kept in the doc: per-page GROUP-BY keyset (3-4x wall) and
  per-uid ANY hydration (~20 µs/index descent).
- SPA listing interning gaps: folder/recent/favorites resources handlers
  (and the WebDAV pseudo-root) called raw Arc::from per row for the
  closed display set ROUND3 interned — now intern_display/intern_mime,
  4→0 allocs/row, byte-identical Arc contents.
- NC PROPFIND child hrefs: username + parent path encoded once per
  request instead of per child (543→165 ns/row, 13→4 allocs); native
  WebDAV href drops its intermediate encode String.
- suggest enrichment: entity clone + field re-clones per keystroke row →
  consume + move (166.5→126.8 µs/200 rows, 20→7 allocs/row).
- list_readable_by returns the cache's Arc (246→128 ns warm hit, 4→0
  allocs) — deep Vec clone per DAV-selector request removed.
- CardDAV REPORT: borrowed props, reused href buffer, exact-size etag
  quoting (3.04→2.34 ms per 5k-contact getetag poll).
- Auth span records: user_id.to_string() per request ×3 →
  tracing::field::display.

Checks: cargo fmt, clippy --all-features --all-targets -D warnings,
cargo test --workspace (523 passed). Follow-ups (CardDAV streaming,
&[&str] id batches, ::text UUID casts A/B, share-landing join) recorded
in benches/ROUND5.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
2026-07-17 15:19:00 +00:00
Claude 82ee7da0d2 perf: serve ranges from RAM cache, stream ZIPs, overlap ingest settle, O(1) chunk gate
Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):

- Range requests (REST/DAV/shares) answered from the moka content cache
  for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
  256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
  with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
  Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
  per-session counter (lazy rebuild on cold start). 1,000-chunk upload
  gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
  buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
  latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
  the source stream keeps flowing during PG pin + backend writes;
  rollback ledger shared + lock-serialized so compensation stays exact
  on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
  OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
  a bounded Web Worker pool (File handles by reference); vitest gate
  asserts the pool beats sequential (first gate draft posting buffers
  was 2.6x slower and was rewritten — copies dominated).

Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
2026-07-16 16:50:07 +00:00
Edouard Vanbelle 37467ed9d3 feat(drive): remove all owner_id from {File,Folder}Dto 2026-07-03 01:13:53 +02:00
Edouard Vanbelle 206a780494 feat(drive): stop writing in {file,folder}.user_id
plan:

    ```
    What step 5 is actually doing

    Goal: stop writing to storage.files.user_id / storage.folders.user_id on new INSERTs. This is a prerequisite for step 6 (dropping the column entirely). The column has existed since the pre-D0 single-owner era; every read that used to key on it has already been migrated to drive-membership grants over the last N days.

    What has to change together for this to be safe:

    ┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────┐
    │    Piece     │                                                          What changes                                                           │                                                  Why                                                  │
    ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
    │ Migration    │ Swap storage.files uniqueness indexes from (folder_id, name, user_id) to (drive_id, folder_id, name)                            │ Otherwise, two new rows with user_id = NULL would both be allowed (PG treats NULLs as distinct) —     │
    │ (a)          │                                                                                                                                 │ uniqueness silently breaks                                                                            │
    ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
    │ Migration    │ ALTER user_id DROP NOT NULL on both tables                                                                                      │ Otherwise, dropping the INSERT bind violates NOT NULL and every write 500s                            │
    │ (b)          │                                                                                                                                 │                                                                                                       │
    ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
    │ Migration    │ Drop dead user_id-leading indexes                                                                                               │ Cheap cleanup — nothing scans them anymore                                                            │
    │ (c)          │                                                                                                                                 │                                                                                                       │
    ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
    │ PL/pgSQL (d) │ Rewrite storage.copy_folder_tree without user_id in the INSERT column list                                                      │ Cross-drive copy runs entirely in SQL, needs the same treatment                                       │
    ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
    │ Rust (e)     │ ~12 INSERT column-list drops across folder_db_repository, file_blob_write_repository, drive_pg_repository, dedup_service,       │ Actual write path                                                                                     │
    │              │ folder_service, load-seed.rs                                                                                                    │                                                                                                       │
    ├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
    │ Rust (f)     │ Simplify folder_db_repository::create_folder's parent lookup to only fetch drive_id (was fetching (user_id, drive_id))          │ It's fetching a value it no longer needs                                                              │
    └──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
    ```
2026-07-03 00:23:35 +02:00
Edouard Vanbelle f5f5b1167f feat(drive): remove _for_owner() and use authz
remove related IDOR protection as Hurl tests are covering this surface
2026-07-02 01:14:02 +02:00
Edouard Vanbelle 583af517c2 fix(drive): fix drive_id on copy to other drive
- fix alsot drive_id exposition
2026-06-29 23:21:17 +02:00
Bradley Nelson 8e3e31da4d feat(mounts): P2 — read-write REST for external mounts
Adds full CRUD on mount contents, mirroring the P1 read pattern (handlers/
services classify; authorization stays in the service via the mount-root
folder grant; the provider does the I/O).

- mkdir / rename / delete / move-within branch inside FolderService and
  FileManagementService (router injected into both)
- streaming upload via a new ExternalUploadService: the upload handler detects a
  mount destination BEFORE the CAS ingest and streams the multipart body
  straight to the provider (no BLAKE3/dedup). `write_stream` now takes a
  lifetime-bound boxed stream so the borrowing multipart field can be passed
  without buffering.
- deletes on mounts are permanent (no trash): the trash-first folder handler
  routes `ext:` ids straight to the provider delete; file delete goes through the
  branched delete_and_cleanup
- cross-backend move/copy (mount ↔ native, or between mounts) is forbidden
  (UnsupportedOperation); the mount root itself cannot be renamed/moved/deleted
- every mutation emits a `target:"audit" event="external_mount.write"` line
- shared mount_dto builders synthesize FolderDto/FileDto from a provider MountStat

Tests: 529 unit + integration tests for mkdir/rename/delete, file rename/delete,
streaming upload, cross-boundary forbid, and stranger-denied — all against real
Postgres + a real provider (testcontainers).
2026-06-25 00:30:10 -06:00
Bradley Nelson 3c31695579 feat(mounts): external file mounts P1 — pluggable provider + read-only REST
Adds the foundation for external file mounts: admin-configured backends
(raw host filesystem in v1; sftp/webdav/… as future provider kinds) surfaced
as a folder inside a user's drive. Mount contents are virtual/live-passthrough
— read straight from the backend, never stored in storage.files — and are a
deliberately separate, limited storage type (no dedup/sharing/trash/search).
The feature is dark by default (OXICLOUD_ENABLE_EXTERNAL_MOUNTS=false).

P1 scope (this PR): data model, the pluggable provider abstraction, and the
read-only REST surface (mount listing + download). Read-write (P2),
WebDAV/NextCloud path resolution (P3), and the admin UI (P4) follow.

Core model
- Mount root = a real storage.folders row; authorization for everything inside
  collapses onto that folder UUID (ltree-ancestry grant cascade).
- Children are virtual, addressed by ext:<mount_id>:<base64url(node_id)> where
  node_id is provider-owned and opaque to the rest of the system.
- A lock-free (arc-swap) MountRegistry maps mount-root UUID -> provider; a thin
  MountRouter::classify() is the single cheap hook handlers call before parsing
  an id as a UUID. With no mounts configured it always returns Regular, so
  existing code paths are unchanged.

Added
- migrations/20260805000000_external_mounts.sql (storage.external_mounts, kind + config JSONB)
- domain/services/external_mount_id (id envelope + virtual etags)
- application/ports/external_mount_ports (ExternalMountProvider, MountProviderFactory, repo port)
- infrastructure local_fs_mount_provider (tokio::fs, symlink-escape-safe) + factory
- application MountRegistry + MountRouter, pg ExternalMountRepository
- DI wiring (AppState.mount_router), FeaturesConfig.enable_external_mounts
- listing branch (FolderService::list_mount_dir_with_perms + folder_handler) and
  download branch (FileRetrievalService stat/open mount methods + file_handler)

Authorization stays in the service layer (authz.require(Resource::Folder(mount_id)));
handlers only classify. Cross-backend operations are out of scope for P1.

Tests: 529 unit tests + 5 testcontainers integration tests (real Postgres 17),
including end-to-end authorization (owner allowed, stranger denied). Line
coverage of the new modules is 84–100% (cargo-llvm-cov). Known gap:
file_handler::download_mount_file (HTTP glue) needs a full-app test (P4).
2026-06-24 23:52:01 -06:00
DioCrafts 8981c1dfb9 refactor(api): remove 5 dead routes + stale deprecation markers
An audit (backend /api routes vs SvelteKit frontend usage, adversarially
verified across the whole repo) found these 5 routes have ZERO callers — no
frontend, no test, no protocol layer, no internal caller — and are superseded:

- GET  /api/folders/paginated            no-op duplicate of GET /api/folders
                                         (discards the page arg); superseded by
                                         the cursor-paginated /{id}/resources.
- POST /api/dedup/upload                 superseded by /api/files/upload, which
                                         does the identical CDC dedup ingest.
- POST /api/people/{id}/hide             the hide-person toggle was never built
                                         into the UI (is_hidden never read).
- GET  /api/admin/settings/general       never called anywhere.
- GET  /api/admin/settings/registration  only the PUT is used; the GET had no
                                         caller (PUT kept).

Removes each route, its handler + _impl, the now-orphaned DedupUploadResponse
DTO + its two serialization tests, the set_hidden service method (only caller
was hide_person), and the OpenAPI path/schema registrations.

Also cleans 4 stale markers: two #[allow(deprecated)] that no longer suppress
anything (zero #[deprecated] remain), the "Legacy folder endpoints (contents,
listing)" comment (both already removed), and a "Re-export AppError for backward
compatibility" comment describing a re-export that doesn't exist.

Net -323 lines. The 31 other unused-by-frontend routes (device-code auth,
CardDAV contact-groups, people/photos & music WIP, dedup/admin debug, i18n,
openapi.json) are intentional surface and were left untouched.

cargo clippy --all-features --all-targets -D warnings: clean. cargo test: 446 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 11:35:09 +02:00
DioCrafts 54639d466a chore(frontend): remove the legacy vanilla-JS frontend and its tooling
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).

Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
  to frontend/static/ so they ship with the SPA. This also fixes the
  favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
  to the SvelteKit /nextcloud/error route.

Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
  the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
  fallback so `just dev` works without a prior build.

Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
  pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).

Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
  scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
  and drop check-contrast/check-headings (coupled to the old token
  taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.

Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
  (superseded by /resources).

Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 03:20:10 +02:00
DioCrafts 2b5339b73e refactor(api): remove 5 deprecated list endpoints superseded by /resources
The normalized cursor-paginated /resources API is live and the bundled
frontend already uses it for favorites, recent, trash and grants. These five
deprecated old-format endpoints had no remaining frontend or protocol
consumers (the Nextcloud handlers call the service layer directly, not these
HTTP routes), so remove them for a uniform API and less duplicate listing
logic:

- GET /api/folders/{id}/contents          → use /api/folders/{id}/resources
- GET /api/folders/{id}/contents/paginated → use /api/folders/{id}/resources
- GET /api/favorites                       → use /api/favorites/resources
- GET /api/recent                          → use /api/recent/resources
- GET /api/trash                           → use /api/trash/resources

Removes the HTTP handlers, their routes, OpenAPI path registrations, and the
now-dead list_folder_contents{,_paginated}_impl helpers + unused imports. The
underlying service methods (favorites_service.get_favorites,
trash_service.get_trash_items, etc.) are KEPT — the Nextcloud OCS/trashbin
handlers depend on them.

Deliberately NOT removed: GET /api/folders/{id}/listing. It is still the Files
view's primary data path and offers ETag/304 conditional caching plus
one-shot favorite/share badge sets that /resources does not yet provide;
migrating it needs a separate parity pass on /resources first.

Updates the OpenAPI structure test and the two docs that referenced the
removed paths. `cargo clippy -D warnings` clean; 443 lib tests pass; OpenAPI
regenerates with the 5 paths gone and the /resources replacements present.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 23:46:58 +02:00
Dionisio Pozo ddfc481616 Merge pull request #479 from EdouardVanbelle/feat/drive-impl
feat/drive impl
2026-06-19 18:48:32 +02:00
Claude 9ccaeef0ab perf(listing): return per-item is_favorite/is_shared, drop client badge fetches
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.

Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
  level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
  concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
  `PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
  which already covers public links as 'token' grants — same membership the
  /grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).

Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
  badge sets straight from the listing and updates them optimistically on
  favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
  the single, authoritative, fetch-free source.

Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
2026-06-19 15:21:01 +00:00
Edouard Vanbelle e7f4826778 feat(drive): complete updated_by created_by 2026-06-19 16:06:38 +02:00
Edouard Vanbelle 50fb34659e feat(drive): limit NC request by disk 2026-06-19 16:06:38 +02:00
Edouard Vanbelle bde7c83932 feat(content_hash): propagate etag and hash_content to */resources 2026-06-06 20:11:13 +02:00
Edouard Vanbelle 0135930da9 refactor(file|folder): separate etag and blob_hash 2026-06-06 18:49:14 +02:00
Edouard Vanbelle 8b4edb1189 fix(openapi): expose route missing /api/.../resources to openapi 2026-05-31 20:58:01 +02:00
Edouard Vanbelle 3f70c7e20a refactor(server): remove sort by size in pg_acl_engine.rs (dead code) 2026-05-29 13:14:16 +02:00
Edouard Vanbelle 6e30353c4b chore(deprecated api): log a warning on serverside if a deprecated route is still used 2026-05-29 13:13:23 +02:00
Edouard Vanbelle 5790a1459f feat(folders): add curser and the normalized way to get foler's item list. add reverse order 2026-05-28 12:12:22 +02:00
Edouard Vanbelle dfb21b746d feat(openapi): show explicitly path requiring bearerAuth 2026-05-28 00:48:20 +02:00
Edouard Vanbelle 3362e277ab feat(authz): check permission on read handlers + check create permission on folder 2026-05-21 20:30:53 +02:00
Edouard Vanbelle dfb082fdf4 refactor(server): file_management_service: move all method without owner check into private, add folder_ports 2026-05-20 15:39:53 +02:00
Edouard Vanbelle f8b30e78a6 refactor(create_folder): add an ownership check while creating a folder + refactor code 2026-05-20 12:59:04 +02:00
Edouard Vanbelle d0c6bd6a73 feat(openapi): upgrade code to gnerate openapi on all path 2026-05-04 12:02:33 +02:00
Edouard Vanbelle badf35f08f chore: remove all executable attributes on non bash files 2026-04-01 23:14:42 +02:00
Diocrafts 06ed0455ce perf: migrate all user/session/auth IDs from VARCHAR(36) to native UUID
- Schema: all ~15 VARCHAR(36) columns → UUID with DEFAULT gen_random_uuid()
- Domain entities: User, Session, DeviceCode, AppPassword, Share → id: Uuid
- DTOs: CurrentUser.id → Uuid (API boundary DTOs keep String for JSON)
- Auth middleware: parse JWT claims.sub (String) → Uuid at boundary
- All repository traits, port traits, service impls updated end-to-end
- Handlers: pass Uuid by value (Copy, 16 bytes) instead of String refs
- Settings chain: updated_by column → Uuid (was text, caused setup crash)
- Removed ~650 lines of String↔Uuid conversion boilerplate
- Eliminates per-request heap allocations for ID cloning
- 16-byte binary comparison vs 36-byte string comparison in all queries
- Native UUID indexing in PostgreSQL (btree on 16 bytes vs 36-char text)

85 files changed, 1090 insertions(+), 1739 deletions(-)
2026-03-07 14:59:32 +01:00
Dionisio f2d35ca792 feat: auto-persist JWT secret, remove setup token requirement
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
2026-03-05 22:12:53 +01:00
zjean 190527edfb style: apply rustfmt formatting to fix CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 21:28:51 +01:00
Dionisio 33cfb0faef fix: security audit — patch vulnerabilities V-02 through V-16
- V-02: XSS via innerHTML in profile.js — wrap err.message in escapeHtml()
- V-03: IDOR upload to other users' folders — add folder ownership check
- V-04: IDOR create folders in other users' trees — add parent ownership check
- V-06: Content-Disposition header injection — RFC 5987 percent-encoding
- V-08: WebDAV MOVE/COPY destination without ownership — add assert_owner checks
- V-09: .gitignore missing cert/key patterns — add *.pem, *.key, *.p12, etc.
- V-11: Username accepts XSS payloads — restrict to [a-zA-Z0-9._-]
- V-12: Minimal email validation — reject forbidden chars, require domain dot
- V-13: admin_reset_password doesn't invalidate sessions — revoke all sessions
- V-14: Rate limiting bypassable via X-Forwarded-For — gate behind OXICLOUD_TRUST_PROXY_HEADERS
- V-15: Cookie Secure flag off by default — default to true (safe-by-default)
- V-16: LIKE wildcard injection in searches — add like_escape() helper across 9 sites
2026-03-05 14:52:11 +01:00
Dionisio b503e08384 security: fix vulnerabilities 1-7 from security audit
- Fix #1: Share handler IDOR - enforce owner check on share operations
- Fix #2: list_files_query IDOR - bind folder queries to authenticated user
- Fix #3: Dedup handler IDOR - restrict dedup operations to file owner
- Fix #4: Trash handler OptionalAuthUser - require full AuthUser
- Fix #5: Error info leakage - sanitize 500 error responses
- Fix #6: Chunked upload IDOR - bind upload sessions to user_id,
  add verify_session_owner() check on all session operations
- Fix #7: CSP unsafe-inline removal - migrate all inline scripts,
  styles and event handlers to external files, tighten CSP to
  script-src 'self'; style-src 'self'

New files:
  - static/js/core/theme-init.js (render-blocking theme init)
  - static/js/core/sw-register.js (service worker registration)
  - static/css/views/device-verify.css (extracted inline styles)
  - static/js/views/device-verify/device-verify.js (extracted inline script)
2026-03-05 13:15:34 +01:00
Diocrafts ee86c3a128 fix: resolve all clippy warnings and convert integration_tests to custom cfg
- Add type aliases (FileRow, FolderRow, FolderRowPaginated, FolderRowOptUser) to reduce type complexity
- Simplify redundant closures in app_password_handler and webdav_handler
- Remove needless borrow in auth_handler
- Collapse nested if/let chains in login_lockout, webdav_lock, auth, rate_limit
- Box LockEntry in acquire() Err variant to fix large enum variant warning
- Rename DeviceCodeStatus::from_str to parse to avoid should_implement_trait lint
- Add #[allow(clippy::too_many_arguments)] and #[allow(clippy::result_unit_err)] where appropriate
- Convert integration_tests from cargo feature to custom cfg attribute
- Add check-cfg lint config in Cargo.toml for integration_tests cfg
2026-03-04 23:55:08 +01:00
Claude 1b49135ca9 perf: replace dyn trait objects with concrete types to eliminate vtable overhead
Remove async-trait dependency and use native Rust async fn in traits.
Replace Arc<dyn Trait> with Arc<ConcreteType> throughout the codebase
to enable monomorphization and eliminate dynamic dispatch overhead.

Key changes:
- Remove write-behind cache (no implementation existed)
- Fix should_transcode static method call
- Use ContactStorageAdapter directly instead of dyn AddressBookUseCase
- Clean up unused trait imports across services and DI

https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
2026-03-03 15:36:42 +00:00
Dionisio 97cf6402e2 fix: resolve all clippy warnings and enforce cargo fmt
- display_helpers: convert module doc-comments to regular comments,
  merge identical text/markdown + text/ branches
- search_service: replace needless range loops with slice-based pagination
- folder_repository, folder_db_repository: collapse nested if statements
- favorites_pg_repository: remove unnecessary borrow on generic arg
- file_blob_read_repository: collapse 6 nested if-let blocks
- file_blob_write_repository: collapse nested if for dedup ref decrement
- chunked_upload_service: use div_ceil(), collapse 2 nested if blocks
- folder_handler: collapse nested if-let for owner check
- webdav_handler: replace 7x io::Error::new(ErrorKind::Other, ..) with
  io::Error::other(..)
- cargo fmt applied to all files

Passes: cargo clippy --all-targets --all-features -- -D warnings
2026-02-25 10:28:34 +01:00
Dionisio 71c2cb5edb perf: Arc<AppState>, streaming PROPFIND, spawn_blocking SHA-256
- Issue #4: Wrap AppState in Arc — eliminates 42 Arc::clone + 16 String::clone per request
- Issue #2: Reject Depth:infinity with 403 + streaming XML with paginated DB queries
- Issue #5: Move chunked upload assembly (SHA-256 hash-on-write) to spawn_blocking
- Remove ~270 lines dead code from di.rs (unused builders, Default impl, stubs)
- Clean up unused tokio imports in chunked_upload_service.rs
2026-02-24 15:11:56 +01:00
Dionisio 41af7f0933 fix(zip): prevent premature temp file deletion during ZIP download
Use NamedTempFile::into_parts() to reuse the existing fd instead of
opening a second one, and store TempPath in response extensions so the
file is only deleted after the body stream finishes.

Before: temp_file was dropped when the handler returned (before Axum
streamed the body). Worked only by accident on Unix (unlinked files
remain readable while an fd is open) but used 2 fds and was fragile.

After: single fd, explicit lifetime guarantee, cross-platform correct.
2026-02-24 12:54:11 +01:00
Dionisio cba34056dc perf: remove HTTP cache middleware, add service-level ETags
- Delete cache.rs middleware that buffered entire response bodies (up to 10MB)
  in RAM on every cache miss, defeating streaming and causing memory spikes
- Also buffered non-GET responses unnecessarily via response_map_body()
- Add lightweight ETag support (based on max modified_at + count) to:
  - FolderHandler::list_folder_listing (combined folder+files endpoint)
  - FileHandler::list_files_query (file listing endpoint)
- Both support If-None-Match / 304 Not Modified without any body buffering
- File downloads already had ETag/304 support at handler level
- Service-level caches (FileContentCache, SearchService, ThumbnailService)
  remain unchanged — they handle caching without HTTP body materialization
2026-02-24 09:52:22 +01:00
Diocrafts 5b4cd30e2b fix(zip): stream ZIP to temp file instead of loading entire archive into RAM
Solution C - Hybrid temp-file streaming:
- ZipPort trait now returns NamedTempFile instead of Vec<u8>
- ZipService writes to a temp file via ZipWriter<std::fs::File> (O(1) RAM)
- Files are read in 64KB stream chunks via get_file_stream() instead of get_file_content()
- HTTP response streams the temp file via ReaderStream (never loads full ZIP in memory)
- Temp file auto-deleted on drop after response completes
- Removed dead imports (HeaderName, HeaderValue, Cursor, Read)
2026-02-22 22:29:07 +01:00