Commit Graph

142 Commits

Author SHA1 Message Date
Bradley Nelson 8e3e31da4d feat(mounts): P2 — read-write REST for external mounts
Adds full CRUD on mount contents, mirroring the P1 read pattern (handlers/
services classify; authorization stays in the service via the mount-root
folder grant; the provider does the I/O).

- mkdir / rename / delete / move-within branch inside FolderService and
  FileManagementService (router injected into both)
- streaming upload via a new ExternalUploadService: the upload handler detects a
  mount destination BEFORE the CAS ingest and streams the multipart body
  straight to the provider (no BLAKE3/dedup). `write_stream` now takes a
  lifetime-bound boxed stream so the borrowing multipart field can be passed
  without buffering.
- deletes on mounts are permanent (no trash): the trash-first folder handler
  routes `ext:` ids straight to the provider delete; file delete goes through the
  branched delete_and_cleanup
- cross-backend move/copy (mount ↔ native, or between mounts) is forbidden
  (UnsupportedOperation); the mount root itself cannot be renamed/moved/deleted
- every mutation emits a `target:"audit" event="external_mount.write"` line
- shared mount_dto builders synthesize FolderDto/FileDto from a provider MountStat

Tests: 529 unit + integration tests for mkdir/rename/delete, file rename/delete,
streaming upload, cross-boundary forbid, and stranger-denied — all against real
Postgres + a real provider (testcontainers).
2026-06-25 00:30:10 -06:00
Bradley Nelson 3c31695579 feat(mounts): external file mounts P1 — pluggable provider + read-only REST
Adds the foundation for external file mounts: admin-configured backends
(raw host filesystem in v1; sftp/webdav/… as future provider kinds) surfaced
as a folder inside a user's drive. Mount contents are virtual/live-passthrough
— read straight from the backend, never stored in storage.files — and are a
deliberately separate, limited storage type (no dedup/sharing/trash/search).
The feature is dark by default (OXICLOUD_ENABLE_EXTERNAL_MOUNTS=false).

P1 scope (this PR): data model, the pluggable provider abstraction, and the
read-only REST surface (mount listing + download). Read-write (P2),
WebDAV/NextCloud path resolution (P3), and the admin UI (P4) follow.

Core model
- Mount root = a real storage.folders row; authorization for everything inside
  collapses onto that folder UUID (ltree-ancestry grant cascade).
- Children are virtual, addressed by ext:<mount_id>:<base64url(node_id)> where
  node_id is provider-owned and opaque to the rest of the system.
- A lock-free (arc-swap) MountRegistry maps mount-root UUID -> provider; a thin
  MountRouter::classify() is the single cheap hook handlers call before parsing
  an id as a UUID. With no mounts configured it always returns Regular, so
  existing code paths are unchanged.

Added
- migrations/20260805000000_external_mounts.sql (storage.external_mounts, kind + config JSONB)
- domain/services/external_mount_id (id envelope + virtual etags)
- application/ports/external_mount_ports (ExternalMountProvider, MountProviderFactory, repo port)
- infrastructure local_fs_mount_provider (tokio::fs, symlink-escape-safe) + factory
- application MountRegistry + MountRouter, pg ExternalMountRepository
- DI wiring (AppState.mount_router), FeaturesConfig.enable_external_mounts
- listing branch (FolderService::list_mount_dir_with_perms + folder_handler) and
  download branch (FileRetrievalService stat/open mount methods + file_handler)

Authorization stays in the service layer (authz.require(Resource::Folder(mount_id)));
handlers only classify. Cross-backend operations are out of scope for P1.

Tests: 529 unit tests + 5 testcontainers integration tests (real Postgres 17),
including end-to-end authorization (owner allowed, stranger denied). Line
coverage of the new modules is 84–100% (cargo-llvm-cov). Known gap:
file_handler::download_mount_file (HTTP glue) needs a full-app test (P4).
2026-06-24 23:52:01 -06:00
Claude 6e26d1c694 refactor: remove dead OptimizedFileContent::Mmap variant
OptimizedFileContent::Mmap was constructed nowhere — the documented "Tier 2:
memory-mapped I/O (10-100 MB)" path was never wired, so optimized_inner only
ever returns Bytes (<10 MB) or Stream (>=10 MB). The variant survived only as
an enum case plus two dead match arms in the file and share download handlers.

Remove the variant and its arms, and fix the now-misleading retrieval-service
tier docs (everything >=10 MB streams via CDC chunk reassembly with the
backend read-ahead; there is no mmap tier). Behaviour is unchanged — the
deleted arms were unreachable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
2026-06-22 08:10:54 +00:00
Claude 5b8b740233 perf(blob): small read-ahead for the local backend (read_prefetch 1 -> 2)
The local backend inherited the trait's conservative read_prefetch() = 1
(strictly sequential chunk reassembly), while S3/Azure already use 8. The
prior rationale was that concurrent opens over scattered content-addressed
chunk files turn one sequential read into competing random I/O ("slower
cold"). Benchmarked that assumption with examples/bench_blob_prefetch
(sweeps the buffered(N) depth over a real LocalBlobBackend under disk-bound
vs network-bound consumers and warm vs cold page cache).

Result on SSD-class storage (median MB/s vs N=1):
  warm  disk-bound   N=2 +11.8%   N=8 +3.9%   N=16 -4.4%
  cold  disk-bound   N=2  +7.2%   (no cold regression on SSD)
  network-bound (throttled)  ~0% at any N — the socket, not the disk, caps it

So N=8 is wrong for local (leaves gain on the table, risks HDD seek thrash)
and the network-bound win the analysis assumed doesn't materialize: buffered()
here overlaps the per-chunk File::open (cheap on local disk), not the data
read. N=2 captures most of the disk-bound gain — which covers localhost/LAN
downloads AND the internal blob reads that drain as fast as the disk delivers
(thumbnail render, transcode, ZIP export, content extraction) — at the lowest
fan-out. Env-tunable via OXICLOUD_LOCAL_READ_PREFETCH (set 1 on seek-bound
HDDs to restore the old behaviour; raise on fast NVMe). Signature unchanged,
so all ~16 LocalBlobBackend::new call sites are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
2026-06-22 08:10:46 +00:00
DioCrafts 5722481c4a feat(thumbnails): server-side video thumbnails via ffmpeg
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).

- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
  (infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
  extract one representative frame as PNG, bounded by its own semaphore + a
  per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
  videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
  generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
  to a size- and time-bounded temp file on the data volume, extract a frame, and
  reuse the shared render_and_persist_all_webp helper — so video thumbnails are
  WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
  Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
  at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
  path; the server is now the source of truth.

Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.

Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 23:23:04 +02:00
DioCrafts e7b85e56e2 feat(thumbnails): WebP output with Accept content negotiation
Thumbnails are now generated eagerly as lossy WebP (the primary codec) and
served to clients that advertise `Accept: image/webp`; JPEG is kept as a lazy
fallback for older clients and NextCloud, generated on first request and then
cached like WebP.

- ThumbnailFormat{Webp,Jpeg} enum threaded through encode/render/generate, the
  on-disk path ({hash}.webp / {hash}.jpg), the moka cache key
  (file_id, size, format), and cleanup (both formats removed).
- file_handler: parse Accept -> format, format-keyed ETag, `Vary: Accept` on
  every response (incl. 304) so shared caches never serve the wrong codec;
  Content-Type is byte-sniffed (infer) so it always matches the bytes.
- preview_handler (NextCloud) pins JPEG.
- webp = "0.3" (vendored libwebp via cc, no system dependency).

WEBP_QUALITY=82, chosen via a quality sweep (bench Table E1): SSIM within
~0.005 of JPEG q80 (imperceptible at thumbnail scale) for ~62% fewer bytes. On
the photo-realistic bench corpus the full set (3 sizes x 3 photos) drops 65.6%
(213->73 KB); real photos with edges/text land nearer ~25-40%. Encode is +5ms,
paid once in the eager background generator (off the request path).

The bench corpus is now photo-realistic (per-channel sums of low-frequency
sinusoids) instead of white noise, which had distorted codec byte ratios.
Methodology + numbers in benches/WEBP.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 21:16:08 +02:00
Edouard Vanbelle e5b9a4a8db feat(drive): remove create_home_folder
remove create_home_folder() & ensure_home_folder()

    now: on_user_created() and on_user_login both() call provision_if_needed()
    which calls **create_personal_drive_atomic()**

    add a helper to find Personal drive for a user and also it's root directorry
2026-06-19 16:06:38 +02:00
Edouard Vanbelle e7f4826778 feat(drive): complete updated_by created_by 2026-06-19 16:06:38 +02:00
Edouard Vanbelle 50fb34659e feat(drive): limit NC request by disk 2026-06-19 16:06:38 +02:00
Edouard Vanbelle aa155dffa0 fix_next_cloud 2026-06-19 16:06:38 +02:00
Edouard Vanbelle 16ea08b093 feat(drive): improve Drive model
now Drive is purely a metadata
    each drive has always a root folder
    this model minimize Oxicloud changes, and simplify
    the Drive name is simply the folder's root's name
    note: owner of Drive has more permission that an owner of the root folder
2026-06-19 16:06:37 +02:00
Edouard Vanbelle eab7a609b9 feat(drive): start implementation of drive
- add storage.drives
    - prepare migration phase
    - add created_by and updated_by on storage.folders
2026-06-19 16:06:37 +02:00
Dionisio Pozo 1c9bb30475 Merge pull request #485 from AtalayaLabs/claude/zealous-faraday-58s1at
feat: Photos evolution — Places (map) & People (faces) + gallery polish
2026-06-19 14:31:14 +02:00
Claude 524e573756 feat(faces): domain entities, ports & no-op analyzer
Phase 2 increment 2:
- domain: Face, Person, BoundingBox, DetectedFace (512-d embeddings).
- ports: FaceAnalyzerPort (detect + embed from raw bytes; decodes internally
  so the application layer stays image/ML-crate agnostic) and FaceRepository
  (user-scoped face/person persistence).
- DTOs: PersonDto, FaceBoxDto.
- NoopFaceAnalyzer — reports is_ready()==false and returns no faces, so the
  whole People pipeline compiles and runs inert until the operator wires a
  real ONNX-backed analyzer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
2026-06-19 11:34:46 +00:00
Claude 9a6b174a30 Defer CDC chunk reclamation in manifest dereference to GC
remove_manifest_reference unlinked a chunk's backing file right after the
row-delete committed — the same TOCTOU the GC grace window was added to
close: a concurrent upload of identical content can re-reference (pin) the
chunk in the gap between commit and unlink, after which the deferred
unlink strands a referenced chunk with no bytes.

Route physical chunk reclamation through the single grace-protected path:
on last reference, delete the manifest and decrement its chunks (stamping
orphaned_at on the ones that reach 0), but leave the chunk rows and files
for garbage_collect() to reclaim once orphaned past the grace window. The
manifest deletion and its blob-keyed thumbnail hook stay eager.

remove_legacy_reference and cleanup_if_orphaned's legacy path are left as
eager deletes on purpose: a legacy whole-file hash can never be re-created
by an ingest (uploads are always CDC now), so there is no writer to race —
the existing "row gone ⇒ no resurrection" reasoning holds for them.

Adds an integration test asserting a CDC manifest dereference leaves
chunks orphaned-but-present, then reclaimed by a post-grace GC.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
2026-06-19 09:58:51 +00:00
Edouard Vanbelle 72129af0bd refactor(role): use grant only
- remove permission centric mode
    - finalize migration drop all tables with permissions
    - ensure roles are ENUM (owner is always displayed first)
2026-06-18 02:47:06 +02:00
Edouard Vanbelle f168c4578f feat(roles): prepare migration ReBAC to roles
prepare migration of permission to roles
    this simplify drastically database (permission are now simply defined in code)
    and will permit reuse of the same ReBAC engine to define owners of drives

    mapping:

    ```
        Role::Viewer => &[Permission::Read],
        Role::Commenter => &[Permission::Read, Permission::Comment],
        Role::Contributor => &[Permission::Read, Permission::Create],
        Role::Editor => &[
            Permission::Read,
            Permission::Comment,
            Permission::Create,
            Permission::Update,
        ],
        Role::Owner => &[
            Permission::Read,
            Permission::Comment,
            Permission::Create,
            Permission::Update,
            Permission::Share,
            Permission::Delete,
            Permission::Manage,
        ],
    ```
2026-06-18 02:47:06 +02:00
Dionisio Pozo b8a0018785 Merge pull request #473 from EdouardVanbelle/fix/nextcloud+webdav
fix(nextcloud+webdav) fix bugs found via end to end tests
2026-06-17 12:42:31 +02:00
Bradley Nelson b3e1e42e93 Clean up 2026-06-17 00:28:10 -06:00
Bradley Nelson 4427b1613b plugin logging 2026-06-16 23:00:23 -06:00
Bradley Nelson 803150635c add frontend 2026-06-16 21:26:36 -06:00
Bradley Nelson 87d68c5b6f init plugins 2026-06-16 17:57:57 -06:00
Edouard Vanbelle f9de7ac596 fix(webdav): native COPY honours destination filename (M8)
Threads `new_name: Option<&str>` through FileWritePort::copy_file and
FileManagementUseCase::copy_file_with_perms so a same-folder
COPY /a.txt → /b.txt picks up the destination name via a single
COALESCE($3::text, name) in the CTE. Without it the new row inherits
the source's filename and collides on the (folder_id, name, user_id)
unique index — the "Already Exists" 500 M8 was hitting.

handle_copy in the native WebDAV surface now passes
`(file.name != dest_name).then(|| dest_name.into())`, keeping the
"same name in a different folder" case at None so existing semantics
are preserved.
2026-06-17 01:49:13 +02:00
Claude f70dffeaf5 perf(dedup): backend-aware chunk read-ahead for CDC reassembly
read_blob_stream / read_blob_range_stream reassembled a CDC file by fetching
its chunks with `buffered(1)` — strictly sequential, so the next chunk's
backend fetch (a file `open` locally; a full request round-trip on S3/Azure)
only started after the current chunk was fully drained.

A benchmark of the exact pipeline (stream::iter(chunks).map(get).buffered(K)
.try_flatten()) showed a blind `buffered(4)` is the WRONG fix: on a local
disk it is neutral on a warm page cache and ~37% SLOWER cold, because
concurrent opens turn one sequential read into several competing random-I/O
streams over content-addressed (scattered) chunk files. The win is entirely
on remote backends, where per-chunk request latency dominates and overlapping
fetches hide it (≈ linear in K).

So the read-ahead depth is now a backend hint, not a constant:
- BlobStorageBackend::read_prefetch() default 1 (sequential; safe for local).
- S3 / Azure override to 8 (overlap GETs to hide TTFB).
- cached / encrypted / retry / migration delegate to the backend that serves
  the bytes.
- Both CDC read paths use `self.backend.read_prefetch().max(1)`.

Net: local backend unchanged (no regression); remote reassembly ~4-8x faster.
Ordered `buffered` (not buffer_unordered) keeps chunks in sequence.

Bench (per-chunk fetch-latency model): buffered(1)->(4)/(8) = x3.9 / x7.8
@1ms, x4.0 / x8.1 @5ms, x4.0 / x8.0 @20ms. Local warm: 230ms@1 vs 227ms@4
(noise); local cold: 425ms@1 vs 585ms@4 (why local stays at 1).

https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
2026-06-15 12:34:56 +00:00
Claude 5c09f916f7 Merge origin/main (Tantivy content search) into delta-sync branch
Both sides added a parameter to create_application_services and a
setup step before it: this branch's storage-usage/quota service (for
the instant-upload path) and main's Tantivy content index (for
SearchService). The resolution keeps both — the signature takes both
arguments and the build runs storage usage as step 3c and the content
index as 3d.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 18:32:27 +00:00
Claude 8dab135090 Add embedded Tantivy full-text content search
/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.

Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
  (storage.search_index_dirty) - every write surface (REST, WebDAV,
  NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
  text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
  N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
  batched single-writer Tantivy commits; queue rows are deleted only
  after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
  reseeds it from Postgres, which remains the single source of truth

SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.

The frontend renders the snippet under the file name in list view.

New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.

https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
2026-06-11 15:16:03 +00:00
Claude e3f04d58aa Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 13:06:33 +00:00
Claude 23de7e503b Cache Arc<TokenClaims> in JWT validation; bump Docker base images
JWT validation cache now stores Arc<TokenClaims> and validate_token
returns Arc<TokenClaims>. On a cache hit — the 99% path for every
authenticated request — the moka lookup was deep-cloning the whole
claims struct (5 Strings: sub, jti, username, email, role) on every
call. It is now a refcount bump. Read-only callers (admin middleware)
go through Deref and allocate nothing; the auth middleware clones only
the three fields it moves into CurrentUser (was 5 clones, now 3), and
the admin paths clone only role (was 5, now 1). A new test asserts the
hit path returns a pointer-equal Arc.

TokenServicePort::validate_token is the single trait method touched;
its only implementor is JwtTokenService and the only production callers
are the auth and admin middleware (the WOPI handler uses a separate
WopiTokenService).

Dockerfile: rust:1.94.1-alpine3.23 -> rust:1.96-alpine3.24 and
alpine:3.23.3 -> alpine:3.24.0 for the runtime stage.

https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
2026-06-11 10:56:33 +00:00
Claude 6a27c742d4 Fix CalDAV/CardDAV multiget loading entire collections into memory
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.

- Add find_events_by_ical_uids / get_contacts_by_uids through every
  layer (domain repo trait → PG repo → storage port/adapter → use case
  → handler) using one indexed `= ANY($2)` query, mirroring the
  existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
  uid_from_multiget_href helper (percent-decoding, case-insensitive
  extension strip, collection hrefs rejected). Exact UID matching also
  removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
  get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
  GET /api/address-books/{id}/contacts: optional ListQuery fields
  thread through ContactUseCase::list_contacts to a paginated repo
  query. Omitted params keep returning the full book (frontend and
  DAV listing/sync paths unchanged).

https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
2026-06-11 09:21:35 +00:00
Dionisio Pozo fe0053bc79 Optimize encryption, storage, and media streaming performance (#447)
- AES-256-GCM in-place decryption halves peak RAM in encrypted blob backend
- Offload crypto ≥64 KiB to spawn_blocking (unblocks async runtime)
- Fix off-by-one in encrypted range stream (end now exclusive)
- Collapse 3 DB round-trips for quota updates into 1 correlated UPDATE
- Set-based reconciliation sweep replaces per-user task spawning
- Eliminate entity re-read after file overwrite via RETURNING clause
- Lightbox streams video/photos inline instead of fetch→blob
2026-06-10 14:53:55 +02:00
Claude 26ff8f2ac9 Merge origin/main into claude/upbeat-wright-btruto
Resolves the four-file conflict with PR #444, which landed a parallel
implementation of the same batch-fsync design (unsynced chunk writes plus
one sync_blobs durability barrier). Both sides converged on identical
trait signatures; main's version is kept for all overlapping code since
it is already merged and carries unit tests:

- blob_storage_ports.rs: doc-comment-only conflict, main's docs kept
- dedup_service.rs: main's pipeline kept. The naive textual merge would
  have retained BOTH durability-barrier/INSERT blocks, double-counting
  ref_count for every new chunk; the duplicated block from this branch
  is dropped
- encrypted_blob_backend.rs, local_blob_backend.rs: main's
  implementations kept (fsync_paths_parallel + tests); this branch's
  equivalent helpers (fsync_dir, fsync_blob_file, SYNC_BLOBS_CONCURRENCY)
  are referenced nowhere else and are dropped

Everything unique to this branch (streaming PROPFIND, range requests,
batch ID resolution, CalDAV/CardDAV indexed lookups, transcode and
breadcrumb changes) merges cleanly and is unchanged.

https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
2026-06-10 11:51:54 +00:00
Claude 908b8f4d4b perf(upload): one durability barrier per file instead of two fsyncs per chunk
The CDC chunk path paid `sync_all` + parent-dir fsync + one ref-count
upsert round-trip PER ~256 KB chunk: a new 1 GB file ≈ 8 000 fsyncs +
4 000 sequential INSERTs on the upload critical path (seconds on SSD,
tens of seconds on HDD).

- New `put_blob_from_bytes_unsynced` + `sync_blobs` on
  BlobStorageBackend. Defaults delegate to the durable variants, so
  remote backends (S3/Azure — durable on PUT) are untouched.
  LocalBlobBackend writes chunks without fsync and `sync_blobs` then
  fsyncs all files concurrently (kernel coalesces the writeback) plus
  one fsync per DISTINCT shard directory — at most 256 — instead of one
  per chunk. EncryptedBlobBackend delegates both so encryption over the
  local backend keeps the optimization.
- store_chunks: uploads use the unsynced variant; one `sync_blobs`
  barrier runs before anything references the chunks, and the per-chunk
  ref-count upserts collapse into a single
  `INSERT ... SELECT unnest(...) ON CONFLICT` statement.

Durability contract is unchanged: every chunk is on stable storage
before the manifest row that references it commits. A crash mid-upload
leaves chunk files without DB rows — the same orphan class the
per-chunk scheme already produced, just a wider window.

https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
2026-06-10 09:52:47 +00:00
Claude 7687766bf7 perf(dav-collections): indexed UID lookups for single-object operations
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.

Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.

https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
2026-06-10 09:52:28 +00:00
Edouard Vanbelle 41aad26702 feat(upload): cover chunk upload + add support of different digest hash
Prefer stream storage rather using buffered (in memory)

  note: on many unix like tmpfs are in-memory, sungle PUT are sized limited

  Storage map (NC stands for Nextcloud gateway)

  ┌───────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────┐
  │                   Streaming surface                   │                            Destination                             │                Configurable via                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST chunked PUT /api/uploads/{id} chunk              │ {storage_path}/.uploads/{upload_id}/chunk_{NNNNNN}                 │ OXICLOUD_STORAGE_PATH (the .uploads subdir is   │
  │                                                       │                                                                    │ hard-wired)                                     │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST chunked assemble (during /complete)              │ {storage_path}/.uploads/{upload_id}/assembled                      │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC chunked PUT /dav/uploads/.../{chunk}               │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}  │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC chunked assemble (during MOVE)                     │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/.assembled    │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC single-file PUT /dav/files/.../{path} (via         │ OXICLOUD_UPLOAD_TMPDIR if set, else OS default temp (/tmp on       │ OXICLOUD_UPLOAD_TMPDIR                          │
  │ spool_body_to_temp)                                   │ Linux)                                                             │                                                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST WebDAV PUT /webdav/{path} (via                   │ same as above                                                      │ OXICLOUD_UPLOAD_TMPDIR                          │
  │ spool_body_to_temp)                                   │                                                                    │                                                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST multipart upload /api/files/upload               │ {storage_path}/.dedup_temp/upload-{uuid}                           │ OXICLOUD_STORAGE_PATH (hard-wired subdir)       │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ WOPI PutFile                                          │ OS default temp via NamedTempFile::new() (no override)             │ (none — bug worth tracking)                     │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ Final blob storage (after fsync + rename)             │ {storage_path}/.blobs/{ab}/{abc…}.blob                             │ OXICLOUD_STORAGE_PATH                           │
  └───────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┘

  one caveat: a malicious user can create many chunked upload and saturate local storage
2026-06-09 09:52:09 +02:00
Edouard Vanbelle 8cc21f17c5 feat(notify): add notif to internal users when granted
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
    - add resentd method in share menu item (work for both internal and external users)
    - user can disable email notification via his properties
    - add env variable from admin to disable notifications
2026-06-05 11:25:06 +02:00
Edouard Vanbelle 7db27af7a6 feat(user.prefered_locale): save user's locale + invited have same locale as inviters
- OIDC JIT define the locale only at user creation, user can so change his preference later
    - invited users will inherit inviter's locale
    - email will use prefered_locale
    - login to a new browser will use prefered_locale
2026-06-03 14:37:22 +02:00
Edouard Vanbelle 803849fe2e feat(user): map family/given name to OIDC & cardav 2026-06-03 00:32:00 +02:00
Edouard Vanbelle 458232354b feat(audit): always emit audit log on resource/call not granted / rejected 2026-06-03 00:31:59 +02:00
Edouard Vanbelle d03b9474c6 feat(smtp): add a precious SMTP test for admin only 2026-06-03 00:31:59 +02:00
Edouard Vanbelle 2011d19e71 feat(smtp): add SMTP support to reach MTA 2026-06-03 00:31:59 +02:00
Edouard Vanbelle 5fab0532dc feat(user): add given_name/family_name auth.users
reflect OIDC
  schema migration, User entity additions, and three defense-in-depth gaps closed, with all 280 unit tests and 13 Hurl files green
  infrastructure (lettre + EmailSender port).

  - Migration migrations/20260612000003_users_username_email_login.sql — adds nullable given_name/family_name columns to auth.users.
  - User entity (src/domain/entities/user.rs) — has_login_credential() placeholder-check encapsulation, set_username revalidating setter, given/family-name fields + getters/setters, validate_username widened 32→254 and now accepts email shape.
  from_data_full extended with two new params; all 7 callsites in user_pg_repository.rs updated.
  - Schema-side legacy guards (src/application/services/auth_application_service.rs) — bumped the duplicated 32-char check in setup_create_admin and admin_create_user to 254 to match.
  - Gap #1 (subject_group_service.rs) — add_member now rejects external candidates with an audit-logged AccessDenied. Service gained an Arc<UserPgRepository> field, wired through DI. New integration test test_external_user_cannot_be_added_as_member.
  - Gap #2 (user_repository.rs + auth_ports.rs + user_pg_repository.rs) — list_users/search_users gained an include_external: bool param defaulting effectively to false everywhere internal-user-facing. auth_application_service exposes a new
  list_users_including_external for the admin surface.
  - Gap #3 (pg_acl_engine.rs) — expand_user now SELECTs is_external and skips INTERNAL_GROUP_ID for externals; defaults to is_external=true on missing user to fail closed.
2026-06-03 00:31:59 +02:00
Edouard Vanbelle 6a6f070106 feat(userLifecycle): plug actions to on_user_logout and on_user_deleted
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
  - SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
  - DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
  - Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
2026-06-01 22:51:57 +02:00
Edouard Vanbelle bb6429a620 refactor(userLifecycle): add user lifecycle, more clarety + better integration for the future 2026-06-01 22:51:53 +02:00
Edouard Vanbelle 63722c868e feat(myshares): new version of myshares sections, supporting grants Users + Tokens 2026-05-29 13:12:59 +02:00
Edouard Vanbelle 8800353900 refactor(share,grants): migrate expiration from legacy share into grants, simplify legacy share, normalize shareModal for better UX 2026-05-28 20:00:06 +02:00
Edouard Vanbelle 73e78904e7 feat(recent): add cursor + groupby support 2026-05-28 12:12:22 +02:00
Edouard Vanbelle 607ae5e6df feat(resources): add cursor, group by on /api/favorites/resources 2026-05-28 12:12:22 +02:00
Edouard Vanbelle 5790a1459f feat(folders): add curser and the normalized way to get foler's item list. add reverse order 2026-05-28 12:12:22 +02:00
Edouard Vanbelle 5afb30ebfd feat(swimlane): add swimlane engine with first version on SharedWithMe section
added group by:

        - None (= ordered by folders/file name)
        - Type (Folder first, then Image, Vidao, Audio, Document, etc...)
        - Owner
        - Size (With logarithmic groups))
        - Shared date (with groups: today, last 7 days, last 30 days, then year)
2026-05-28 00:15:05 +02:00
Edouard Vanbelle b0c5e7827e feat(user-avatar): users can now edit there image (image is taken from OIDC picture) 2026-05-27 11:29:33 +02:00