Commit Graph

511 Commits

Author SHA1 Message Date
Edouard Vanbelle 3f70c7e20a refactor(server): remove sort by size in pg_acl_engine.rs (dead code) 2026-05-29 13:14:16 +02:00
Edouard Vanbelle 63722c868e feat(myshares): new version of myshares sections, supporting grants Users + Tokens 2026-05-29 13:12:59 +02:00
Edouard Vanbelle 8800353900 refactor(share,grants): migrate expiration from legacy share into grants, simplify legacy share, normalize shareModal for better UX 2026-05-28 20:00:06 +02:00
Edouard Vanbelle 73e78904e7 feat(recent): add cursor + groupby support 2026-05-28 12:12:22 +02:00
Edouard Vanbelle 607ae5e6df feat(resources): add cursor, group by on /api/favorites/resources 2026-05-28 12:12:22 +02:00
Edouard Vanbelle 5790a1459f feat(folders): add curser and the normalized way to get foler's item list. add reverse order 2026-05-28 12:12:22 +02:00
Edouard Vanbelle 5afb30ebfd feat(swimlane): add swimlane engine with first version on SharedWithMe section
added group by:

        - None (= ordered by folders/file name)
        - Type (Folder first, then Image, Vidao, Audio, Document, etc...)
        - Owner
        - Size (With logarithmic groups))
        - Shared date (with groups: today, last 7 days, last 30 days, then year)
2026-05-28 00:15:05 +02:00
Edouard Vanbelle b0c5e7827e feat(user-avatar): users can now edit there image (image is taken from OIDC picture) 2026-05-27 11:29:33 +02:00
Edouard Vanbelle 8365608bd5 feat(list view): show the owner of the File or Folder in list view 2026-05-25 22:48:59 +02:00
Edouard Vanbelle 093c1ad3a5 feat(grants): add /api/grants/incoming/resources with a cursor for pagination 2026-05-25 22:47:01 +02:00
Edouard Vanbelle 73f0b0fa47 refactor(lifecycle hooks): simplify integration of new services
* make more coherent lifecycles
  * remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
  * reduce risk of orphean objects
  * ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
  * more details on docs/architecture/file-and-blob-lifecycle.md :

```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {

    fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
    fn on_file_updated(file_id, blob_hash, content_type);
    fn on_file_copied(file_id, blob_hash, content_type, source_id)
    fn on_file_deleted(file_id);
}

// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {

    fn on_blob_created(blob_hash, content_type);
    fn on_blob_deleted(blob_hash);
}
```
2026-05-22 13:40:58 +02:00
Edouard Vanbelle 191f725199 feat(ui/trash): show original file path in tooltip on mouse over + display thumbnails
* fix: permission also check elements trashed elements
 * tested manually
 * all automated tests ok
2026-05-22 02:26:11 +02:00
Edouard Vanbelle 76a85949e7 fix(trash)+refactor(file life cycle)
* fix issue with the empty trash (wasn't calling thumbnail clean up)
 * refactor file service life cycle (TrashService don't call directly ThumbnailService, but call the on_file_deleted() hook
 * remove unused mehod: _validate_user_ownership()
2026-05-22 00:28:39 +02:00
Edouard Vanbelle cb35775f77 fix(dedub): correct ref count on hashes, many thanks to you api tests... 2026-05-21 21:12:38 +02:00
Edouard Vanbelle a53c09f361 feat(authz): covert and test chunked upload with permissions 2026-05-21 20:30:58 +02:00
Edouard Vanbelle cba9be8c21 feat(rebac): first pass 2026-05-20 22:56:00 +02:00
Edouard Vanbelle dfb082fdf4 refactor(server): file_management_service: move all method without owner check into private, add folder_ports 2026-05-20 15:39:53 +02:00
Edouard Vanbelle f8b30e78a6 refactor(create_folder): add an ownership check while creating a folder + refactor code 2026-05-20 12:59:04 +02:00
Dionisio Pozo 1e34c5f3f8 Merge pull request #330 from EdouardVanbelle/fix/thumbnail-on-update
bugfix/thumbnails on update
2026-05-14 00:38:03 +02:00
Edouard Vanbelle d85b8055b8 refactor(services): add file_lifecycle and blob_lifecycle
- complete src/application/ports/blob_lifecycle.rs with traits:
    * BlobCreationHook
    * BlobDeletionHook

 - add src/application/ports/file_lifecycle.rs with traits:
    * FileCreatedHook
    * FileDeletedHook
    * FileUpdatedHook
2026-05-14 00:18:12 +02:00
Dionisio Pozo 412e31388d Merge pull request #347 from abnvle/feat/share-folder-browsing 2026-05-14 00:07:45 +02:00
Edouard Vanbelle 28e25f9d16 fix+test: check hash ref count on copy-on-write (a duplicated beeing updated) 2026-05-13 15:55:41 +02:00
Edouard Vanbelle 78cb37b311 feat: check thumbnail cleanup on files deletion + correct ref counter 2026-05-13 15:55:38 +02:00
Edouard Vanbelle 9f57776ec9 bugfix/thumbnails on update
Bug 1 & 2 (webdav_handler.rs handle_put() update branch):
  - After a successful file update via WebDAV PUT, if the content type is a supported image:
    a. delete_thumbnails(file_id) — evicts the stale moka cache entry
    b. Spawns a background task to read the new blob bytes and call generate_all_sizes_background_from_bytes

  Bug 3 & 4 (dedup_service.rs):
  - Added thumbnail_service: Option<Arc<ThumbnailService>> field with a with_thumbnail_service() builder
  - In remove_legacy_reference(): calls delete_blob_thumbnails(hash) when ref_count hits 0
  - In remove_manifest_reference(): calls delete_blob_thumbnails(file_hash) when manifest's last ref is dropped
  - Wired in di.rs — the thumbnail service is created before dedup service so the ordering works cleanly
2026-05-12 09:00:26 +02:00
abnvle e8399443e9 Merge remote-tracking branch 'origin/main' into feat/share-folder-browsing
# Conflicts:
#	static/js/views/public/publicShare.js
2026-05-10 20:46:29 +02:00
abnvle ec55e7ec55 fix(share): tighten public browsing (UUID guard, cache headers, abort race) 2026-05-10 20:43:59 +02:00
Edouard Vanbelle 89ec0d2c24 feat(contacts): add API to address-books, contacts, groups
* full gateway to /carddav
 * add system readonly address-book issued from OxiCloud users
2026-05-10 00:39:25 +02:00
Dionisio Pozo 52f4a865a9 Merge pull request #349 from EdouardVanbelle/feat/item-path-tool-tip+go-to-parent
show item path on Recent + Favorite, add go to parent folder, fix Folder browsing in Favorites
2026-05-08 01:22:08 +02:00
Edouard Vanbelle b90fa6f619 security: prevent re-use of refresh token (reduce surface for any stolen token)
Security: session hardening

  Refresh token rotation with theft detection (family_id)
  - Added family_id column to auth.sessions (migration 20260507000000_session_family.sql) grouping all tokens issued from the same login into a family
  - On refresh, the new session inherits the parent's family_id
  - If a revoked token is replayed (indicates the token was stolen after rotation), the entire family is immediately invalidated and a warning is logged — forcing re-authentication on all devices

  SameSite=Strict on refresh cookie
  - Access cookie stays SameSite=Lax (needed for top-level navigation)
  - Refresh cookie upgraded to SameSite=Strict — it is only ever used for explicit POST to /api/auth/refresh, never via cross-site navigation

  Refresh token TTL: 30 days → 7 days
  - With rotation, active sessions auto-renew and effectively never expire
  - Inactive sessions expire after 7 days instead of 30, reducing the theft window
2026-05-07 13:57:10 +02:00
Edouard Vanbelle 8e1e9fe201 show item path on Recent + Favorites, add go to parent folder, fix Folder browsing in Favorites 2026-05-06 23:36:05 +02:00
abnvle d15d7b8f8e feat(share): public folder browsing API + range support + zip
Five new public endpoints under /api/s/{token}/...:
  GET /contents
  GET /contents/{folder_id}
  GET /file/{file_id}
  GET /zip
  GET /zip/{folder_id}

All honour the unlock cookie from /verify, so password-protected
folder shares work end-to-end.

Folder/file IDs are validated against the share subtree via a single
ltree containment query (O(log N) on the existing GiST index).
Out-of-scope IDs return 404.

download_shared_file refactored to a Range/304/206/416-aware
serve_share_file helper, shared with the new /file/{file_id}
endpoint. content_disposition extracted from FileHandler so RFC 5987
formatting is identical across auth and share download paths.
2026-05-05 22:41:55 +02:00
abnvle 3bc5c0873b fix(share): password-protected downloads via signed unlock cookie
After successful POST /api/s/{token}/verify, the server issues a
short-lived signed JWT cookie (oxi_share_unlock_<token>; HttpOnly;
SameSite=Lax; 1h TTL). Subsequent /api/s/{token} and
/api/s/{token}/download requests honour the cookie to bypass the
password gate, so password-protected file downloads work end-to-end.

- New module src/infrastructure/services/share_unlock_cookie.rs:
  issue_jwt, verify_jwt, extract_from_cookie_header, build_set_cookie
  + 10 unit tests.
- New ShareService methods issue_unlock_jwt and
  get_shared_link_with_unlock; trait method get_shared_link_by_token
  delegates to a private fetch_share_resolved with an
  allow_password_protected flag.
- access_shared_item, verify_shared_item_password, and
  download_shared_file honour the unlock cookie.

Reuses OXICLOUD_JWT_SECRET, no new env var. Auth-token JWTs and
unlock-cookie JWTs cannot be confused: auth requires
username/email/role/jti claims, unlock has only sub/exp/iat.
Cross-share replay rejected via claims.sub == requested_token check.
2026-05-05 21:56:49 +02:00
DeepRot c041dfece4 fix(carddav): resolve PROPFIND 404 and MKCOL 500 bugs 2026-05-01 09:10:23 +02:00
Edouard Vanbelle 3a315ae75d chore(rust): use same params for clippy as the CI + correct clippy warnings 2026-04-27 20:25:33 +02:00
DioCrafts 30fd3cc488 Fix thumbnail generation for CDC blob storage 2026-04-26 11:55:05 +02:00
onthebed fa5efe0fd0 fix(wopi): decode escaped discovery action URLs
Decode XML-escaped WOPI discovery action attributes before storing them
so generated editor URLs use real query separators instead of literal
&amp; entities. Add regression coverage for parsed actions and generated
edit URLs.
2026-04-25 03:43:30 +08:00
Diocrafts 761d159a92 feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking
  (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing
- Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[])
- Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends)
- 3-phase store_chunks pipeline:
  Phase 0: batch-check existing chunks (single PG query)
  Phase 1: selective disk read (skip existing chunks entirely)
  Phase 2: parallel upload with buffer_unordered(8)
- CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback
- Transactional manifest + chunk ref-count cascade on remove_reference
- 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.)
- Update deduplication.md to reflect new architecture
2026-04-14 23:17:39 +02:00
Diocrafts cd3733b459 feat: pluggable storage backends (S3, Azure, local) with admin UI
Implement 4-phase external storage backends architecture:

Phase 1 - Foundation:
- BlobStorageBackend trait (application/ports/blob_storage_ports.rs)
- LocalBlobBackend: extracted all tokio::fs ops from DedupService
- S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2)
- DedupService refactored to use Arc<dyn BlobStorageBackend>

Phase 2 - Admin Panel:
- StorageSettingsService with DB persistence + env override
- Storage tab in admin panel (backend selector, S3 form, provider presets)
- GET/PUT/POST endpoints for storage settings + connection test
- i18n keys (en/es) and BEM CSS

Phase 3 - Migration:
- MigrationBlobBackend decorator (dual-read: target-first + source fallback)
- Background migration job with parallel transfers + progress tracking
- Migration UI (progress bar, ETA, pause/resume/verify/complete)
- 6 admin API endpoints for migration lifecycle

Phase 4 - Enterprise Extras:
- CachedBlobBackend: LRU disk cache for remote backends
- EncryptedBlobBackend: AES-256-GCM at-rest encryption
- AzureBlobBackend: Azure Blob Storage support
- RetryBlobBackend: exponential backoff for transient errors
- Decorator composition in DI: retry → encryption → cache

All 223 tests passing, clippy clean, fmt verified.
2026-04-14 21:33:38 +02:00
Edouard Vanbelle 62a7713af5 refactor(server): apply fmt + lint recos 2026-04-14 19:05:01 +02:00
Diocrafts 5be035a172 fix: restore ext-thumbnail disk lookup in get_cached_thumbnail
After the blob_hash dedup change, get_cached_thumbnail with
blob_hash=None (cache-first fast path) only checked moka. External
video thumbnails stored as ext-{file_id}.jpg on disk were not found
after a server restart when moka is empty.

Now checks ext-{file_id}.jpg on disk before falling through to the
blob-hash disk lookup, so video thumbnails survive restarts.
2026-04-12 00:56:58 +02:00
Diocrafts 2dde4da5cf feat: thumbnail dedup — store thumbnails by blob_hash instead of file_id (#233)
Thumbnails are now keyed by blob_hash on disk so identical files share
a single set of thumbnails (icon/preview/large). For 4000 duplicate
files with the same content, this reduces thumbnail storage from 12,000
files to just 3.

Changes:
- get_thumbnail_path() keys by blob_hash instead of file_id
- get_thumbnail(), get_cached_thumbnail(), generate_all_sizes_background()
  accept blob_hash parameter for disk dedup
- generate_all_sizes_background() fast path: if blob-hash thumbnails
  already exist on disk, skip image processing entirely and just
  populate moka cache for the new file_id
- delete_thumbnails() only invalidates moka cache (shared disk
  thumbnails must not be deleted when one file is removed)
- delete_blob_thumbnails() added for GC; garbage_collect() now cleans
  up orphaned thumbnail files alongside blob files
- External thumbnails (video frames) stored as ext-{file_id}.jpg
  since they are client-generated and not dedup-able
- ThumbnailPort trait updated with blob_hash parameters
- All handler call sites updated (file_handler, preview_handler)
- Tests updated for new signatures
2026-04-12 00:50:10 +02:00
Diocrafts 78fcf5f08f perf: move audio metadata I/O to spawn_blocking + stream reextract_all
- Move all sync I/O (id3::Tag, mp3_duration) into spawn_blocking via
  extract_metadata_blocking() to avoid stalling Tokio worker threads
- Replace fetch_all with streaming .fetch() in reextract_all_audio_metadata
  for O(1) memory usage regardless of audio file count
- Consolidate get_duration_secs into extract_metadata_blocking, eliminating
  redundant file open (tag was read twice before)
2026-04-11 19:36:34 +02:00
Diocrafts 63bcd0ffe7 perf: non-blocking hash + parallel thumbnail generation
- Replace blocking blake3::hash() with DedupService::hash_file() in
  file_upload_service (create_file, update_file). Uses spawn_blocking +
  mmap_rayon instead of blocking the Tokio async worker.
- Parallelize thumbnail resize+encode with rayon par_iter() for the 3
  sizes (Icon, Preview, Large) inside spawn_blocking.
2026-04-11 19:13:48 +02:00
Diocrafts a2aece0752 perf: scale thumbnail decode semaphore with available CPUs
Replace hardcoded DEFAULT_MAX_CONCURRENT_DECODES=4 with adaptive
max_concurrent_decodes() that uses available_parallelism()/2 (min 2).
Matches the pattern already used in image_transcode_service.

Improves thumbnail throughput on 16+ core servers by 2-4x.
2026-04-11 18:14:33 +02:00
Diocrafts d32a32c30a perf: add TTL/TTI to content cache and invalidate on file mutations 2026-04-11 17:39:17 +02:00
Diocrafts 81f33458e0 perf: streaming hash-on-write dedup upload, remove dead code 2026-04-11 16:17:47 +02:00
Andrey Tkachenko da066f47fa Music Player & Playlist Manager 2026-04-08 15:14:03 +03:00
Edouard Vanbelle badf35f08f chore: remove all executable attributes on non bash files 2026-04-01 23:14:42 +02:00
Claude dacb8f388e fix(thumbnail): add timeout protection for large image processing
Resolves conflict between main and PR #229 by applying timeout support
on top of main's drop(data) memory optimization. Changes:
- Add thumbnail_generation_ms to TimeoutConfig (default 30s)
- Add generation_timeout field to ThumbnailService
- Wrap spawn_blocking in tokio::time::timeout to prevent hanging
- Update DI to pass timeout from config
- Update tests to provide timeout parameter

https://claude.ai/code/session_015FD62aAoMYx1XBZbzPvUu8
2026-03-28 18:40:34 +00:00
BillionClaw 9794bba975 fix(thumbnails): drop encoded image data after decoding to reduce memory usage
Explicitly drop the encoded image buffer after decoding and extracting
EXIF orientation data. This reduces peak memory consumption during
thumbnail generation by the size of the original file.

The encoded data is no longer needed once the image is decoded into
a DynamicImage, but it was being held in memory until the end of the
spawn_blocking scope.

Fixes excessive memory consumption in thumbnail generation.
2026-03-18 11:51:39 +08:00