explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
to help administratorrs to defined correctly their storage architecture
purpose: avoid current scheme:
1. write to disk
2. reopen file to read data and digest it
now: digest is done while writing data to disk
all other implementation than Nextcloud are corrrect
add OXICLOUD_CHUNK_MAX_BYTES which correspond to the max upload chunk allowed
(differs from OXICLOUD_MAX_UPLOAD_SIZE which is the max total size of a file)
hurl test validate the change
Streams the request body straight to the chunk file with peak heap of
~one HTTP frame, regardless of chunk size or the configured cap. The
`storage.chunk_max_bytes` config (env `OXICLOUD_CHUNK_MAX_BYTES`,
default 100 MB) bounds a single PUT — separate from `max_upload_size`
which governs whole-file uploads. Without this separation, a client
could submit a chunk up to the whole-file cap (10 GB default) and
monopolise server memory.
PROPFIND on upload chunk is necessary for application to resume the upload
it is used at least by Nextcloud Android app
issue has been raised via #415
unable to test, so I am using a playbook in test/webdav to simulate a
PROPIND on remote.php/dav/uploads/{user}/{session}
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:
- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
Now streams the body to a temp file with incremental BLAKE3 and goes
through update_file_streaming (shared spool helper with the native
WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.
- DedupService::store_chunks materialized every new chunk's data in a Vec
before uploading. Now reads each new chunk by positioned I/O
(read_exact_at, off the runtime via spawn_blocking) just before its
upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.
- The upload spool used the OS temp dir, often tmpfs/RAM in containers
where its page-cache counts against the cgroup memory limit. Add
OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.
Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.
Refs #404
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
- add resentd method in share menu item (work for both internal and external users)
- user can disable email notification via his properties
- add env variable from admin to disable notifications
- OIDC JIT define the locale only at user creation, user can so change his preference later
- invited users will inherit inviter's locale
- email will use prefered_locale
- login to a new browser will use prefered_locale
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
username: now optional, if defined 2..64 chars
password: now optional (no mode __NO_PASSWORD...__)
oidc: now optional
important: if need Nextcloud, username must be defined
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
reflect OIDC
schema migration, User entity additions, and three defense-in-depth gaps closed, with all 280 unit tests and 13 Hurl files green
infrastructure (lettre + EmailSender port).
- Migration migrations/20260612000003_users_username_email_login.sql — adds nullable given_name/family_name columns to auth.users.
- User entity (src/domain/entities/user.rs) — has_login_credential() placeholder-check encapsulation, set_username revalidating setter, given/family-name fields + getters/setters, validate_username widened 32→254 and now accepts email shape.
from_data_full extended with two new params; all 7 callsites in user_pg_repository.rs updated.
- Schema-side legacy guards (src/application/services/auth_application_service.rs) — bumped the duplicated 32-char check in setup_create_admin and admin_create_user to 254 to match.
- Gap #1 (subject_group_service.rs) — add_member now rejects external candidates with an audit-logged AccessDenied. Service gained an Arc<UserPgRepository> field, wired through DI. New integration test test_external_user_cannot_be_added_as_member.
- Gap #2 (user_repository.rs + auth_ports.rs + user_pg_repository.rs) — list_users/search_users gained an include_external: bool param defaulting effectively to false everywhere internal-user-facing. auth_application_service exposes a new
list_users_including_external for the admin surface.
- Gap #3 (pg_acl_engine.rs) — expand_user now SELECTs is_external and skips INTERNAL_GROUP_ID for externals; defaults to is_external=true on missing user to fail closed.
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
- SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
- DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
- Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
this implements first version (manageable only by admin right now)
routes:
GET /api/groups
List subject groups (paginated). Admin-only.
POST /api/groups
Create a new ReBAC subject group. Admin-only. The name must match the RFC 5321 local-part shape and be globally unique (case-insensitive).
GET /api/groups/search
Search non-virtual groups by name substring. Authenticated only (no admin role required) — backs the share-dialog recipient autocomplete.
GET /api/groups/{id}
Fetch a single group's details. Admin-only.
DELETE /api/groups/{id}
Delete a group. Cascades to `subject_group_members` (FK) and to `access_grants` rows referencing this group as a subject. Admin-only.
PATCH /api/groups/{id}
Update a group's metadata. Admin-only. v1 only persists name renames.
GET /api/groups/{id}/effective-members
List every user transitively reached through this group (members of members of members, etc.). Used by admin / audit tooling. Admin-only.
GET /api/groups/{id}/members
List the *direct* members of a group (one level only). Admin-only.
POST /api/groups/{id}/members
Add a member to a group. Exactly one of `user_id` / `group_id` must be provided. Adding a group-member runs a write-time cycle check and a nesting-depth check (max 8). Admin-only.
DELETE /api/groups/{id}/members/group/{gid}
Remove a nested group-member from a group. Admin-only.
DELETE /api/groups/{id}/members/user/{uid}
Remove a user-member from a group. Admin-only.
fix hurl
groups
round
groups
* make more coherent lifecycles
* remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
* reduce risk of orphean objects
* ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
* more details on docs/architecture/file-and-blob-lifecycle.md :
```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {
fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
fn on_file_updated(file_id, blob_hash, content_type);
fn on_file_copied(file_id, blob_hash, content_type, source_id)
fn on_file_deleted(file_id);
}
// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {
fn on_blob_created(blob_hash, content_type);
fn on_blob_deleted(blob_hash);
}
```
Bug 1 & 2 (webdav_handler.rs handle_put() update branch):
- After a successful file update via WebDAV PUT, if the content type is a supported image:
a. delete_thumbnails(file_id) — evicts the stale moka cache entry
b. Spawns a background task to read the new blob bytes and call generate_all_sizes_background_from_bytes
Bug 3 & 4 (dedup_service.rs):
- Added thumbnail_service: Option<Arc<ThumbnailService>> field with a with_thumbnail_service() builder
- In remove_legacy_reference(): calls delete_blob_thumbnails(hash) when ref_count hits 0
- In remove_manifest_reference(): calls delete_blob_thumbnails(file_hash) when manifest's last ref is dropped
- Wired in di.rs — the thumbnail service is created before dedup service so the ordering works cleanly
After successful POST /api/s/{token}/verify, the server issues a
short-lived signed JWT cookie (oxi_share_unlock_<token>; HttpOnly;
SameSite=Lax; 1h TTL). Subsequent /api/s/{token} and
/api/s/{token}/download requests honour the cookie to bypass the
password gate, so password-protected file downloads work end-to-end.
- New module src/infrastructure/services/share_unlock_cookie.rs:
issue_jwt, verify_jwt, extract_from_cookie_header, build_set_cookie
+ 10 unit tests.
- New ShareService methods issue_unlock_jwt and
get_shared_link_with_unlock; trait method get_shared_link_by_token
delegates to a private fetch_share_resolved with an
allow_password_protected flag.
- access_shared_item, verify_shared_item_password, and
download_shared_file honour the unlock cookie.
Reuses OXICLOUD_JWT_SECRET, no new env var. Auth-token JWTs and
unlock-cookie JWTs cannot be confused: auth requires
username/email/role/jti claims, unlock has only sub/exp/iat.
Cross-share replay rejected via claims.sub == requested_token check.
Decode XML-escaped WOPI discovery action attributes before storing them
so generated editor URLs use real query separators instead of literal
& entities. Add regression coverage for parsed actions and generated
edit URLs.
After the blob_hash dedup change, get_cached_thumbnail with
blob_hash=None (cache-first fast path) only checked moka. External
video thumbnails stored as ext-{file_id}.jpg on disk were not found
after a server restart when moka is empty.
Now checks ext-{file_id}.jpg on disk before falling through to the
blob-hash disk lookup, so video thumbnails survive restarts.
Thumbnails are now keyed by blob_hash on disk so identical files share
a single set of thumbnails (icon/preview/large). For 4000 duplicate
files with the same content, this reduces thumbnail storage from 12,000
files to just 3.
Changes:
- get_thumbnail_path() keys by blob_hash instead of file_id
- get_thumbnail(), get_cached_thumbnail(), generate_all_sizes_background()
accept blob_hash parameter for disk dedup
- generate_all_sizes_background() fast path: if blob-hash thumbnails
already exist on disk, skip image processing entirely and just
populate moka cache for the new file_id
- delete_thumbnails() only invalidates moka cache (shared disk
thumbnails must not be deleted when one file is removed)
- delete_blob_thumbnails() added for GC; garbage_collect() now cleans
up orphaned thumbnail files alongside blob files
- External thumbnails (video frames) stored as ext-{file_id}.jpg
since they are client-generated and not dedup-able
- ThumbnailPort trait updated with blob_hash parameters
- All handler call sites updated (file_handler, preview_handler)
- Tests updated for new signatures
- Move all sync I/O (id3::Tag, mp3_duration) into spawn_blocking via
extract_metadata_blocking() to avoid stalling Tokio worker threads
- Replace fetch_all with streaming .fetch() in reextract_all_audio_metadata
for O(1) memory usage regardless of audio file count
- Consolidate get_duration_secs into extract_metadata_blocking, eliminating
redundant file open (tag was read twice before)
- Replace blocking blake3::hash() with DedupService::hash_file() in
file_upload_service (create_file, update_file). Uses spawn_blocking +
mmap_rayon instead of blocking the Tokio async worker.
- Parallelize thumbnail resize+encode with rayon par_iter() for the 3
sizes (Icon, Preview, Large) inside spawn_blocking.
Replace hardcoded DEFAULT_MAX_CONCURRENT_DECODES=4 with adaptive
max_concurrent_decodes() that uses available_parallelism()/2 (min 2).
Matches the pattern already used in image_transcode_service.
Improves thumbnail throughput on 16+ core servers by 2-4x.
Edouard Vanbelle