This solve issue with 2 migrations made the same day, due to merge on pull request, DB migration is blocking
the same version prefix:
- 20260625000000_files_user_size_index.sql (Dio)
- 20260625000000_folder_tree_modified_at.sql (Ed)
They were renamed to ...0001 and ...0002 (disjoint versions) + protection like "IF NOT EXISTS"
I have opt for an automated clean up of old entry:
`DELETE FROM _sqlx_migrations WHERE version = 20260625000000;`
runned on startup
affected users: Dio, myself and any dev that wanted to work on this project since eb0ba58158
GET /api/auth/me ran a synchronous O(N) SUM(size) over all the user's
files plus an unconditional UPDATE of auth.users on every call — one of
the most frequently hit endpoints — adding per-request latency, DB write
load, dead tuples and WAL even when nothing changed.
- /api/auth/me now serves the cached storage_used_bytes column instead of
recomputing it inline.
- New StorageUsageService::start_reconciliation_job runs a periodic sweep
on the maintenance pool that keeps the cached value current for every
mutation (uploads, deletes, trash), so freshness no longer depends on
hitting /me. Interval via OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default
600s, floored at 30s; first sweep deferred one interval to avoid boot load).
- update_storage_usage only writes when the value actually changes
(IS DISTINCT FROM), so the sweep produces no dead tuple / WAL on no-ops.
- New covering partial index idx_files_user_size_active makes the usage
SUM an index-only scan instead of a heap scan over all the user's files.
Also collapse the same pre-existing clippy collapsible_else_if in
carddav_handler that blocks the -D warnings gate on this base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:
- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
Now streams the body to a temp file with incremental BLAKE3 and goes
through update_file_streaming (shared spool helper with the native
WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.
- DedupService::store_chunks materialized every new chunk's data in a Vec
before uploading. Now reads each new chunk by positioned I/O
(read_exact_at, off the runtime via spawn_blocking) just before its
upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.
- The upload spool used the OS temp dir, often tmpfs/RAM in containers
where its page-cache counts against the cgroup memory limit. Add
OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.
Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.
Refs #404
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TL;DR:
fix duplicate filename via:
```
docker exec <container> migrate-nfc-filenames --dry-run # preview
docker exec <container> migrate-nfc-filenames # execute
```
== issue ==
Last week I uploaded Capture d'écran 2026-06-03 à 20.04.24.png from the web. It synced down to Nextcloud on my Mac. Two minutes later, the Web UI was showing the file twice.
Both rows had:
- the same name
- the same size
- the same content hash
So why two rows? Because to PostgreSQL, the names weren't the same.
Web upload (browser → Postgres):
"é" stored as 1 codepoint (U+00E9) bytes: c3 a9 ← NFC
NiextCloud client (macOS → Postgres):
"é" stored as 2 codepoints (e + U+0301) bytes: 65 cc 81 ← NFD
macOS's APFS keeps filenames in NFD (decomposed); browsers send NFC (composed). Visually é and é are identical. To WHERE name = $1 they're two different keys. Our UNIQUE index on (folder_id, name, user_id) never fired — and the row count quietly drifted every time a Mac user touched an accented
filename.
== The fix is two halves ==
1. No new duplicates — every name-receiving boundary (file upload, NC PUT, rename, MOVE, path lookup) now NFC-normalizes before touching the database. The storage invariant becomes "every stored name is NFC".
2. Clean up existing data — one-shot migrate-nfc-filenames binary walks storage.files, NFC-normalizes any non-NFC row, and resolves the collisions we've accumulated. Same-content duplicates go to trash (recoverable); different-content collisions get renamed with a .duplicate suffix.
== use of the clean up ==
example of use (do not forget to define env **DATABASE_URL**)
either
`cargo run --bin migrate-nfc-filenames -- --dry-run`
or
`cargo build --bin migrate-nfc-filenames`
`./target/debug/migrate-nfc-filenames --dry-run`
example:
```
% ./target/debug/migrate-nfc-filenames --dry-run
=== NFC filename migration (DRY RUN — no writes) ===
Loaded 543 non-trashed file rows
NORMALIZE 163451b5-5e6c-404b-9b1e-f4b01a2b7269 user=42433185-4717-416d-9a15-4580fff171ec 'Capture d’écran 2026-03-20 à 14.44.50.png' → 'Capture d’écran 2026-03-20 à 14.44.50.png'
NORMALIZE 827dddec-4dd5-48c2-a120-dec5289f7d29 user=969deca6-7935-4f12-a430-4d636b62fa3e 'Capture d’écran 2026-04-03 à 15.43.38.png' → 'Capture d’écran 2026-04-03 à 15.43.38.png'
NORMALIZE 09559934-a620-472d-9ba8-fc3cfeb6dc6f user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-03 à 20.05.38.png' → 'Capture d’écran 2026-06-03 à 20.05.38.png'
NORMALIZE 5ce6dbf9-0562-4758-8783-671aa9069590 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-05 à 11.07.25.png' → 'Capture d’écran 2026-06-05 à 11.07.25.png'
DEDUP newer=26bcf82b-99cc-45c8-9d69-dd7e5c4484ff (trash, same blob) older=df3adc67-a778-424d-a817-b930c75f3b06 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 hash=0d2cc7b0ffce2850
=== Summary ===
scanned : 543
already in NFC : 538
normalized in place (no collision) : 4
dedup-trashed (same content) : 1
renamed to .duplicate : 0
DRY RUN — no rows were written. Re-run without --dry-run to apply.
```
once valid remove --dry-run
on a file or folder change, the etag of each parent will be updated, this is O(n)
this is required to ensure NextCloud client will be aware of changes
In the future a Cursor will be better, this will be for OxiCloud-desktop
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
- add resentd method in share menu item (work for both internal and external users)
- user can disable email notification via his properties
- add env variable from admin to disable notifications
- OIDC JIT define the locale only at user creation, user can so change his preference later
- invited users will inherit inviter's locale
- email will use prefered_locale
- login to a new browser will use prefered_locale
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
magic-link as now 2 modes:
- invitation: long TTL (24), no challenge
- passwordless login: short TTL (10min), cookie challenge to ensure that
user goes back to same browser (no man in the middle capturing email)
Important: anti-enumeration is active only if SMTP is defined, welcome email can be used
otherwise it is a classic registration with ok or conflic if account alrady exists
OXICLOUD_MAGIC_LINK_OPEN_TO_PASSWORD_USERS (default false)
For security I recommand to keep it false
OIDC cannot be bypassed because OIDC may have MFA in place
Backend
- RegisterDto — username and password both become Option<String> with #[serde(default)] so JSON can omit them entirely.
- AuthApplicationService::register — username uniqueness check skipped when None (multiple NULLs OK under the UNIQUE index); password hashing skipped when None; User::new called with the actual Options instead of forcing Some(...).
- auth_handler::register — branches on dto.password.is_none(). With password → existing 201 + UserDto. Without → triggers MagicLinkInviteService::send_login_link(&email) best-effort, then returns 200 + {"message": "Check your email…"}. The
OIDC-mode-disables-password-registration gate now only fires for the password path (email-only signup is still allowed even in OIDC-only mode, because it doesn't store a password).
- magic_link_handler::redirect_target — new 3-way decision tree:
- Resource target (folder invitation) → /#/files/folder/{id} (existing)
- NULL resource + is_external = false → /#/files (the welcome path for new internal users — they have a home folder)
- NULL resource + is_external = true → /#/sharedwithme (the existing external-user landing)
Tests
- New tests/api/registration.hurl with 9 requests covering: classic (with-password) register → 201 + UserDto, email-only register → 200 + uniform message + welcome magic-link captured, redemption → 302 to /#/files + cookies set, profile read → username
absent + is_external: false, resend magic-link works (eligible while passwordless), cleanup deletes both new users.
- Wired into tests/api/run.sh right after auth_login.hurl.
Plan additions
- auth-simplification.md gained PR 22 at the bottom of the PR sequence — device-bound magic-link redemption via challenge cookie + asymmetric TTLs (login: 10 min, invitation: 24 h). Full design recap, schema migration, config knobs
(OXICLOUD_MAGIC_LINK_LOGIN_TTL_MINUTES / _INVITE_TTL_HOURS), and Hurl coverage outline are in the plan. Slots in before PR 21's docs so the architecture page describes the final state from the start.
Checks — cargo fmt, cargo clippy --all-features --all-targets -- -D warnings, cargo test --lib (297 passed), biome, stylelint, tsc, full Hurl suite (16 files) all green.
username: now optional, if defined 2..64 chars
password: now optional (no mode __NO_PASSWORD...__)
oidc: now optional
important: if need Nextcloud, username must be defined
- add possibility to grant an external user.
- route /api/users/{id} added (rate limited for security)
- security: start route limitation for external users
ex: they must not browse /api/users/{id} nor addressbook
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
reflect OIDC
schema migration, User entity additions, and three defense-in-depth gaps closed, with all 280 unit tests and 13 Hurl files green
infrastructure (lettre + EmailSender port).
- Migration migrations/20260612000003_users_username_email_login.sql — adds nullable given_name/family_name columns to auth.users.
- User entity (src/domain/entities/user.rs) — has_login_credential() placeholder-check encapsulation, set_username revalidating setter, given/family-name fields + getters/setters, validate_username widened 32→254 and now accepts email shape.
from_data_full extended with two new params; all 7 callsites in user_pg_repository.rs updated.
- Schema-side legacy guards (src/application/services/auth_application_service.rs) — bumped the duplicated 32-char check in setup_create_admin and admin_create_user to 254 to match.
- Gap #1 (subject_group_service.rs) — add_member now rejects external candidates with an audit-logged AccessDenied. Service gained an Arc<UserPgRepository> field, wired through DI. New integration test test_external_user_cannot_be_added_as_member.
- Gap #2 (user_repository.rs + auth_ports.rs + user_pg_repository.rs) — list_users/search_users gained an include_external: bool param defaulting effectively to false everywhere internal-user-facing. auth_application_service exposes a new
list_users_including_external for the admin surface.
- Gap #3 (pg_acl_engine.rs) — expand_user now SELECTs is_external and skips INTERNAL_GROUP_ID for externals; defaults to is_external=true on missing user to fail closed.
prepare identity service for external users, support of:
- magic_link (url challenge via email)
- self issued oidc (eventually social login)
- open cloud mesh
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
- SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
- DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
- Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
Edouard Vanbelle