Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.
- Add find_events_by_ical_uids / get_contacts_by_uids through every
layer (domain repo trait → PG repo → storage port/adapter → use case
→ handler) using one indexed `= ANY($2)` query, mirroring the
existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
uid_from_multiget_href helper (percent-decoding, case-insensitive
extension strip, collection hrefs rejected). Exact UID matching also
removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
GET /api/address-books/{id}/contacts: optional ListQuery fields
thread through ContactUseCase::list_contacts to a paginated repo
query. Omitted params keep returning the full book (frontend and
DAV listing/sync paths unchanged).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
StoragePath::join deep-cloned the whole Vec<String> (every segment String)
just to append one element. Take self by value and push in place. All callers
pass owned values except PathService::create_file_path, which holds a borrow
and now clones explicitly — the same copy the old &self join already made.
File::with_name / with_folder / with_size took &self and rebuilt the struct,
cloning every carried-over field (id, mime_type, folder_id, blob_hash, ...).
Consume self and mutate only the fields that change. Behaviour is identical;
the fallible builders now drop the input on Err, which is fine for these
rename/move/resize transforms (all current callers replace the file).
Impact is small in practice — with_folder/with_size have no callers and
with_name is test-only, while the one hot join caller (create_file_path)
must copy segments regardless — but the consuming form is the idiomatic one.
Verified: cargo fmt + clippy --all-features --all-targets -D warnings clean;
domain tests (path_service::, entities::file::) pass — 30 + 6.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
TL;DR:
fix duplicate filename via:
```
docker exec <container> migrate-nfc-filenames --dry-run # preview
docker exec <container> migrate-nfc-filenames # execute
```
== issue ==
Last week I uploaded Capture d'écran 2026-06-03 à 20.04.24.png from the web. It synced down to Nextcloud on my Mac. Two minutes later, the Web UI was showing the file twice.
Both rows had:
- the same name
- the same size
- the same content hash
So why two rows? Because to PostgreSQL, the names weren't the same.
Web upload (browser → Postgres):
"é" stored as 1 codepoint (U+00E9) bytes: c3 a9 ← NFC
NiextCloud client (macOS → Postgres):
"é" stored as 2 codepoints (e + U+0301) bytes: 65 cc 81 ← NFD
macOS's APFS keeps filenames in NFD (decomposed); browsers send NFC (composed). Visually é and é are identical. To WHERE name = $1 they're two different keys. Our UNIQUE index on (folder_id, name, user_id) never fired — and the row count quietly drifted every time a Mac user touched an accented
filename.
== The fix is two halves ==
1. No new duplicates — every name-receiving boundary (file upload, NC PUT, rename, MOVE, path lookup) now NFC-normalizes before touching the database. The storage invariant becomes "every stored name is NFC".
2. Clean up existing data — one-shot migrate-nfc-filenames binary walks storage.files, NFC-normalizes any non-NFC row, and resolves the collisions we've accumulated. Same-content duplicates go to trash (recoverable); different-content collisions get renamed with a .duplicate suffix.
== use of the clean up ==
example of use (do not forget to define env **DATABASE_URL**)
either
`cargo run --bin migrate-nfc-filenames -- --dry-run`
or
`cargo build --bin migrate-nfc-filenames`
`./target/debug/migrate-nfc-filenames --dry-run`
example:
```
% ./target/debug/migrate-nfc-filenames --dry-run
=== NFC filename migration (DRY RUN — no writes) ===
Loaded 543 non-trashed file rows
NORMALIZE 163451b5-5e6c-404b-9b1e-f4b01a2b7269 user=42433185-4717-416d-9a15-4580fff171ec 'Capture d’écran 2026-03-20 à 14.44.50.png' → 'Capture d’écran 2026-03-20 à 14.44.50.png'
NORMALIZE 827dddec-4dd5-48c2-a120-dec5289f7d29 user=969deca6-7935-4f12-a430-4d636b62fa3e 'Capture d’écran 2026-04-03 à 15.43.38.png' → 'Capture d’écran 2026-04-03 à 15.43.38.png'
NORMALIZE 09559934-a620-472d-9ba8-fc3cfeb6dc6f user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-03 à 20.05.38.png' → 'Capture d’écran 2026-06-03 à 20.05.38.png'
NORMALIZE 5ce6dbf9-0562-4758-8783-671aa9069590 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-05 à 11.07.25.png' → 'Capture d’écran 2026-06-05 à 11.07.25.png'
DEDUP newer=26bcf82b-99cc-45c8-9d69-dd7e5c4484ff (trash, same blob) older=df3adc67-a778-424d-a817-b930c75f3b06 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 hash=0d2cc7b0ffce2850
=== Summary ===
scanned : 543
already in NFC : 538
normalized in place (no collision) : 4
dedup-trashed (same content) : 1
renamed to .duplicate : 0
DRY RUN — no rows were written. Re-run without --dry-run to apply.
```
once valid remove --dry-run
on a file or folder change, the etag of each parent will be updated, this is O(n)
this is required to ensure NextCloud client will be aware of changes
In the future a Cursor will be better, this will be for OxiCloud-desktop
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
- add resentd method in share menu item (work for both internal and external users)
- user can disable email notification via his properties
- add env variable from admin to disable notifications
- OIDC JIT define the locale only at user creation, user can so change his preference later
- invited users will inherit inviter's locale
- email will use prefered_locale
- login to a new browser will use prefered_locale
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
magic-link as now 2 modes:
- invitation: long TTL (24), no challenge
- passwordless login: short TTL (10min), cookie challenge to ensure that
user goes back to same browser (no man in the middle capturing email)
username: now optional, if defined 2..64 chars
password: now optional (no mode __NO_PASSWORD...__)
oidc: now optional
important: if need Nextcloud, username must be defined
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
reflect OIDC
schema migration, User entity additions, and three defense-in-depth gaps closed, with all 280 unit tests and 13 Hurl files green
infrastructure (lettre + EmailSender port).
- Migration migrations/20260612000003_users_username_email_login.sql — adds nullable given_name/family_name columns to auth.users.
- User entity (src/domain/entities/user.rs) — has_login_credential() placeholder-check encapsulation, set_username revalidating setter, given/family-name fields + getters/setters, validate_username widened 32→254 and now accepts email shape.
from_data_full extended with two new params; all 7 callsites in user_pg_repository.rs updated.
- Schema-side legacy guards (src/application/services/auth_application_service.rs) — bumped the duplicated 32-char check in setup_create_admin and admin_create_user to 254 to match.
- Gap #1 (subject_group_service.rs) — add_member now rejects external candidates with an audit-logged AccessDenied. Service gained an Arc<UserPgRepository> field, wired through DI. New integration test test_external_user_cannot_be_added_as_member.
- Gap #2 (user_repository.rs + auth_ports.rs + user_pg_repository.rs) — list_users/search_users gained an include_external: bool param defaulting effectively to false everywhere internal-user-facing. auth_application_service exposes a new
list_users_including_external for the admin surface.
- Gap #3 (pg_acl_engine.rs) — expand_user now SELECTs is_external and skips INTERNAL_GROUP_ID for externals; defaults to is_external=true on missing user to fail closed.
this implements first version (manageable only by admin right now)
routes:
GET /api/groups
List subject groups (paginated). Admin-only.
POST /api/groups
Create a new ReBAC subject group. Admin-only. The name must match the RFC 5321 local-part shape and be globally unique (case-insensitive).
GET /api/groups/search
Search non-virtual groups by name substring. Authenticated only (no admin role required) — backs the share-dialog recipient autocomplete.
GET /api/groups/{id}
Fetch a single group's details. Admin-only.
DELETE /api/groups/{id}
Delete a group. Cascades to `subject_group_members` (FK) and to `access_grants` rows referencing this group as a subject. Admin-only.
PATCH /api/groups/{id}
Update a group's metadata. Admin-only. v1 only persists name renames.
GET /api/groups/{id}/effective-members
List every user transitively reached through this group (members of members of members, etc.). Used by admin / audit tooling. Admin-only.
GET /api/groups/{id}/members
List the *direct* members of a group (one level only). Admin-only.
POST /api/groups/{id}/members
Add a member to a group. Exactly one of `user_id` / `group_id` must be provided. Adding a group-member runs a write-time cycle check and a nesting-depth check (max 8). Admin-only.
DELETE /api/groups/{id}/members/group/{gid}
Remove a nested group-member from a group. Admin-only.
DELETE /api/groups/{id}/members/user/{uid}
Remove a user-member from a group. Admin-only.
fix hurl
groups
round
groups
- Add src/application/dtos/cursor.rs with three shared types:
· PageCursor trait — default base64url+JSON encode/decode; one bare
impl line per cursor struct
· CursorQuery struct — standard limit/cursor/sort_by query params with
limit_clamped() and decode_cursor<C>() helpers; compose via flatten
· CursorListResponse<T> — standard {items, next_cursor?} envelope with
from_oversized() and with_cursor() builders
- Migrate GrantCursor to impl PageCursor (remove duplicate encode/decode)
- Update GET /api/grants/incoming/resources:
· SharedWithMeQuery now embeds CursorQuery via #[serde(flatten)]
· Replace file/folder nullable pair with ResourceContentDto (untagged
enum) under a single always-present 'resource' field
· SharedWithMeDto is now a type alias for CursorListResponse<SharedWithMeItemDto>
· Handler uses q.paging.limit_clamped() and decode_cursor<GrantCursor>()
- Add docs/architecture/resource-listing.md — authoritative contract for
all listing endpoints (cursor design, SQL keyset WHERE, sort_by naming,
Rust + JS skeletons, compliance table, migration guide)
- Register doc in VitePress sidebar and architecture index
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- create common function `validate_storage_name()` to check files & folder name
- replace previous duplicate check with this one
- better check on create_folder rename_folder (was only checking non empty)
- use correct error class to ensure 400 on API (was 500)
fix: #345
note: no unit test here, but plan e2e test for that
next: improve UI to display error
Security: session hardening
Refresh token rotation with theft detection (family_id)
- Added family_id column to auth.sessions (migration 20260507000000_session_family.sql) grouping all tokens issued from the same login into a family
- On refresh, the new session inherits the parent's family_id
- If a revoked token is replayed (indicates the token was stolen after rotation), the entire family is immediately invalidated and a warning is logged — forcing re-authentication on all devices
SameSite=Strict on refresh cookie
- Access cookie stays SameSite=Lax (needed for top-level navigation)
- Refresh cookie upgraded to SameSite=Strict — it is only ever used for explicit POST to /api/auth/refresh, never via cross-site navigation
Refresh token TTL: 30 days → 7 days
- With rotation, active sessions auto-renew and effectively never expire
- Inactive sessions expire after 7 days instead of 30, reducing the theft window
Five new public endpoints under /api/s/{token}/...:
GET /contents
GET /contents/{folder_id}
GET /file/{file_id}
GET /zip
GET /zip/{folder_id}
All honour the unlock cookie from /verify, so password-protected
folder shares work end-to-end.
Folder/file IDs are validated against the share subtree via a single
ltree containment query (O(log N) on the existing GiST index).
Out-of-scope IDs return 404.
download_shared_file refactored to a Range/304/206/416-aware
serve_share_file helper, shared with the new /file/{file_id}
endpoint. content_disposition extracted from FileHandler so RFC 5987
formatting is identical across auth and share download paths.
The database schema defines owner_id as UUID, but the Calendar entity
stored it as String. This caused a type mismatch error when creating
calendars via CalDAV clients like DAVx:
column "owner_id" is of type uuid but expression is of type text
Changes:
- Change Calendar.owner_id from String to Uuid
- Update new() and with_id() to accept Uuid
- Update owner_id() getter to return &Uuid
- Update belongs_to() to accept &Uuid
- Update calendar_storage_adapter to pass Uuid directly
- Update tests to use Uuid::new_v4()
Fixes#200
The Nextcloud Android client compares ETags before and after upload to
verify its write landed. OxiCloud was returning the stable file UUID as
the ETag, which never changed on content updates, causing false
SYNC_CONFLICT errors on every upload.
Five fixes applied:
1. Thread blob_hash (SHA-256) through File entity, FileDto, all read/write
queries, and all WebDAV/PROPFIND responses as the ETag — changes on
every content update, no DB migration needed.
2. Honor X-OC-Mtime header: parse the client-supplied mtime and use it
for updated_at via COALESCE(to_timestamp($n), NOW()).
3. Disable phantom checksum capability (preferredUploadType/supportedTypes)
that the server never actually implemented, stopping retry loops.
4. Add nc:creation_time and nc:upload_time to PROPFIND responses.
5. Return oc-etag header in chunked upload MOVE (assemble) responses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix all clippy lints (collapsible if, clone on Copy, needless borrow,
redundant bindings, unused params) and apply rustfmt across the codebase.
Update test mocks to match Uuid-based trait signatures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add FileParts struct and File::into_parts() to yield owned fields
- Rewrite From<File> for FileDto to move id, name, path, folder_id,
owner_id by ownership instead of .to_string() copies
- Eliminates ~5 heap allocations per file (5000 saved per 1000-file listing)
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
Implement a complete Nextcloud client compatibility layer so that
Nextcloud desktop/mobile sync clients can connect to OxiCloud.
Key additions:
- Login Flow v2 (device auth) with OIDC bridge support
- WebDAV handler compatible with Nextcloud clients (PROPFIND, GET,
PUT, DELETE, MKCOL, MOVE, COPY, HEAD, PROPPATCH)
- OCS API endpoints (user info, capabilities, notifications stubs,
sharees, unified search)
- Basic Auth middleware with app password verification, account
lockout integration, and blake3-keyed auth cache
- App password management: create, list, revoke via both native
API (JWT-authenticated profile page) and Nextcloud OCS endpoints
- Nextcloud file ID mapping (oc:fileid) with persistent DB storage
- Chunked upload support (Nextcloud v2 chunking protocol)
- Trashbin WebDAV interface
- Avatar (SVG placeholder) and preview (redirect) handlers
- User profile page with app password management UI
- URL user validation on all DAV routes (403 on mismatch)
- Database schema for app_passwords and nextcloud_object_ids tables
All services are behind a `nextcloud.enabled` config flag and
cleanly separated under src/interfaces/nextcloud/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Share password bypass (HIGH): enforce password check in get_shared_link_by_token,
verify_shared_link_password now returns ShareDto only on correct password.
2. WebDAV MOVE ownership (MEDIUM): add assert_owner on destination parent folder
for file moves in both PathResolver and legacy branches.
3. Path traversal defense-in-depth (LOW): add reject_path_traversal() to WebDAV,
CalDAV, and CardDAV handlers rejecting '..' segments at HTTP boundary.
4. Setup race condition (LOW): atomic INSERT ... ON CONFLICT DO NOTHING in
try_claim_initialization prevents duplicate admin creation.
Claude