Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
to help administratorrs to defined correctly their storage architecture
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
- add resentd method in share menu item (work for both internal and external users)
- user can disable email notification via his properties
- add env variable from admin to disable notifications
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
1. Identity model — email is the identity, three optional credential slots (username, password_hash, oidc_subject), the @-in-username ban that makes namespaces provably disjoint.
2. Login dispatcher — @-in-input decides the path. Single DB lookup.
3. Login paths — four ways in (username+password, email+password, email+magic-link, OIDC redirect) with their pre-conditions.
4. Magic-link eligibility ladder — the three-branch table with OIDC unconditional reject, password flag-gated, no-credential allowed.
5. Registration paths — four ways (/api/auth/register with full creds, email-only, grant-by-email invitation, OIDC JIT).
6. Anti-enumeration — per-endpoint table showing what each surface returns and why; explicit note that instance-wide policy stays visible.
7. Security trade-offs — mailbox-as-bypass (lenient vs strict), no native MFA, magic-link as bearer token, enumeration via timing.
8. Rate limits — the five caps from PR 12 + classic auth, with env knobs.
9. Audit events — table of 12 event names with reason keys; every entry verified grep-able in src/.
10. Migration path — what existing instances see when PR 16-20 land (forward-only, non-destructive).
11. Future direction — login_strategy — the seven-row matrix sketch as the explicit-policy north star.
12. Out of scope — TOTP/WebAuthn, external→internal promotion, session_kind, differentiated session TTL, OCM, email-verified gates (PR 23), device-bound tokens (PR 22), anti-enum timing parity, per-user opt-out, login_strategy implementation.
13. Related documents — cross-refs to magic-link-auth, ReBAC, share-integration, env-vars.
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
prepare identity service for external users, support of:
- magic_link (url challenge via email)
- self issued oidc (eventually social login)
- open cloud mesh
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
- SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
- DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
- Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
this implements first version (manageable only by admin right now)
routes:
GET /api/groups
List subject groups (paginated). Admin-only.
POST /api/groups
Create a new ReBAC subject group. Admin-only. The name must match the RFC 5321 local-part shape and be globally unique (case-insensitive).
GET /api/groups/search
Search non-virtual groups by name substring. Authenticated only (no admin role required) — backs the share-dialog recipient autocomplete.
GET /api/groups/{id}
Fetch a single group's details. Admin-only.
DELETE /api/groups/{id}
Delete a group. Cascades to `subject_group_members` (FK) and to `access_grants` rows referencing this group as a subject. Admin-only.
PATCH /api/groups/{id}
Update a group's metadata. Admin-only. v1 only persists name renames.
GET /api/groups/{id}/effective-members
List every user transitively reached through this group (members of members of members, etc.). Used by admin / audit tooling. Admin-only.
GET /api/groups/{id}/members
List the *direct* members of a group (one level only). Admin-only.
POST /api/groups/{id}/members
Add a member to a group. Exactly one of `user_id` / `group_id` must be provided. Adding a group-member runs a write-time cycle check and a nesting-depth check (max 8). Admin-only.
DELETE /api/groups/{id}/members/group/{gid}
Remove a nested group-member from a group. Admin-only.
DELETE /api/groups/{id}/members/user/{uid}
Remove a user-member from a group. Admin-only.
fix hurl
groups
round
groups
initially server was accepting multiple instances on same port (Linux and MacOS only)
this can create issues during development (if a dev forget another running instance...)
Add OXICLOUD_REUSE_PORT variable to activate it, so only admins knowing this feature can activate itt
(permits multiple instance + let the OS schedduler to decide which process will handle a request)
if not enabled, other instance will receive exit with a:
`Error: Os { code: 48, kind: AddrInUse, message: "Address already in use" }`
- Add src/application/dtos/cursor.rs with three shared types:
· PageCursor trait — default base64url+JSON encode/decode; one bare
impl line per cursor struct
· CursorQuery struct — standard limit/cursor/sort_by query params with
limit_clamped() and decode_cursor<C>() helpers; compose via flatten
· CursorListResponse<T> — standard {items, next_cursor?} envelope with
from_oversized() and with_cursor() builders
- Migrate GrantCursor to impl PageCursor (remove duplicate encode/decode)
- Update GET /api/grants/incoming/resources:
· SharedWithMeQuery now embeds CursorQuery via #[serde(flatten)]
· Replace file/folder nullable pair with ResourceContentDto (untagged
enum) under a single always-present 'resource' field
· SharedWithMeDto is now a type alias for CursorListResponse<SharedWithMeItemDto>
· Handler uses q.paging.limit_clamped() and decode_cursor<GrantCursor>()
- Add docs/architecture/resource-listing.md — authoritative contract for
all listing endpoints (cursor design, SQL keyset WHERE, sort_by naming,
Rust + JS skeletons, compliance table, migration guide)
- Register doc in VitePress sidebar and architecture index
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* make more coherent lifecycles
* remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
* reduce risk of orphean objects
* ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
* more details on docs/architecture/file-and-blob-lifecycle.md :
```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {
fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
fn on_file_updated(file_id, blob_hash, content_type);
fn on_file_copied(file_id, blob_hash, content_type, source_id)
fn on_file_deleted(file_id);
}
// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {
fn on_blob_created(blob_hash, content_type);
fn on_blob_deleted(blob_hash);
}
```
Claude