plan:
```
What step 5 is actually doing
Goal: stop writing to storage.files.user_id / storage.folders.user_id on new INSERTs. This is a prerequisite for step 6 (dropping the column entirely). The column has existed since the pre-D0 single-owner era; every read that used to key on it has already been migrated to drive-membership grants over the last N days.
What has to change together for this to be safe:
┌──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Piece │ What changes │ Why │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Swap storage.files uniqueness indexes from (folder_id, name, user_id) to (drive_id, folder_id, name) │ Otherwise, two new rows with user_id = NULL would both be allowed (PG treats NULLs as distinct) — │
│ (a) │ │ uniqueness silently breaks │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ ALTER user_id DROP NOT NULL on both tables │ Otherwise, dropping the INSERT bind violates NOT NULL and every write 500s │
│ (b) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Migration │ Drop dead user_id-leading indexes │ Cheap cleanup — nothing scans them anymore │
│ (c) │ │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ PL/pgSQL (d) │ Rewrite storage.copy_folder_tree without user_id in the INSERT column list │ Cross-drive copy runs entirely in SQL, needs the same treatment │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (e) │ ~12 INSERT column-list drops across folder_db_repository, file_blob_write_repository, drive_pg_repository, dedup_service, │ Actual write path │
│ │ folder_service, load-seed.rs │ │
├──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Rust (f) │ Simplify folder_db_repository::create_folder's parent lookup to only fetch drive_id (was fetching (user_id, drive_id)) │ It's fetching a value it no longer needs │
└──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
repl user_id by caller has read access in readonly functions
using CALLER_CAN_READ_DRIVE constant
ensure webdav preview is using the permission handler
add `include_in_photo_index` and `include_in_music_index` policies
both true for default personal drive
photo is implemented
music is not yet implemented
ensure that dead properties are correctly deleted on resource deletion
**IMPORTANT**: this is a partial fix:
dead properties are not deleted if resource is deleted from API !
code need to be reviewed to attach property directly to resource + use on delete cascade
Complete RFC 4918 litmus test suite compliance (basic 16/16,
copymove 13/13, props 30/30) by fixing six categories of failures:
- PropPatchOp: parse PROPPATCH in document order (RFC 4918 §9.2)
so that remove-then-set and set-then-remove yield different results
- Null namespace: resolve_name now correctly maps xmlns="" to empty-ns
QualifiedName instead of falling through to DAV: namespace
- Unicode chars: handle quick-xml 0.39's Event::GeneralRef for character
references (𐀀) — the parser emits these as GeneralRef, not Text
- MOVE preserves dead props: call rename_resource on MOVE, clearing
stale destination data even when source has no registered properties
- Malformed PROPFIND: return 400 when body lacks a complete <propfind>
element (test 2: bare <foo>, test 3: invalid xmlns:prefix="" binding)
- PROPPATCH document order: process ops via Vec<PropPatchOp> instead of
separate (sets, removes) to honour interleaved remove/set sequences
- COPY: return 403 on self-copy, 409 when destination parent missing,
409 (not 500) when overwriting a locked or conflicting resource
- MOVE: same parent-missing and conflict handling as COPY
- MKCOL: return 405 when collection already exists, 409 when parent
is missing (no auto-creation of ancestors)
- PUT: return 201 Created for new resources, 204 No Content for overwrites
- PROPFIND: use client-facing URI path (not internal home-folder path)
for DAV:href values so responses match the request URI
- PROPFIND: include dead properties from DeadPropertyStore in responses
- PROPPATCH: persist set/remove operations to DeadPropertyStore
- Return unknown properties in a separate 404 propstat (RFC 4918 §9.2)
via folder_prop_is_known/file_prop_is_known classifiers
- Detect unclosed <propfind> element and return 400 Bad Request
- Add write_folder/file_entry_with_dead_props to include stored dead
properties in PROPFIND responses
- Change write_folder/file_requested_props to accept &[&QualifiedName]
to avoid clone overhead when using partition output
Add an in-memory DeadPropertyStore backed by RwLock<HashMap> that
stores arbitrary client-supplied XML properties per resource path.
Wire it into AppState so PROPPATCH can persist dead props and PROPFIND
can retrieve them across requests.
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
Stamp `orphaned_at = now()` on the ref-0 row so it sits inside the
GC grace window for the duration of this test. Without it,
`orphaned_at IS NULL` is treated by `garbage_collect` as
"pre-migration, immediately reapable" — and any sibling test in
the shared pool that calls `garbage_collect()` (e.g.
`garbage_collect_respects_grace_and_cross_checks`) would race
with the pin below and delete the row first.
call fire_blob_hooks to respect lifecycle
dedup_service::garbage_collect_with_grac must call fire_blob_hooks() once blob are dropped
so other services like thumbnail can proceed to their cleanup
renable thumbnail test, ensure that blob lifecycle correctly
trigger thumbnail cleanup on blob deletion
need to call `/api/admin/internal/trigger-gc?force=true`
- quota per drive
- add 2 internal API endpoints to test purpose
disabled by default, enable it via `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
this enable:
/api/admin/internal/trigger-sweep
to sweep the trash and recalculated quota
/api/admin/internal/trigger-gc
to garbage orphan blobs
use full for end to end tests and validate lifecycles
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
This reverts ee51b32. The create_new change showed no measurable throughput
benefit — three 9-rep interleaved runs on the same ext4 device swung −12%..+21%
at the 256 KiB CDC size (a negative stat on a warm dentry cache is ~µs, below
the shared-disk noise floor). Applying the same "no change without a measured
win" bar used for the pool-sizing revert: the idiom/TOCTOU angle is real but the
race is already prevented upstream by the PG pin-or-classify serialisation, so
it's defence-in-depth only — not enough to keep an unmeasured change. Reverts
the production change, the bench, and its doc together.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the try_exists(stat) + File::create(O_CREAT|O_TRUNC) pair with a single
OpenOptions::create_new (O_CREAT|O_EXCL), treating AlreadyExists as the existing
idempotent skip. One metadata syscall per new chunk instead of two (each a
spawn_blocking round-trip), and O_EXCL closes the check-then-create TOCTOU the
old pair left open (a racing writer could be truncated).
Honest measurement caveat (benches/BLOB-WRITE.md): the wall-clock throughput
effect is BELOW the noise floor of the test environment — three 9-rep
interleaved runs on the same ext4 device swing −12%..+21% at the 256 KiB CDC
size, because a negative stat on a warm dentry cache is ~µs, dwarfed by the
chunk's create+write+flush. So this is justified as a code-quality / correctness
change (canonical idiom, strictly fewer syscalls, closes a TOCTOU, zero
downside), NOT as a benchmarked perf win.
The sibling idea — reusing the File handle for the fsync sweep — is deliberately
NOT done: sync_blobs is a single end-of-stream sweep over all the upload's new
hashes, so retaining handles would hold thousands of FDs open (>ulimit) on a
large upload. The re-open sweep is a deliberate FD-frugal design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
This reverts the image-pool migration (commit 5629ba6). The bench
(bench_pool_concurrency / POOL-CONCURRENCY.md) measured the one pool it could
isolate — the thumbnail decode semaphore — and found flat throughput, p99 AND
peak RSS (137 MiB) from K=1..16: shrink-on-load already makes each decode
RAM-cheap, so sizing it to the CFS quota gains nothing measurable. Adding code
without a measured benefit isn't worth it.
Kept: the effective_parallelism() helper (it has a *measured* win in the Tokio
runtime — benches/RUNTIME.md) and the benchmark itself (reusable). The ffmpeg
video fan-out has a plausible a-priori case (one OS process per permit) but is
left as a future, deliberately-measured change rather than shipped on
speculation. Doc updated to record the decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Edouard Vanbelle