Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
+5
-13
@@ -55,22 +55,14 @@ OXICLOUD_SERVER_HOST=127.0.0.1
|
||||
#OXICLOUD_CHUNK_MAX_BYTES=104857600
|
||||
|
||||
# ── Upload spool directories ─────────────────────────────────────────
|
||||
# Where in-flight uploads land BEFORE being promoted into final blob
|
||||
# storage. Same-filesystem placement (with `.blobs/`) makes the
|
||||
# promotion an atomic rename(2); NVMe placement speeds up intake.
|
||||
# Direct (non-chunked) uploads stream straight into the blob store —
|
||||
# no spool directory. Only chunked-upload sessions accumulate on disk.
|
||||
# See docs/config/storage-fine-tuning.md for layout examples.
|
||||
|
||||
# Directory for non-chunked PUT spool tempfiles. Default: OS temp dir
|
||||
# ($TMPDIR / /tmp), often tmpfs (RAM) in containers — writing a large
|
||||
# upload there fills page-cache that counts against the cgroup memory
|
||||
# limit and can OOMKill the process. Point this at a real-disk path
|
||||
# (same FS as the storage backend is ideal).
|
||||
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
|
||||
|
||||
# Root directory for chunked-upload sessions (REST + NextCloud chunked
|
||||
# share this root). Default: {STORAGE_PATH}/.uploads. Pointing this at
|
||||
# NVMe accelerates the chunk-write + assembly loop; pointing it at the
|
||||
# same FS as `.blobs/` makes blob promotion atomic.
|
||||
# share this root). Default: {STORAGE_PATH}/.uploads. Avoid tmpfs in
|
||||
# containers (parts count against the cgroup memory limit); NVMe
|
||||
# placement accelerates chunk PUTs and the /complete streaming pass.
|
||||
#OXICLOUD_CHUNK_DIR=/var/lib/oxicloud/.uploads
|
||||
|
||||
# How often (seconds) the background sweep reconciles each user's cached
|
||||
|
||||
Reference in New Issue
Block a user