Stream uploads directly into the CDC chunk store (no spool, single write)
Every upload surface previously wrote each byte to disk twice: the HTTP body was spooled to a temp file (or assembled from chunk parts), then mmap-re-read for FastCDC analysis, and finally the new chunks were written to the blob backend. CDC could not start until the last byte arrived, so large uploads paid receive + reread + rewrite latency. The dedup engine now chunks, hashes and settles the stream WHILE it arrives (fastcdc AsyncStreamCDC + incremental BLAKE3): - Each batch of distinct chunks is pinned-or-classified by ONE `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't be reclaimed mid-upload), and only chunks the store doesn't have are written — a full dedup hit performs zero content writes. - Durability before visibility is preserved: one batched fsync sweep, then one batched INSERT, then the manifest. Identical concurrent uploads are resolved at the manifest INSERT via ON CONFLICT (the loser releases its references and becomes a dedup hit). - A drop guard rolls back pins and surfaces written-but-unregistered chunks to GC if the request future is cancelled mid-stream. - MIME sniffing now peeks the first bytes in-flight; client-requested MD5/SHA-256 checksums are computed by a stream tee — the post-upload re-read of the assembled file is gone. All surfaces converge on the new interfaces::upload_ingest helper: REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup endpoint, and both chunked-upload completions (which now stream their ordered parts straight into the store instead of writing an assembled file — chunk parts persist until finalize, so completion is genuinely retryable). The legacy blob re-chunk migration streams from the backend with no spool file either. Legacy removed: store_from_file + mmap CDC analysers + temp-path plumbing through every port (pre_computed_hash, save_file_from_temp, update_file_content_from_temp), upload_spool + assembled-file assembly in both chunked services, create_file/update_file byte-slice variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR config, and the memmap2 dependency. Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks), identical re-upload (dedup hit, zero writes), 3-byte edit re-upload (26 chunks, 1 written), byte-identical downloads, Range across chunk boundaries, concurrent identical-upload race (manifest ref 2), and trash-empty reclaiming exactly the unshared chunk while the shared 25 survive for the edited file. The empty/sub-8KB multipart path found a post-EOF re-poll panic in the MIME peek (fixed with fuse + regression test). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
@@ -21,6 +21,7 @@ use crate::common::di::AppState;
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::middleware::auth::AuthUser;
|
||||
use crate::interfaces::range_requests::not_modified_response;
|
||||
use crate::interfaces::upload_ingest;
|
||||
use crate::{application::dtos::file_dto::FileDto, domain::services::authorization::Permission};
|
||||
use std::sync::Arc;
|
||||
|
||||
@@ -51,11 +52,12 @@ impl FileHandler {
|
||||
// UPLOAD
|
||||
// ═══════════════════════════════════════════════════════════════════════
|
||||
|
||||
/// Streaming file upload — constant ~64 KB RAM regardless of file size.
|
||||
/// Streaming file upload — bounded RAM regardless of file size.
|
||||
///
|
||||
/// **Hash-on-Write**: BLAKE3 is computed while spooling the multipart
|
||||
/// body to the temp file. This eliminates the second sequential read
|
||||
/// that dedup_service would otherwise need, cutting total I/O in half.
|
||||
/// The multipart body is streamed straight into the CDC chunk store:
|
||||
/// chunking, hashing and dedup checks happen while the bytes arrive.
|
||||
/// No spool file, no re-read — chunks the store already has are never
|
||||
/// written to disk at all.
|
||||
pub async fn upload_file(
|
||||
State(state): State<GlobalState>,
|
||||
auth_user: AuthUser,
|
||||
@@ -71,7 +73,7 @@ impl FileHandler {
|
||||
/// [`Self::upload_file_with_thumbnails`].
|
||||
///
|
||||
/// Returns `(FileDto, blob_hash)` on success. The blob hash is the
|
||||
/// BLAKE3 digest computed during the hash-on-write spool and is
|
||||
/// BLAKE3 digest computed during the streaming ingest and is
|
||||
/// propagated without an extra database round-trip so that callers
|
||||
/// (e.g. thumbnail generation) can resolve the physical blob path
|
||||
/// immediately.
|
||||
@@ -158,120 +160,55 @@ impl FileHandler {
|
||||
}
|
||||
}
|
||||
|
||||
// ── Spool multipart field to temp file + hash-on-write ──
|
||||
// .dedup_temp is created once by DedupService::initialize() at startup
|
||||
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp");
|
||||
let temp_path = temp_dir.join(format!("upload-{}", uuid::Uuid::new_v4()));
|
||||
|
||||
let mut total_size: u64 = 0;
|
||||
let mut hasher = blake3::Hasher::new();
|
||||
let spool_result: Result<(), String> = async {
|
||||
let file = tokio::fs::File::create(&temp_path)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to create temp file: {}", e))?;
|
||||
|
||||
// Pre-allocate if Content-Length is known (reduces fragmentation)
|
||||
let hint = field
|
||||
.headers()
|
||||
.get(axum::http::header::CONTENT_LENGTH)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|s| s.parse::<u64>().ok());
|
||||
if let Some(len) = hint {
|
||||
let _ = file.set_len(len).await; // best-effort
|
||||
}
|
||||
|
||||
// 512 KB buffer — 8× fewer write syscalls than 64 KB
|
||||
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file);
|
||||
let mut field = field;
|
||||
// IMPORTANT: use explicit match instead of `while let Ok(Some(..))`.
|
||||
// The old pattern silently swallowed Err (client disconnect)
|
||||
// and accepted partially received data as a complete upload.
|
||||
loop {
|
||||
match field.chunk().await {
|
||||
Ok(Some(chunk)) => {
|
||||
total_size += chunk.len() as u64;
|
||||
hasher.update(&chunk);
|
||||
tokio::io::AsyncWriteExt::write_all(&mut writer, &chunk)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to write chunk: {}", e))?;
|
||||
}
|
||||
Ok(None) => break, // End of field — upload complete
|
||||
Err(e) => {
|
||||
return Err(format!(
|
||||
"Connection lost during upload (received {} bytes): {}",
|
||||
total_size, e
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
tokio::io::AsyncWriteExt::flush(&mut writer)
|
||||
.await
|
||||
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
|
||||
Ok(())
|
||||
}
|
||||
.await;
|
||||
|
||||
if let Err(e) = spool_result {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("❌ UPLOAD SPOOL FAILED: {} - {}", filename, e);
|
||||
return Err(Self::domain_error_response(
|
||||
crate::common::errors::DomainError::internal_error("FileUpload", e),
|
||||
));
|
||||
}
|
||||
|
||||
// Empty file — use streaming path with the (empty) temp file
|
||||
if total_size == 0 {
|
||||
let hash = hasher.finalize().to_hex().to_string();
|
||||
let dto = upload_service
|
||||
.upload_file_streaming(
|
||||
filename,
|
||||
folder_id,
|
||||
content_type,
|
||||
&temp_path,
|
||||
0,
|
||||
Some(hash.clone()),
|
||||
)
|
||||
.await
|
||||
.map_err(Self::domain_error_response)?;
|
||||
return Ok((dto, hash));
|
||||
}
|
||||
|
||||
// Finalize hash
|
||||
let hash = hasher.finalize().to_hex().to_string();
|
||||
|
||||
// ── MIME detection (magic bytes + extension fallback) ─
|
||||
let content_type = crate::common::mime_detect::refine_content_type_from_file(
|
||||
&temp_path,
|
||||
// ── Stream the field into the CDC chunk store ────────
|
||||
// Chunking (FastCDC) + hashing (BLAKE3) + dedup checks +
|
||||
// MIME sniffing all happen while the bytes arrive; chunks
|
||||
// the store already has never touch the disk. Size is
|
||||
// capped globally by DefaultBodyLimit.
|
||||
let dedup = &state.core.dedup_service;
|
||||
let source = upload_ingest::multipart_field_stream(field);
|
||||
let ingested = match upload_ingest::ingest_stream_to_cas(
|
||||
source,
|
||||
dedup,
|
||||
&filename,
|
||||
&content_type,
|
||||
usize::MAX,
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
.await
|
||||
{
|
||||
Ok(ingested) => ingested,
|
||||
Err(e) => {
|
||||
tracing::error!("❌ UPLOAD INGEST FAILED: {} - {}", filename, e.message);
|
||||
return Err(e.into_response());
|
||||
}
|
||||
};
|
||||
|
||||
// ── Quota enforcement ────────────────────────────────
|
||||
// ── Quota enforcement (exact size now known) ─────────
|
||||
if let Some(storage_svc) = state.storage_usage_service.as_ref()
|
||||
&& let Err(err) = storage_svc
|
||||
.check_storage_quota(auth_user.id, total_size)
|
||||
.check_storage_quota(auth_user.id, ingested.size)
|
||||
.await
|
||||
{
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
upload_ingest::discard_ingested(dedup, &ingested).await;
|
||||
tracing::warn!(
|
||||
"⛔ UPLOAD REJECTED (quota): user={}, file={}, size={}",
|
||||
auth_user.username,
|
||||
filename,
|
||||
total_size
|
||||
ingested.size
|
||||
);
|
||||
return Err(Self::quota_error_response(err));
|
||||
}
|
||||
|
||||
// ── Streaming upload (temp file → blob store, hash pre-computed) ─
|
||||
// ── Register the file row against the ingested blob ──
|
||||
let hash = ingested.hash.clone();
|
||||
let size = ingested.size;
|
||||
match upload_service
|
||||
.upload_file_streaming(
|
||||
filename.clone(),
|
||||
folder_id,
|
||||
content_type,
|
||||
&temp_path,
|
||||
total_size,
|
||||
Some(hash.clone()),
|
||||
ingested.content_type.clone(),
|
||||
ingested.stored(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -279,13 +216,12 @@ impl FileHandler {
|
||||
tracing::info!(
|
||||
"✅ STREAMING UPLOAD: {} ({} bytes, ID: {})",
|
||||
filename,
|
||||
total_size,
|
||||
size,
|
||||
file.id
|
||||
);
|
||||
return Ok((file, hash));
|
||||
}
|
||||
Err(err) => {
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
tracing::error!("❌ UPLOAD FAILED: {} - {}", filename, err);
|
||||
return Err(Self::domain_error_response(err));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user