Delta-upload protocol: negotiate chunks by hash, upload only what changed
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
@@ -0,0 +1,322 @@
|
||||
//! Delta-upload protocol endpoints — "upload only what changed".
|
||||
//!
|
||||
//! Wire surface of [`DeltaUploadService`]; see that module (and
|
||||
//! `docs/delta-upload-protocol.md`) for the protocol and its security
|
||||
//! model. These handlers only authenticate, rate-limit and translate
|
||||
//! the wire formats — every decision lives in the application service.
|
||||
//!
|
||||
//! Chunk-frame wire format (`PUT /api/files/delta/chunks`): the body is a
|
||||
//! sequence of `[u32 big-endian length][length bytes]` frames, one per
|
||||
//! chunk, with `Content-Type: application/octet-stream`. Frames are capped
|
||||
//! at the CDC maximum chunk size (1 MiB) and the whole request at the same
|
||||
//! per-request ceiling as resumable chunk PUTs (`chunk_max_bytes`).
|
||||
|
||||
use axum::{
|
||||
Json,
|
||||
body::Body,
|
||||
extract::State,
|
||||
http::StatusCode,
|
||||
response::{IntoResponse, Response},
|
||||
};
|
||||
use bytes::{Buf, Bytes, BytesMut};
|
||||
use futures::Stream;
|
||||
use std::sync::Arc;
|
||||
use tokio_stream::StreamExt;
|
||||
|
||||
use crate::application::services::delta_upload_service::{
|
||||
DeltaChunksResponse, DeltaCommitOutcome, DeltaCommitRequest, DeltaNegotiateRequest,
|
||||
DeltaNegotiateResponse,
|
||||
};
|
||||
use crate::common::di::AppState;
|
||||
use crate::common::errors::DomainError;
|
||||
use crate::infrastructure::services::dedup_service::CDC_MAX_CHUNK;
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::middleware::auth::AuthUser;
|
||||
use http_body_util::BodyStream;
|
||||
use serde::Serialize;
|
||||
use utoipa::ToSchema;
|
||||
|
||||
/// 409 body of `POST /api/files/delta/commit` when chunks vanished between
|
||||
/// negotiate and commit (GC race) or were never claimable: the client
|
||||
/// uploads exactly these hashes and retries the same commit.
|
||||
#[derive(Debug, Serialize, ToSchema)]
|
||||
pub struct DeltaStillMissingResponse {
|
||||
pub still_missing: Vec<String>,
|
||||
}
|
||||
|
||||
/// Per-caller flood guard shared by the three delta endpoints.
|
||||
fn check_rate_limit(state: &Arc<AppState>, auth_user: &AuthUser) -> Result<(), AppError> {
|
||||
if state
|
||||
.delta_upload_rate_limiter
|
||||
.check_and_increment(&auth_user.id.to_string())
|
||||
.is_err()
|
||||
{
|
||||
tracing::info!(
|
||||
target: "audit",
|
||||
event = "delta_upload.rejected",
|
||||
reason = "rate_limited",
|
||||
caller_id = %auth_user.id,
|
||||
"👮🏻♂️ Delta upload rejected: per-caller rate limit exceeded",
|
||||
);
|
||||
return Err(AppError::new(
|
||||
StatusCode::TOO_MANY_REQUESTS,
|
||||
"Too many delta-upload requests; please retry shortly",
|
||||
"RateLimited",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Parse a `[u32 BE length][bytes]` frame sequence from the request body.
|
||||
///
|
||||
/// Streaming: peak RAM is one frame (≤ 1 MiB) plus one HTTP frame,
|
||||
/// regardless of how many chunks the request carries. `max_total` bounds
|
||||
/// the whole request body.
|
||||
fn parse_chunk_frames(
|
||||
body: Body,
|
||||
max_total: usize,
|
||||
) -> impl Stream<Item = Result<Bytes, DomainError>> + Send {
|
||||
async_stream::try_stream! {
|
||||
let mut body_stream = BodyStream::new(body);
|
||||
let mut buf = BytesMut::new();
|
||||
let mut expecting: Option<usize> = None;
|
||||
let mut total: usize = 0;
|
||||
|
||||
loop {
|
||||
// Drain every complete frame already buffered.
|
||||
loop {
|
||||
match expecting {
|
||||
None => {
|
||||
if buf.len() < 4 {
|
||||
break;
|
||||
}
|
||||
let len =
|
||||
u32::from_be_bytes([buf[0], buf[1], buf[2], buf[3]]) as usize;
|
||||
buf.advance(4);
|
||||
if len == 0 || len > CDC_MAX_CHUNK {
|
||||
Err(DomainError::validation_error(format!(
|
||||
"Chunk frame of {len} bytes out of bounds (1 ..= {CDC_MAX_CHUNK})"
|
||||
)))?;
|
||||
}
|
||||
expecting = Some(len);
|
||||
}
|
||||
Some(len) => {
|
||||
if buf.len() < len {
|
||||
break;
|
||||
}
|
||||
let frame = buf.split_to(len).freeze();
|
||||
expecting = None;
|
||||
yield frame;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
match body_stream.next().await {
|
||||
Some(Ok(http_frame)) => {
|
||||
if let Some(data) = http_frame.data_ref() {
|
||||
total += data.len();
|
||||
if total > max_total {
|
||||
Err(DomainError::validation_error(format!(
|
||||
"Request body exceeds the {max_total}-byte per-request cap; \
|
||||
split the chunk upload into several requests"
|
||||
)))?;
|
||||
}
|
||||
buf.extend_from_slice(data);
|
||||
}
|
||||
}
|
||||
Some(Err(e)) => {
|
||||
Err(DomainError::validation_error(format!(
|
||||
"Failed to read request body: {e}"
|
||||
)))?;
|
||||
}
|
||||
None => {
|
||||
if expecting.is_some() || !buf.is_empty() {
|
||||
Err(DomainError::validation_error(
|
||||
"Truncated chunk frame at end of body",
|
||||
))?;
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
post,
|
||||
path = "/api/files/delta/negotiate",
|
||||
request_body = DeltaNegotiateRequest,
|
||||
responses(
|
||||
(status = 200, description = "Chunks the caller must upload (the rest are claimable without bytes)", body = DeltaNegotiateResponse),
|
||||
(status = 400, description = "Malformed chunk list (hash format, size bounds, count ceiling)"),
|
||||
(status = 429, description = "Rate limited"),
|
||||
),
|
||||
security(("bearerAuth" = [])),
|
||||
tag = "delta-upload"
|
||||
)]
|
||||
pub async fn delta_negotiate(
|
||||
State(state): State<Arc<AppState>>,
|
||||
auth_user: AuthUser,
|
||||
Json(request): Json<DeltaNegotiateRequest>,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
check_rate_limit(&state, &auth_user)?;
|
||||
let response = state
|
||||
.applications
|
||||
.delta_upload_service
|
||||
.negotiate_with_perms(auth_user.id, &request)
|
||||
.await
|
||||
.map_err(AppError::from)?;
|
||||
Ok(Json(response))
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
put,
|
||||
path = "/api/files/delta/chunks",
|
||||
request_body(content_type = "application/octet-stream",
|
||||
description = "Sequence of [u32 BE length][bytes] frames, one per chunk (each ≤ 1 MiB)"),
|
||||
responses(
|
||||
(status = 200, description = "Server-computed identity of every received frame, in order", body = DeltaChunksResponse),
|
||||
(status = 400, description = "Malformed framing, oversized frame, or oversized request"),
|
||||
(status = 429, description = "Rate limited"),
|
||||
),
|
||||
security(("bearerAuth" = [])),
|
||||
tag = "delta-upload"
|
||||
)]
|
||||
pub async fn delta_upload_chunks(
|
||||
State(state): State<Arc<AppState>>,
|
||||
auth_user: AuthUser,
|
||||
body: Body,
|
||||
) -> Result<impl IntoResponse, AppError> {
|
||||
check_rate_limit(&state, &auth_user)?;
|
||||
let frames = parse_chunk_frames(body, state.core.config.storage.chunk_max_bytes);
|
||||
let response = state
|
||||
.applications
|
||||
.delta_upload_service
|
||||
.receive_chunks(frames)
|
||||
.await
|
||||
.map_err(AppError::from)?;
|
||||
Ok(Json(response))
|
||||
}
|
||||
|
||||
#[utoipa::path(
|
||||
post,
|
||||
path = "/api/files/delta/commit",
|
||||
request_body = DeltaCommitRequest,
|
||||
responses(
|
||||
(status = 201, description = "File created from the committed chunk sequence", body = crate::application::dtos::file_dto::FileDto),
|
||||
(status = 200, description = "Existing file's content replaced", body = crate::application::dtos::file_dto::FileDto),
|
||||
(status = 400, description = "Malformed request, or the declared file_hash does not match the chunk sequence"),
|
||||
(status = 404, description = "Target folder/file not found or not accessible"),
|
||||
(status = 409, description = "Chunks not claimable — upload them and retry", body = DeltaStillMissingResponse),
|
||||
(status = 429, description = "Rate limited"),
|
||||
(status = 507, description = "Storage quota exceeded"),
|
||||
),
|
||||
security(("bearerAuth" = [])),
|
||||
tag = "delta-upload"
|
||||
)]
|
||||
pub async fn delta_commit(
|
||||
State(state): State<Arc<AppState>>,
|
||||
auth_user: AuthUser,
|
||||
Json(request): Json<DeltaCommitRequest>,
|
||||
) -> Result<Response, AppError> {
|
||||
check_rate_limit(&state, &auth_user)?;
|
||||
let outcome = state
|
||||
.applications
|
||||
.delta_upload_service
|
||||
.commit_with_perms(auth_user.id, request)
|
||||
.await
|
||||
.map_err(AppError::from)?;
|
||||
Ok(match outcome {
|
||||
DeltaCommitOutcome::Done { file, created } => {
|
||||
let status = if created {
|
||||
StatusCode::CREATED
|
||||
} else {
|
||||
StatusCode::OK
|
||||
};
|
||||
(status, Json(file)).into_response()
|
||||
}
|
||||
DeltaCommitOutcome::StillMissing(still_missing) => (
|
||||
StatusCode::CONFLICT,
|
||||
Json(DeltaStillMissingResponse { still_missing }),
|
||||
)
|
||||
.into_response(),
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Encode frames the way a client would.
|
||||
fn encode(frames: &[&[u8]]) -> Vec<u8> {
|
||||
let mut out = Vec::new();
|
||||
for f in frames {
|
||||
out.extend_from_slice(&(f.len() as u32).to_be_bytes());
|
||||
out.extend_from_slice(f);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
async fn collect(body: Body, max_total: usize) -> Result<Vec<Bytes>, DomainError> {
|
||||
let stream = parse_chunk_frames(body, max_total);
|
||||
futures::pin_mut!(stream);
|
||||
let mut out = Vec::new();
|
||||
while let Some(item) = stream.next().await {
|
||||
out.push(item?);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn roundtrips_frames_in_order() {
|
||||
let wire = encode(&[b"first", b"second chunk", &[0xAB; 1000]]);
|
||||
let frames = collect(Body::from(wire), usize::MAX).await.unwrap();
|
||||
assert_eq!(frames.len(), 3);
|
||||
assert_eq!(&frames[0][..], b"first");
|
||||
assert_eq!(&frames[1][..], b"second chunk");
|
||||
assert_eq!(frames[2].len(), 1000);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_body_yields_no_frames() {
|
||||
let frames = collect(Body::empty(), usize::MAX).await.unwrap();
|
||||
assert!(frames.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_zero_length_frame() {
|
||||
let wire = encode(&[b""]);
|
||||
assert!(collect(Body::from(wire), usize::MAX).await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_frame_above_cdc_max() {
|
||||
let mut wire = Vec::new();
|
||||
wire.extend_from_slice(&((CDC_MAX_CHUNK as u32) + 1).to_be_bytes());
|
||||
// Header alone is enough — the length is rejected before any data.
|
||||
assert!(collect(Body::from(wire), usize::MAX).await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_truncated_frame() {
|
||||
let mut wire = encode(&[b"complete"]);
|
||||
wire.extend_from_slice(&10u32.to_be_bytes());
|
||||
wire.extend_from_slice(b"only5"); // promises 10, delivers 5
|
||||
assert!(collect(Body::from(wire), usize::MAX).await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rejects_request_above_total_cap() {
|
||||
let wire = encode(&[&[0u8; 600], &[1u8; 600]]);
|
||||
assert!(collect(Body::from(wire), 1000).await.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn accepts_frame_exactly_at_cdc_max() {
|
||||
let big = vec![7u8; CDC_MAX_CHUNK];
|
||||
let wire = encode(&[&big]);
|
||||
let frames = collect(Body::from(wire), usize::MAX).await.unwrap();
|
||||
assert_eq!(frames.len(), 1);
|
||||
assert_eq!(frames[0].len(), CDC_MAX_CHUNK);
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ pub mod carddav_handler;
|
||||
pub mod chunked_upload_handler;
|
||||
pub mod contacts_handler;
|
||||
pub mod dedup_handler;
|
||||
pub mod delta_upload_handler;
|
||||
pub mod device_auth_handler;
|
||||
pub mod favorites_handler;
|
||||
pub mod file_handler;
|
||||
|
||||
@@ -81,6 +81,9 @@ use crate::interfaces::api::handlers::file_handler::MoveFilePayload;
|
||||
handlers::file_handler::list_files_query,
|
||||
handlers::file_handler::upload_file_with_thumbnails,
|
||||
handlers::file_handler::create_file_by_hash,
|
||||
handlers::delta_upload_handler::delta_negotiate,
|
||||
handlers::delta_upload_handler::delta_upload_chunks,
|
||||
handlers::delta_upload_handler::delta_commit,
|
||||
handlers::file_handler::download_file,
|
||||
handlers::file_handler::get_thumbnail,
|
||||
handlers::file_handler::upload_thumbnail,
|
||||
@@ -262,6 +265,13 @@ use crate::interfaces::api::handlers::file_handler::MoveFilePayload;
|
||||
ResourceContentDto,
|
||||
// File schemas
|
||||
FileDto,
|
||||
// Delta-upload schemas
|
||||
crate::application::services::delta_upload_service::ChunkRef,
|
||||
crate::application::services::delta_upload_service::DeltaNegotiateRequest,
|
||||
crate::application::services::delta_upload_service::DeltaNegotiateResponse,
|
||||
crate::application::services::delta_upload_service::DeltaChunksResponse,
|
||||
crate::application::services::delta_upload_service::DeltaCommitRequest,
|
||||
handlers::delta_upload_handler::DeltaStillMissingResponse,
|
||||
MoveFilePayload,
|
||||
PaginationDto,
|
||||
PaginationRequestDto,
|
||||
|
||||
@@ -54,6 +54,9 @@ use crate::interfaces::api::handlers::batch_handler::{self, BatchHandlerState};
|
||||
use crate::interfaces::api::handlers::chunked_upload_handler::{
|
||||
cancel_upload, complete_upload, create_upload, get_upload_status, upload_chunk,
|
||||
};
|
||||
use crate::interfaces::api::handlers::delta_upload_handler::{
|
||||
delta_commit, delta_negotiate, delta_upload_chunks,
|
||||
};
|
||||
use crate::interfaces::api::handlers::file_handler::{
|
||||
create_file_by_hash, delete_file, download_file, get_file_metadata, get_thumbnail,
|
||||
list_files_query, move_file_simple, rename_file, upload_file_with_thumbnails, upload_thumbnail,
|
||||
@@ -230,6 +233,9 @@ pub fn create_api_routes(app_state: &Arc<AppState>) -> Router<Arc<AppState>> {
|
||||
.route("/", get(list_files_query))
|
||||
.route("/upload", post(upload_file_with_thumbnails))
|
||||
.route("/by-hash", post(create_file_by_hash))
|
||||
.route("/delta/negotiate", post(delta_negotiate))
|
||||
.route("/delta/chunks", put(delta_upload_chunks))
|
||||
.route("/delta/commit", post(delta_commit))
|
||||
.route("/{id}", get(download_file))
|
||||
.route(
|
||||
"/{id}/thumbnail/{size}",
|
||||
|
||||
Reference in New Issue
Block a user