Delta-upload protocol: negotiate chunks by hash, upload only what changed

Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:

- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
  which ones the caller must upload. User-scoped and purely advisory.
- PUT  /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
  frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
  Every hash is recomputed server-side; chunks land as ref_count=0 orphans
  that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
  a single UPDATE…RETURNING restricted to chunks the caller is entitled
  to (reachable through their own non-trashed files, or unreferenced
  orphans); anything else returns 409 {still_missing} for the client to
  upload and retry. The pinned sequence is then RE-READ and the whole-file
  BLAKE3 recomputed before any manifest exists — a declared file_hash is
  never trusted, because a forged manifest would poison future whole-file
  dedup hits for other users. The manifest accounting is shared with the
  byte path (attach_manifest, extracted from store_from_stream); the file
  row is created (201) or its content swapped by file_id (200). Owners of
  the exact file_hash short-circuit to a pure reference bump.

Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.

Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 14:34:02 +00:00
parent 0fab4ce17d
commit 44967da7f1
11 changed files with 1641 additions and 27 deletions
@@ -238,6 +238,71 @@ impl FileUploadService {
Ok(dto)
}
/// Swap an existing file's content to an already-ingested blob — the
/// update mode of the delta-upload commit. The caller needs `Write`
/// permission on the file; the blob reference is consumed (released on
/// failure by the write port, like every other registration path).
pub async fn update_file_content_by_id_with_perms(
&self,
caller_id: Uuid,
file_id: &str,
blob: StoredBlob,
) -> Result<FileDto, DomainError> {
let Some(InstantUploadDeps { authz, .. }) = &self.instant_upload else {
return Err(DomainError::internal_error(
"FileUpload",
"instant upload is not wired (authz/dedup/quota missing)",
));
};
let Some(file_read) = &self.file_read else {
return Err(DomainError::internal_error(
"FileUpload",
"read port is not wired",
));
};
let file_uuid = Uuid::parse_str(file_id)
.map_err(|_| DomainError::not_found("File", file_id.to_string()))?;
authz
.require(
Subject::User(caller_id),
Permission::Update,
Resource::File(file_uuid),
)
.await?;
let file = file_read.get_file(file_id).await?;
let (new_hash, updated_at) = self
.file_write
.update_file_content_with_blob(file_id, &blob.hash, blob.size, None)
.await?;
// The file maps to a different blob now — stale cached content must
// never be served for the rest of its TTI window.
if let Some(cc) = &self.content_cache {
cc.invalidate(file_id).await;
}
let parts = file.into_parts();
let updated = crate::domain::entities::file::File::with_timestamps_and_blob_hash(
parts.id,
parts.name,
parts.storage_path,
blob.size,
parts.mime_type,
parts.folder_id,
parts.created_at,
updated_at as u64,
parts.owner_id,
new_hash,
)
.map_err(|e| DomainError::internal_error("FileUpload", format!("rebuild entity: {e}")))?;
let dto = FileDto::from(updated);
if let Some(hook) = &self.file_lifecycle_hook {
hook.on_file_updated(file_id, &dto.content_hash, &dto.mime_type);
}
Ok(dto)
}
// ── private helpers ──────────────────────────────────────────
/// Optionally update storage usage after a successful upload.