feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing - Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[]) - Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends) - 3-phase store_chunks pipeline: Phase 0: batch-check existing chunks (single PG query) Phase 1: selective disk read (skip existing chunks entirely) Phase 2: parallel upload with buffer_unordered(8) - CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback - Transactional manifest + chunk ref-count cascade on remove_reference - 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.) - Update deduplication.md to reflect new architecture
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
-- Content-Defined Chunking (CDC) manifests for sub-file deduplication.
|
||||
--
|
||||
-- Each file uploaded via CDC is split into variable-size chunks (FastCDC).
|
||||
-- The manifest records the ordered list of chunk hashes that compose the file.
|
||||
-- Individual chunks are stored in storage.blobs (shared across manifests).
|
||||
--
|
||||
-- Legacy whole-file blobs (pre-CDC) remain in storage.blobs and are
|
||||
-- accessed directly when no matching manifest row exists.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS storage.chunk_manifests (
|
||||
file_hash VARCHAR(64) PRIMARY KEY,
|
||||
chunk_hashes TEXT[] NOT NULL,
|
||||
chunk_sizes BIGINT[] NOT NULL,
|
||||
total_size BIGINT NOT NULL,
|
||||
chunk_count INTEGER NOT NULL,
|
||||
content_type TEXT,
|
||||
ref_count INTEGER NOT NULL DEFAULT 1 CHECK (ref_count >= 0),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Index for GC: find manifests with no references.
|
||||
CREATE INDEX IF NOT EXISTS idx_chunk_manifests_ref_count_zero
|
||||
ON storage.chunk_manifests (file_hash)
|
||||
WHERE ref_count = 0;
|
||||
Reference in New Issue
Block a user