761d159a92
feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
...
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking
(min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing
- Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[])
- Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends)
- 3-phase store_chunks pipeline:
Phase 0: batch-check existing chunks (single PG query)
Phase 1: selective disk read (skip existing chunks entirely)
Phase 2: parallel upload with buffer_unordered(8)
- CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback
- Transactional manifest + chunk ref-count cascade on remove_reference
- 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.)
- Update deduplication.md to reflect new architecture
2026-04-14 23:17:39 +02:00
e071841ec2
docs: update documentation to reflect 100% blob storage model
...
Rewrite documentation to match the new architecture where all
file metadata lives in PostgreSQL and content is stored as
content-addressed blobs via DedupService.
Updated files:
- internal-architecture.md: complete rewrite — new DB schema,
blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated
DI container, service groups, architecture diagram, data flows
- file-system-safety.md: repurposed as storage-safety.md —
covers PostgreSQL ACID guarantees + DedupService atomic writes
- caching-architecture.md: updated repo references to blob repos,
removed write-behind cache section, updated upload/download flows
- trash-feature-summary.md: rewritten for soft-delete model
(is_trashed flag, trash_items VIEW, TrashDbRepository)
- share-integration.md: clarified ShareFsRepository scope,
updated DI snippet, added blob storage context note
- deduplication.md: updated DI snippet (dedup injected into repos)
- deployment.md: updated feature matrix (file storage requires DB)
- important-delta-sync-implementation.md: updated DI references
Removed legacy references: IdMappingPort, StorageMediator,
WriteBehindCache, FsFileRepository, FsFolderRepository,
TrashFsRepository, folder_ids.json, file_ids.json.
2026-02-14 18:27:30 +01:00
Diocrafts