docs: migrate legacy docs to official site

docs/guide/deduplication.md

@@ -1,46 +1,65 @@
 # File Deduplication
 
-OxiCloud uses **SHA-256 content-addressable storage** to avoid storing duplicate files. If two users upload the same file, only one copy is stored on disk.
+OxiCloud uses **content-defined chunking (CDC)** with **FastCDC** and **BLAKE3** to deduplicate files at the sub-file level. Instead of storing only whole-file blobs, OxiCloud can split a file into variable-size chunks, reuse unchanged chunks across versions, and track the ordered chunk list in PostgreSQL.
 
 ## How It Works
 
-1. When a file is uploaded, its SHA-256 hash is computed
-2. The hash is checked against the blob store (`.blobs/{prefix}/{hash}.blob`)
-3. If a blob with that hash already exists, the file metadata points to the existing blob (no extra disk usage)
-4. If not, the content is saved as a new blob
-5. A reference counter tracks how many files point to each blob
+1. OxiCloud analyzes the uploaded file with FastCDC
+2. The file is split into variable-size chunks from **64 KB** to **1 MB**, targeting an average of **256 KB**
+3. Each chunk is hashed with **BLAKE3** and checked against the blob index
+4. Only new chunks are written to the blob backend
+5. A manifest in PostgreSQL maps the whole-file hash to the ordered chunk hash list
+6. Reference counts are updated so identical chunks are stored once even across multiple files or edited versions
 
-## Automatic Cleanup
+## Storage Model
+
+```text
+storage.files      -> metadata rows that reference content
+chunk_manifests    -> file_hash -> [chunk_hashes] + chunk_sizes + ref_count
+storage.blobs      -> per-chunk blob metadata and reference counts
+blob backend       -> actual chunk bytes on disk or remote storage
+```
+
+The manifest table is created in `migrations/20260414000000_chunk_manifests.sql` and keeps:
+
+- `file_hash`
+- ordered `chunk_hashes`
+- `chunk_sizes`
+- `total_size`
+- `chunk_count`
+- `ref_count`
+
+## Why CDC Matters
+
+Whole-file dedup only helps when two files are byte-for-byte identical. CDC helps when files are similar but not identical, for example:
+
+- edited office documents
+- versioned project archives
+- large media files with partial changes
+
+In those cases, unchanged chunks can be reused and only the modified portions need new storage.
+
+## Backward Compatibility
+
+Older uploads stored before CDC are still readable. When OxiCloud does not find a matching manifest row, it falls back to legacy whole-file blob reads.
+
+## Cleanup Behavior
 
 When a file is permanently deleted:
 
-1. The blob's reference count is decremented
-2. If the reference count reaches zero, the blob is removed from disk
+1. OxiCloud decrements the manifest reference count
+2. If the last manifest reference disappears, chunk refcounts are decremented
+3. Chunks with `ref_count = 0` are removed from the blob index and then deleted from the backend
 
-This means disk space is only freed when the **last** reference to a blob is removed.
-
-## Storage Layout
-
-```
-storage/
-├── .blobs/
-│   ├── a1/
-│   │   └── a1b2c3d4...sha256.blob
-│   ├── f8/
-│   │   └── f8e7d6c5...sha256.blob
-│   └── ...
-```
-
-The first two hex characters of the hash are used as a directory prefix to avoid having millions of files in a single directory.
+This keeps storage correct even when multiple files share the same chunk set.
 
 ## Benefits
 
-- **Disk savings** — identical files across users consume storage only once
-- **Instant uploads** — if the blob already exists, the upload completes immediately
-- **Integrity** — SHA-256 ensures bit-for-bit correctness
+- Better storage savings for edited and versioned files
+- Faster repeat uploads when many chunks already exist
+- BLAKE3 hashing for fast content verification
+- PostgreSQL-backed manifests for durable indexing and cleanup
 
-## Limitations
+## Related Endpoints
 
-- Deduplication is based on exact content match (byte-identical files)
-- Near-duplicate files (e.g., a JPEG re-saved at slightly different quality) are stored separately
-- Encryption at rest would require per-user keys, which breaks deduplication (planned as opt-in)
+The dedup subsystem is also exposed through helper endpoints under `/api/dedup` for hash checks, deduplicated uploads, statistics, and maintenance operations.