From e071841ec2456ccd6d1887a8e4d7752c6cebdd34 Mon Sep 17 00:00:00 2001 From: Dionisio Date: Sat, 14 Feb 2026 18:27:30 +0100 Subject: [PATCH] docs: update documentation to reflect 100% blob storage model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrite documentation to match the new architecture where all file metadata lives in PostgreSQL and content is stored as content-addressed blobs via DedupService. Updated files: - internal-architecture.md: complete rewrite — new DB schema, blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated DI container, service groups, architecture diagram, data flows - file-system-safety.md: repurposed as storage-safety.md — covers PostgreSQL ACID guarantees + DedupService atomic writes - caching-architecture.md: updated repo references to blob repos, removed write-behind cache section, updated upload/download flows - trash-feature-summary.md: rewritten for soft-delete model (is_trashed flag, trash_items VIEW, TrashDbRepository) - share-integration.md: clarified ShareFsRepository scope, updated DI snippet, added blob storage context note - deduplication.md: updated DI snippet (dedup injected into repos) - deployment.md: updated feature matrix (file storage requires DB) - important-delta-sync-implementation.md: updated DI references Removed legacy references: IdMappingPort, StorageMediator, WriteBehindCache, FsFileRepository, FsFolderRepository, TrashFsRepository, folder_ids.json, file_ids.json. --- doc/caching-architecture.md | 644 ++++++-------- doc/deduplication.md | 9 +- doc/deployment.md | 8 +- doc/file-system-safety.md | 281 +++--- doc/important-delta-sync-implementation.md | 4 +- doc/internal-architecture.md | 951 ++++++++++----------- doc/share-integration.md | 55 +- doc/trash-feature-summary.md | 196 +++-- 8 files changed, 1033 insertions(+), 1115 deletions(-) diff --git a/doc/caching-architecture.md b/doc/caching-architecture.md index ddf3a288..ab1ecdd6 100644 --- a/doc/caching-architecture.md +++ b/doc/caching-architecture.md @@ -1,370 +1,274 @@ -# 04 - Caching Architecture - -OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads. - -## Cache Layers Summary - -``` -┌─────────────────────────────────────────────────────┐ -│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints -├─────────────────────────────────────────────────────┤ -│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads -├─────────────────────────────────────────────────────┤ -│ Layer 2: MMAP (memmap2, 10-100MB files) │ Downloads -├─────────────────────────────────────────────────────┤ -│ Layer 3: Streaming (FramedRead, ≥100MB files) │ Downloads -├─────────────────────────────────────────────────────┤ -│ Layer 4: File Metadata Cache (adaptive TTL) │ All file ops -├─────────────────────────────────────────────────────┤ -│ Layer 5: Write-Behind Cache (<256KB uploads) │ Uploads -├─────────────────────────────────────────────────────┤ -│ Layer 6: Buffer Pool (reusable I/O buffers) │ Compression -└─────────────────────────────────────────────────────┘ -``` - ---- - -## Layer 0: HTTP Cache Middleware - -**File**: `src/interfaces/middleware/cache.rs` - -Generic HTTP caching layer applied to API endpoints. - -| Parameter | Value | -|---|---| -| Max entries | 1,000 | -| Default max-age | 60 seconds | -| Eviction | LRU (oldest 10% when full) | -| Cleanup | Background task every 5 minutes | - -Features: -- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`) -- `Cache-Control` header injection -- Implements Tower `Layer` + `Service` traits for Axum integration -- Per-request key: method + URI - ---- - -## Layer 1: File Content Cache (Download Tier 1) - -**File**: `src/infrastructure/services/file_content_cache.rs` - -In-memory LRU cache for small files, served directly from RAM. - -| Parameter | Value | -|---|---| -| Max file size | 10 MB per file | -| Max total cache size | 512 MB | -| Max entries | 10,000 | -| Structure | `lru::LruCache` | -| Latency | ~0.1ms | - -**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }` - -Methods: -- `should_cache(size)` -- checks if file fits in cache -- `get(file_id)` → `Option<(Bytes, String, String)>` -- returns (data, etag, content_type) -- `put(file_id, content, etag, content_type)` -- inserts with LRU eviction -- `invalidate(file_id)`, `clear()` -- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }` - -Port: implements **ContentCachePort** trait. - ---- - -## Layer 2: MMAP (Download Tier 2) - -**File**: `src/infrastructure/repositories/file_fs_read_repository.rs` - -Memory-mapped I/O for medium files using `memmap2`. - -| Parameter | Value | -|---|---| -| File range | 10 MB - 100 MB | -| Implementation | `memmap2::Mmap` via `spawn_blocking` | -| Latency | ~1-5ms | - -Current implementation copies mmap'd data to `Bytes` (`Bytes::copy_from_slice(&mmap[..])`). Not true zero-copy, but still benefits from kernel page cache. - ---- - -## Layer 3: Streaming (Download Tier 3) - -**File**: `src/infrastructure/repositories/file_fs_read_repository.rs` - -Chunked streaming for large files using tokio-util codecs. - -| Parameter | Value | -|---|---| -| File range | ≥100 MB | -| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) | -| Implementation | `FramedRead` + `BytesCodec` | -| RAM usage | Near zero (one chunk at a time) | - ---- - -## Layer 4: File Metadata Cache - -**File**: `src/infrastructure/services/file_metadata_cache.rs` - -Caches filesystem metadata (existence, size, MIME type, timestamps) to avoid repeated `stat()` calls. - -| Parameter | Value | -|---|---| -| Default file TTL | 60 seconds | -| Default directory TTL | 120 seconds | -| Max entries | 10,000 | -| Adaptive TTL multiplier | 5x for popular entries (≥10 accesses) | -| LRU eviction | Frees 10% capacity when full | -| Cleanup | Background task runs periodically | - -**CachedMetadata:** -```rust -pub struct FileMetadata { - pub path: PathBuf, - pub exists: bool, - pub entry_type: CacheEntryType, // File | Directory | Unknown - pub size: Option, - pub mime_type: Option, - pub created_at: Option, - pub modified_at: Option, - pub last_access: Instant, - pub expires_at: Instant, - pub access_count: usize, -} -``` - -**Adaptive TTL**: entries accessed ≥10 times get 5x the configured TTL, keeping frequently accessed file metadata in cache longer. - -Port: implements **MetadataCachePort** trait. - ---- - -## Layer 5: Write-Behind Cache - -**File**: `src/infrastructure/services/write_behind_cache.rs` - -Buffers small uploads in RAM and confirms immediately. Flushes to disk asynchronously. - -| Parameter | Value | -|---|---| -| Max file size | 1 MB per file | -| Max total cache | 100 MB | -| Max pending duration | 30 seconds | -| Flush interval | 100 ms | -| Write strategy | Atomic (temp file + rename) | - -Architecture: -- `put_pending(file_id, content, target_path)` stores bytes in `HashMap` -- Background `flush_worker` processes **FlushCommands** via `mpsc` channel -- Periodic checker force-flushes entries older than 30 seconds -- `get_pending(file_id)` serves reads while data is still in RAM (before flush) - -Port: implements **WriteBehindCachePort** trait. - -Statistics: -```rust -pub struct WriteBehindStatsDto { - pub pending_count: usize, - pub pending_bytes: usize, - pub total_writes: u64, - pub total_bytes_written: u64, - pub cache_hits: u64, - pub avg_flush_time_us: u64, -} -``` - ---- - -## Layer 6: Buffer Pool - -**File**: `src/infrastructure/services/buffer_pool.rs` - -Reusable byte buffer pool to reduce allocation pressure during compression operations. - -| Parameter | Value | -|---|---| -| Buffer size | 64 KB | -| Max buffers | 100 | -| Buffer TTL | 60 seconds | -| Concurrency control | `tokio::sync::Semaphore` | - -Features: -- `get_buffer()` -- borrows a buffer (blocks if pool exhausted) -- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn` -- Expired buffers are cleaned periodically via `start_cleaner()` -- Tracks stats: gets, hits, misses, returns, evictions, waits - ---- - -## Configuration - -All cache-related config in `src/common/config.rs`: - -```rust -pub struct CacheConfig { - pub file_ttl_ms: u64, // default: 60,000 (1 min) - pub directory_ttl_ms: u64, // default: 120,000 (2 min) - pub max_entries: usize, // default: 10,000 -} - -pub struct ResourceConfig { - pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary) - pub chunk_size_bytes: usize, // 1 MB (streaming chunk size) - pub max_in_memory_file_size_mb: u64, // 50 MB -} -``` - -## Download Flow - -``` -Request → ETag check (304?) → Range request (206?) - → file size < 10MB? → Tier 1: LRU cache (RAM) - → file size < 100MB? → Tier 2: MMAP (kernel page cache) - → file size ≥ 100MB → Tier 3: Streaming (chunked) -``` - ---- - -## Range Requests (HTTP 206 Partial Content) - -**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/file_fs_read_repository.rs` - -**Crate**: `http-range-header = "0.4"` for parsing. - -### Request Processing Flow - -``` -Range header present? - ├─ parse_range_header(range_str) - │ ├─ Parse OK → ranges.validate(file_size) - │ │ ├─ Valid → take first range → get_file_range_stream(start, end+1) - │ │ │ ├─ Stream OK → 206 Partial Content - │ │ │ └─ Stream Err → fall through to normal download (200) - │ │ └─ Invalid → 416 Range Not Satisfiable - │ └─ Parse Err → fall through to normal download (200) - └─ No Range header → normal 3-tier download -``` - -### Response Headers (206) - -| Header | Value | -|---|---| -| `Content-Type` | File MIME type | -| `Content-Range` | `bytes {start}-{end}/{total_size}` | -| `Content-Length` | Range length (end - start + 1) | -| `Accept-Ranges` | `bytes` | -| `ETag` | `"{file_id}-{modified_at}"` | -| `Cache-Control` | `private, max-age=3600, must-revalidate` | - -### 416 Range Not Satisfiable - -Returned when `ranges.validate(file_size)` fails: -``` -HTTP/1.1 416 Range Not Satisfiable -Content-Range: bytes */12345 -``` - -### File Seek Implementation - -`get_file_range_stream()` at the repository level: - -```rust -async fn get_file_range_stream( - &self, id: &str, start: u64, end: Option, -) -> Result + Send>, DomainError> -``` - -1. Opens the file with `TokioFile::open()` -2. Seeks to `start` via `fh.seek(SeekFrom::Start(start))` -3. Limits read to `range_length` via `fh.take(range_length)` -4. Wraps in `FramedRead` + `BytesCodec` - -Adaptive chunk size: - -| Range size | Chunk size | -|---|---| -| ≤ 1 MB | 8 KB | -| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) | - -### Tier Interaction - -Range requests **bypass all download tiers** (LRU, MMAP, write-behind). They always use direct file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path. - -### Limitations - -- **Multipart ranges not supported**: only the first range in a multi-range request is served. Additional ranges are ignored. -- **`If-Range` not handled**: no conditional range support. -- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked. - ---- - -## Upload Flow - -``` -Request → file size < 256KB? → Write-behind cache (instant 201, async flush) - → file size < 1MB? → Buffered write (sync) - → file size ≥ 1MB → Streaming write (chunk-by-chunk to temp + rename) -``` - -### Upload Strategy Selection - -**File**: `src/application/services/file_upload_service.rs` - -```rust -pub enum UploadStrategy { - WriteBehind, // < 256 KB — instant response, async disk write - Buffered, // 256 KB – 1 MB — sync write to final path - Streaming, // ≥ 1 MB — chunk-by-chunk write to temp file + rename -} -``` - -| Constant | Value | -|---|---| -| `WRITE_BEHIND_THRESHOLD` | 256 KB | -| `STREAMING_UPLOAD_THRESHOLD` | 1 MB | - -### Handler-Level Buffering - -Both upload handlers (`upload_file` and `upload_file_with_cache`) buffer the **entire multipart body in RAM** as `Vec` before calling the service layer: - -```rust -let mut chunks: Vec = Vec::new(); -while let Some(chunk) = field.chunk().await { - chunks.push(chunk); -} -// All bytes are now in RAM -upload_service.smart_upload(..., chunks, total_size).await -``` - -The "streaming" in `UploadStrategy::Streaming` refers to the **service→repository** path, not the HTTP-body→disk path. By the time `save_file_from_stream()` is called, data is already in memory. - -### Streaming Path (≥ 1 MB): Service → Repository - -`smart_upload()` converts the in-memory `Vec` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`: - -```rust -let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c))); -self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await -``` - -**`save_file_from_stream()` implementation** (`file_fs_write_repository.rs`): - -1. Resolves target path + generates unique name if collision -2. Creates temp file: `{target_path}.tmp.upload` -3. Iterates stream, writing each chunk with `fh.write_all(&chunk)` -4. Calls `fh.flush()` + `fh.sync_all()` for durability -5. Atomic rename: `fs::rename(temp_path, final_path)` -6. Post-write: ID mapping, cache invalidation, metadata update - -### Buffered Path (256 KB - 1 MB) - -Uses `save_file()` -- writes all bytes directly to the **final path** (no temp file). For larger content, writes in chunks of **ResourceConfig.chunk_size_bytes** (1 MB). - -### Write-Behind Path (< 256 KB) - -See **Layer 5** above. Instant `201`, background flush within 30 seconds. - -### Dedup Pre-Check - -Runs for **all upload strategies** before writing. Re-combines all chunks into a single `Vec` for hash computation, which means data is temporarily duplicated in RAM during dedup processing. +# 04 - Caching Architecture + +OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads. + +## Cache Layers Summary + +``` +┌─────────────────────────────────────────────────────┐ +│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints +├─────────────────────────────────────────────────────┤ +│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads +├─────────────────────────────────────────────────────┤ +│ Layer 2: MMAP (memmap2, 10-100MB blobs) │ Downloads +├─────────────────────────────────────────────────────┤ +│ Layer 3: Streaming (FramedRead, ≥100MB blobs) │ Downloads +├─────────────────────────────────────────────────────┤ +│ Layer 4: Buffer Pool (reusable I/O buffers) │ Compression +└─────────────────────────────────────────────────────┘ +``` + +> **Note:** File metadata (name, size, MIME type, folder) is served from PostgreSQL — no separate filesystem metadata cache is needed. + +--- + +## Layer 0: HTTP Cache Middleware + +**File**: `src/interfaces/middleware/cache.rs` + +Generic HTTP caching layer applied to API endpoints. + +| Parameter | Value | +|---|---| +| Max entries | 1,000 | +| Default max-age | 60 seconds | +| Eviction | LRU (oldest 10% when full) | +| Cleanup | Background task every 5 minutes | + +Features: +- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`) +- `Cache-Control` header injection +- Implements Tower `Layer` + `Service` traits for Axum integration +- Per-request key: method + URI + +--- + +## Layer 1: File Content Cache (Download Tier 1) + +**File**: `src/infrastructure/services/file_content_cache.rs` + +In-memory LRU cache for small files, served directly from RAM. + +| Parameter | Value | +|---|---| +| Max file size | 10 MB per file | +| Max total cache size | 512 MB | +| Max entries | 10,000 | +| Structure | `lru::LruCache` | +| Latency | ~0.1ms | + +**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }` + +Methods: +- `should_cache(size)` — checks if file fits in cache +- `get(file_id)` → `Option<(Bytes, String, String)>` — returns (data, etag, content_type) +- `put(file_id, content, etag, content_type)` — inserts with LRU eviction +- `invalidate(file_id)`, `clear()` +- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }` + +Port: implements **ContentCachePort** trait. + +--- + +## Layer 2: MMAP (Download Tier 2) + +**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs` + +Memory-mapped I/O for medium blobs using `memmap2`. + +| Parameter | Value | +|---|---| +| File range | 10 MB - 100 MB | +| Implementation | `memmap2::Mmap` via `spawn_blocking` | +| Latency | ~1-5ms | + +The blob file (`.blobs/{prefix}/{hash}.blob`) is memory-mapped and its contents copied to `Bytes`. Benefits from kernel page cache for frequently accessed blobs. + +--- + +## Layer 3: Streaming (Download Tier 3) + +**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs` + +Chunked streaming for large blobs using tokio-util codecs. + +| Parameter | Value | +|---|---| +| File range | ≥100 MB | +| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) | +| Implementation | `FramedRead` + `BytesCodec` | +| RAM usage | Near zero (one chunk at a time) | + +--- + +## Layer 4: Buffer Pool + +**File**: `src/infrastructure/services/buffer_pool.rs` + +Reusable byte buffer pool to reduce allocation pressure during compression operations. + +| Parameter | Value | +|---|---| +| Buffer size | 64 KB | +| Max buffers | 100 | +| Buffer TTL | 60 seconds | +| Concurrency control | `tokio::sync::Semaphore` | + +Features: +- `get_buffer()` — borrows a buffer (blocks if pool exhausted) +- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn` +- Expired buffers are cleaned periodically via `start_cleaner()` +- Tracks stats: gets, hits, misses, returns, evictions, waits + +--- + +## Configuration + +All cache-related config in `src/common/config.rs`: + +```rust +pub struct ResourceConfig { + pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary) + pub chunk_size_bytes: usize, // 1 MB (streaming chunk size) + pub max_in_memory_file_size_mb: u64, // 50 MB +} +``` + +## Download Flow + +``` +Request → ETag check (304?) → Range request (206?) + → file size < 10MB? → Tier 1: LRU cache (RAM) + → file size < 100MB? → Tier 2: MMAP (kernel page cache on blob file) + → file size ≥ 100MB → Tier 3: Streaming (chunked from blob file) +``` + +In all tiers, metadata (file name, size, MIME type) comes from a PostgreSQL `SELECT` on `storage.files`. Content is read from the DedupService blob at `.blobs/{prefix}/{hash}.blob`. + +--- + +## Range Requests (HTTP 206 Partial Content) + +**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/pg/file_blob_read_repository.rs` + +**Crate**: `http-range-header = "0.4"` for parsing. + +### Request Processing Flow + +``` +Range header present? + ├─ parse_range_header(range_str) + │ ├─ Parse OK → ranges.validate(file_size) + │ │ ├─ Valid → take first range → get_file_range_stream(start, end+1) + │ │ │ ├─ Stream OK → 206 Partial Content + │ │ │ └─ Stream Err → fall through to normal download (200) + │ │ └─ Invalid → 416 Range Not Satisfiable + │ └─ Parse Err → fall through to normal download (200) + └─ No Range header → normal 3-tier download +``` + +### Response Headers (206) + +| Header | Value | +|---|---| +| `Content-Type` | File MIME type | +| `Content-Range` | `bytes {start}-{end}/{total_size}` | +| `Content-Length` | Range length (end - start + 1) | +| `Accept-Ranges` | `bytes` | +| `ETag` | `"{file_id}-{modified_at}"` | +| `Cache-Control` | `private, max-age=3600, must-revalidate` | + +### 416 Range Not Satisfiable + +Returned when `ranges.validate(file_size)` fails: +``` +HTTP/1.1 416 Range Not Satisfiable +Content-Range: bytes */12345 +``` + +### Blob File Seek Implementation + +`get_file_range_stream()` at the repository level: + +1. Resolves blob path from `blob_hash` via DedupService +2. Opens the blob file with `TokioFile::open()` +3. Seeks to `start` via `fh.seek(SeekFrom::Start(start))` +4. Limits read to `range_length` via `fh.take(range_length)` +5. Wraps in `FramedRead` + `BytesCodec` + +Adaptive chunk size: + +| Range size | Chunk size | +|---|---| +| ≤ 1 MB | 8 KB | +| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) | + +### Tier Interaction + +Range requests **bypass all download tiers** (LRU, MMAP). They always use direct blob file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path. + +### Limitations + +- **Multipart ranges not supported**: only the first range in a multi-range request is served. +- **`If-Range` not handled**: no conditional range support. +- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked. + +--- + +## Upload Flow + +``` +Request → file size < 1MB? → Buffered write (sync to blob store) + → file size ≥ 1MB → Streaming write (chunk-by-chunk to blob store) +``` + +### Upload Strategy Selection + +**File**: `src/application/services/file_upload_service.rs` + +```rust +pub enum UploadStrategy { + Buffered, // < 1 MB — collect bytes, write to blob store + Streaming, // ≥ 1 MB — chunk-by-chunk write via save_file_from_stream +} +``` + +| Constant | Value | +|---|---| +| `STREAMING_UPLOAD_THRESHOLD` | 1 MB | + +### Handler-Level Buffering + +Upload handlers buffer the multipart body in RAM as `Vec` before calling the service layer: + +```rust +let mut chunks: Vec = Vec::new(); +while let Some(chunk) = field.chunk().await { + chunks.push(chunk); +} +upload_service.smart_upload(..., chunks, total_size).await +``` + +### Buffered Path (< 1 MB) + +Uses `save_file()` — all bytes are passed to `FileBlobWriteRepository`, which calls `DedupService.store_bytes()` to compute hash and store the blob, then INSERTs metadata into `storage.files`. + +### Streaming Path (≥ 1 MB) + +`smart_upload()` converts the in-memory `Vec` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`: + +```rust +let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c))); +self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await +``` + +`FileBlobWriteRepository.save_file_from_stream()` collects the stream, stores via DedupService, and INSERTs metadata. + +### Dedup Integration + +Deduplication is handled at the **repository layer** (not the service layer) for all upload strategies. `FileBlobWriteRepository` always calls `DedupService.store_bytes()` which: + +1. Computes SHA-256 hash of content +2. Checks if blob already exists (dedup hit → increment ref count, skip write) +3. If new → atomic write to `.blobs/{prefix}/{hash}.blob` +4. Returns the hash for storage in `storage.files.blob_hash` diff --git a/doc/deduplication.md b/doc/deduplication.md index 2025a65f..1e8499a9 100644 --- a/doc/deduplication.md +++ b/doc/deduplication.md @@ -285,9 +285,12 @@ pub struct CoreServices { // ... } -// Injected into application services: -FileUploadService::new_full(... core.dedup_service.clone()) -FileManagementService::new_full(... core.dedup_service.clone()) +// Injected into blob repositories (which handle dedup internally): +FileBlobReadRepository::new(pool, core.dedup_service.clone(), folder_repo) +FileBlobWriteRepository::new(pool, core.dedup_service.clone(), folder_repo) + +// Also injected into FileManagementService for ref cleanup on delete: +FileManagementService::new_full(write, read, trash, core.dedup_service.clone()) ``` ## Persistence diff --git a/doc/deployment.md b/doc/deployment.md index 327beb85..1e687b32 100644 --- a/doc/deployment.md +++ b/doc/deployment.md @@ -174,17 +174,17 @@ Hardcoded defaults in `src/common/config.rs`: | Feature | Requires DB | Requires Auth | Feature Flag | |---|---|---|---| -| File storage | No | No | Always on | +| File storage | Yes | No | Always on | | Authentication | Yes | -- | `OXICLOUD_ENABLE_AUTH` | | OIDC / SSO | Yes | Yes | `OXICLOUD_OIDC_ENABLED` | | File sharing | Yes | Yes | `OXICLOUD_ENABLE_FILE_SHARING` | -| Trash | No | No | `OXICLOUD_ENABLE_TRASH` | -| Search | No | No | `OXICLOUD_ENABLE_SEARCH` | +| Trash | Yes | No | `OXICLOUD_ENABLE_TRASH` | +| Search | Yes | No | `OXICLOUD_ENABLE_SEARCH` | | Favorites | Yes | Yes | Always on (when DB available) | | Recent items | Yes | Yes | Always on (when DB available) | | Storage quotas | Yes | Yes | `OXICLOUD_ENABLE_USER_STORAGE_QUOTAS` | | Admin panel | Yes | Yes | Always on (when auth enabled) | -| WebDAV | No | Optional | Always on | +| WebDAV | Yes | Optional | Always on | | CalDAV | Yes | Yes | Always on (when DB available) | | CardDAV | Yes | Yes | Always on (when DB available) | | Deduplication | No | No | Always on | diff --git a/doc/file-system-safety.md b/doc/file-system-safety.md index 5fe9cf60..d78e663f 100644 --- a/doc/file-system-safety.md +++ b/doc/file-system-safety.md @@ -1,156 +1,125 @@ -# 03 - File System Safety - -OxiCloud ensures data integrity and durability during file operations through atomic writes, fsync, and directory synchronization. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss. - ---- - -## The Problem: Buffered I/O - -Standard filesystem operations use buffered I/O by default: - -```rust -// This operation may not immediately persist to disk -fs::write(path, content) -``` - -When an application writes data, the OS typically: - -1. Accepts the write into memory buffers -2. Acknowledges completion to the application -3. Schedules the actual disk write for later - -A crash during that window means data loss -- the data exists only in memory buffers that haven't been flushed. - ---- - -## OxiCloud's Approach - -All safety mechanisms live in the **FileSystemUtils** service. - -### Atomic Write Pattern - -Files are written using write-then-rename: - -```rust -/// Writes data to a file with fsync to ensure durability -/// Uses a safe atomic write pattern: write to temp file, fsync, rename -pub async fn atomic_write>(path: P, contents: &[u8]) -> Result<(), IoError> -``` - -Steps: -1. Write to a temporary file in the same directory -2. Call `fsync` to ensure data is on disk -3. Atomically rename the temp file to the target file -4. Sync the parent directory to ensure the rename is persisted - -### Directory Synchronization - -```rust -/// Creates directories with fsync -pub async fn create_dir_with_sync>(path: P) -> Result<(), IoError> -``` - -Directories are created, their entries persisted to disk, and parent directories synchronized too. - -### Rename and Delete Operations - -```rust -/// Renames a file or directory with proper syncing -pub async fn rename_with_sync, Q: AsRef>(from: P, to: Q) -> Result<(), IoError> - -/// Removes a file with directory syncing -pub async fn remove_file_with_sync>(path: P) -> Result<(), IoError> -``` - -Both complete the operation itself, then update and sync the parent directory entry. - ---- - -## Implementation Details - -### fsync on Files - -```rust -// Write file content -file.write_all(contents).await?; - -// Ensure data is synced to disk -file.flush().await?; -file.sync_all().await?; -``` - -`sync_all()` instructs the OS to flush data and metadata to the physical storage device. - -### fsync on Directories - -```rust -// Sync a directory to ensure its contents (entries) are durable -async fn sync_directory>(path: P) -> Result<(), IoError> { - let dir_file = OpenOptions::new().read(true).open(path).await?; - dir_file.sync_all().await -} -``` - -Required after any operation that modifies directory entries (create, rename, delete). - ---- - -## Usage in the Codebase - -### File Write Repository - -```rust -// Write the file to disk using atomic write with fsync -tokio::time::timeout( - self.config.timeouts.file_write_timeout(), - FileSystemUtils::atomic_write(&abs_path, &content) -).await -``` - -### File Move Operations - -```rust -// Move the file physically with fsync -time::timeout( - self.config.timeouts.file_timeout(), - FileSystemUtils::rename_with_sync(&old_abs_path, &new_abs_path) -).await -``` - -### Directory Creation - -```rust -// Ensure the parent directory exists with proper syncing -self.ensure_parent_directory(&abs_path).await?; - -// Implementation uses FileSystemUtils -async fn ensure_parent_directory(&self, abs_path: &PathBuf) -> FileRepositoryResult<()> { - if let Some(parent) = abs_path.parent() { - time::timeout( - self.config.timeouts.dir_timeout(), - FileSystemUtils::create_dir_with_sync(parent) - ).await - } -} -``` - ---- - -## Benefits - -1. **Data durability** -- critical data is synced to persistent storage -2. **Crash resilience** -- recovery from unexpected failures without data loss -3. **Consistency** -- file operations maintain a consistent filesystem state -4. **Atomic operations** -- file writes appear as all-or-nothing - ---- - -## Performance Considerations - -Syncing to disk costs more than buffered writes. OxiCloud mitigates this by: - -1. Applying these measures only to critical operations -2. Using timeouts to prevent indefinite blocking -3. Implementing parallel processing for large files - -The tradeoff favors safety for critical data while maintaining good performance for most operations. +# 03 - Storage Safety + +OxiCloud ensures data integrity and durability through a combination of PostgreSQL transactional guarantees and atomic blob writes. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss. + +--- + +## Storage Model + +OxiCloud uses a **100% blob storage model**: + +- **Metadata** (file names, folder hierarchy, sizes, MIME types, trash status) lives in **PostgreSQL** — protected by ACID transactions. +- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob` — protected by atomic writes and fsync. + +--- + +## PostgreSQL Safety (Metadata) + +All file and folder metadata operations use PostgreSQL transactions: + +- **Single-row operations** (INSERT, UPDATE, DELETE) are inherently atomic. +- **Multi-step operations** (e.g., move file: UPDATE folder_id + UPDATE path) use explicit transactions via `sqlx`. +- **Foreign key constraints** prevent orphaned records (e.g., files referencing non-existent folders). +- **Unique constraints** prevent duplicate names within the same parent folder. +- **Soft-delete** for trash (`is_trashed = TRUE`) preserves data until explicit permanent deletion. + +The `storage.trash_items` VIEW provides a unified read interface over trashed files and folders without duplicating data. + +--- + +## Blob Storage Safety (Content) + +### DedupService Atomic Writes + +**File:** `src/infrastructure/services/dedup_service.rs` + +When storing file content, DedupService uses the following pattern: + +1. **Hash computation** — SHA-256 hash of content determines the blob path +2. **Deduplication check** — if a blob with the same hash exists, only increment the reference counter (no write needed) +3. **Atomic write** — if new content: + - Write to a temporary file (`.blob.tmp`) + - Call `fsync` to ensure data reaches persistent storage + - Atomically rename temp file to final path (`.blobs/{prefix}/{hash}.blob`) +4. **Reference counting** — track how many files reference each blob + +This ensures that a blob either fully exists or doesn't — no partial writes. + +### FileSystemUtils + +**File:** `src/infrastructure/services/file_system_utils.rs` + +Low-level utilities used internally by DedupService and other infrastructure services: + +```rust +/// Atomic write: temp file → fsync → rename +pub async fn atomic_write>(path: P, contents: &[u8]) -> Result<(), IoError> + +/// Directory creation with fsync +pub async fn create_dir_with_sync>(path: P) -> Result<(), IoError> + +/// Rename with directory sync +pub async fn rename_with_sync(from: P, to: Q) -> Result<(), IoError> + +/// Delete with directory sync +pub async fn remove_file_with_sync>(path: P) -> Result<(), IoError> +``` + +### fsync Guarantees + +- `sync_all()` on written files ensures data and metadata reach the physical storage device +- Directory entries are synced after create/rename/delete operations +- Prevents data loss during crashes or power failures between OS buffer flush and disk write + +--- + +## Transaction Flow: File Upload + +``` +1. DedupService.store_bytes(content) + → Compute SHA-256 hash + → Check if blob exists (dedup hit → increment ref, return hash) + → Write to .blobs/{prefix}/{hash}.blob.tmp + → fsync + rename → .blobs/{prefix}/{hash}.blob + +2. FileBlobWriteRepository.save_file() + → BEGIN TRANSACTION + → INSERT INTO storage.files (name, folder_id, blob_hash, size, ...) + → COMMIT +``` + +If step 1 fails, no metadata is written. If step 2 fails, the blob exists but is unreferenced (cleaned up by garbage collection). Data is never in an inconsistent state. + +## Transaction Flow: File Deletion + +``` +1. FileBlobWriteRepository.delete_file_permanently() + → BEGIN TRANSACTION + → DELETE FROM storage.files WHERE id = $1 (captures blob_hash first) + → COMMIT + +2. DedupService.decrement_ref(blob_hash) + → Decrement reference counter + → If counter reaches 0, delete the blob file +``` + +If step 2 fails, an unreferenced blob may remain on disk (occupies space but is not a correctness issue). Future garbage collection can clean these up. + +--- + +## Benefits + +1. **ACID transactions** — metadata operations are atomic, consistent, isolated, and durable +2. **Content-addressable storage** — identical content is stored once, referenced by hash +3. **Crash resilience** — atomic blob writes + PostgreSQL WAL ensure recovery +4. **No partial writes** — temp file + rename pattern guarantees all-or-nothing +5. **Referential integrity** — foreign keys prevent orphaned metadata + +--- + +## Performance Considerations + +- PostgreSQL connection pooling (`sqlx::PgPool`) amortizes connection overhead +- Dedup hash computation is CPU-bound but avoids unnecessary disk writes for duplicate content +- Blob fsync adds latency vs. buffered writes, but ensures durability for critical user data +- Content cache (in-memory LRU) serves repeat reads without disk or DB access diff --git a/doc/important-delta-sync-implementation.md b/doc/important-delta-sync-implementation.md index 0b5de672..374e0016 100644 --- a/doc/important-delta-sync-implementation.md +++ b/doc/important-delta-sync-implementation.md @@ -462,7 +462,7 @@ src/ │ └── delta_sync_handler.rs # NUEVO: Endpoints API │ └── common/ - └── di.rs # Añadir: delta_sync_service a CoreServices + └── di.rs # Añadir: delta_sync_service a AppState ``` ### Main service (delta_sync_service.rs) @@ -1133,7 +1133,7 @@ thiserror = "1.0" # Para errores tipados (probablemente ya existe) - [ ] Implement **generate_delta()** - [ ] Implement **apply_delta()** - [ ] Create handler and API endpoints -- [ ] Integrate into DI (**CoreServices**) +- [ ] Integrate into DI (**AppState**) - [ ] Add routes in `routes.rs` - [ ] Integrate with upload (automatic indexing) - [ ] Integrate with delete (signature cleanup) diff --git a/doc/internal-architecture.md b/doc/internal-architecture.md index 55e26d96..a89bdcf3 100644 --- a/doc/internal-architecture.md +++ b/doc/internal-architecture.md @@ -1,477 +1,474 @@ -# 01 - Internal Architecture - -OxiCloud follows a **hexagonal (ports & adapters) architecture** organized in four layers: - -``` -Domain → Application → Infrastructure → Interfaces -``` - -All cross-layer dependencies point inward via trait-based ports. The DI container (**AppServiceFactory**) wires concrete implementations at startup. - ---- - -## Dependency Injection Container - -### AppServiceFactory - -**File:** `src/common/di.rs` - -```rust -pub struct AppServiceFactory { - storage_path: PathBuf, - locales_path: PathBuf, - config: AppConfig, -} -``` - -Initialization order in `build_app_state()`: - -1. **Core services** -- path, caches, ID mapping, thumbnail, write-behind, chunked upload, transcode, dedup, compression -2. **Repository services** -- folder repo (stub mediator first), then **FileSystemStorageMediator** (real), file repos, metadata cache, buffer pool -3. **Trash service** (if **enable_trash** enabled) -4. **Application services** -- folder, file upload/retrieval/management, search, i18n -5. **Share service** (if **enable_file_sharing** enabled) -6. **DB-dependent services** -- favorites, recent, storage usage, auth (via **auth_factory**) -7. **Preload** translations + metadata cache -8. **ZIP service** (needs file retrieval + folder service, wired last) -9. **Assemble AppState** + admin settings + CalDAV/CardDAV - -### AppState (Global State) - -```rust -pub struct AppState { - pub core: CoreServices, - pub repositories: RepositoryServices, - pub applications: ApplicationServices, - pub db_pool: Option>, - pub auth_service: Option, - pub admin_settings_service: Option>, - pub trash_service: Option>, - pub share_service: Option>, - pub favorites_service: Option>, - pub recent_service: Option>, - pub storage_usage_service: Option>, - pub calendar_service: Option>, - pub contact_service: Option>, - pub calendar_use_case: Option>, - pub addressbook_use_case: Option>, - pub contact_use_case: Option>, -} -``` - -Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `with_trash_service()` → ... → `for_routing()`. The `Default` impl uses stubs from `crate::common::stubs`. - -### Service Groups - -```rust -pub struct CoreServices { - pub path_service: Arc, - pub file_content_cache: Arc, - pub id_mapping_service: Arc, // folder IDs - pub file_id_mapping_service: Arc, // file IDs (concrete) - pub id_mapping_optimizer: Arc, - pub thumbnail_service: Arc, - pub write_behind_cache: Arc, - pub chunked_upload_service: Arc, - pub image_transcode_service: Arc, - pub dedup_service: Arc, - pub compression_service: Arc, - pub zip_service: Arc, - pub config: AppConfig, -} - -pub struct RepositoryServices { - pub folder_repository: Arc, - pub file_read_repository: Arc, - pub file_write_repository: Arc, - pub i18n_repository: Arc, - pub storage_mediator: Arc, - pub metadata_cache: Arc, - pub trash_repository: Option>, -} - -pub struct ApplicationServices { - pub folder_service_concrete: Arc, - pub folder_service: Arc, - pub file_upload_service: Arc, - pub file_retrieval_service: Arc, - pub file_management_service: Arc, - pub file_use_case_factory: Arc, - pub i18n_service: Arc, - pub trash_service: Option>, - pub search_service: Option>, - pub share_service: Option>, - pub favorites_service: Option>, - pub recent_service: Option>, -} - -pub struct AuthServices { - pub token_service: Arc, - pub auth_application_service: Arc, -} -``` - ---- - -## ID Mapping System - -Maps bidirectionally between **filesystem StoragePaths** and **UUID identifiers**. Two separate instances exist: one for folders (`folder_ids.json`), one for files (`file_ids.json`). - -### StoragePath (Domain Value Object) - -**File:** `src/domain/services/path_service.rs` - -```rust -#[derive(Debug, Clone, PartialEq, Eq, Default)] -pub struct StoragePath { - segments: Vec, // e.g., ["Mi Carpeta - admin", "file.txt"] -} -``` - -| Method | Description | -|---|---| -| `root()` | Empty path (storage root) | -| `from_string(path)` | Parse from `/`-delimited string | -| `join(segment)` | Append a segment | -| `file_name()` | Last segment | -| `parent()` | All segments except last | -| `to_string()` | Join segments with `/` | - -### IdMappingPort (Application Port) - -**File:** `src/application/ports/outbound.rs` - -```rust -#[async_trait] -pub trait IdMappingPort: Send + Sync + 'static { - async fn get_or_create_id(&self, path: &StoragePath) -> Result; - async fn get_path_by_id(&self, id: &str) -> Result; - async fn update_path(&self, id: &str, new_path: &StoragePath) -> Result<(), DomainError>; - async fn remove_id(&self, id: &str) -> Result<(), DomainError>; - async fn save_changes(&self) -> Result<(), DomainError>; - // Default impls for PathBuf variants: - async fn get_file_path(&self, file_id: &str) -> Result; - async fn update_file_path(&self, file_id: &str, new_path: &PathBuf) -> Result<(), DomainError>; -} -``` - -### IdMappingService (Base Implementation) - -**File:** `src/infrastructure/services/id_mapping_service.rs` - -```rust -pub struct IdMappingService { - map_path: PathBuf, // e.g., storage/file_ids.json - id_map: RwLock, - save_mutex: Mutex<()>, - timeouts: TimeoutConfig, - pending_save: RwLock, -} - -struct IdMap { - path_to_id: HashMap, - id_to_path: HashMap, - version: u32, -} -``` - -Operations: -- `get_or_create_id(path)` -- Read-lock first (cache hit). Write-lock on miss, generates `Uuid::new_v4()`. -- `save_pending_changes()` -- Atomic write: serialize → write `.tmp` file → rename over original (with retry). -- `new(map_path)` -- Loads from JSON, rebuilds inverse map if inconsistent. - -Persistence format (`storage/file_ids.json`): -```json -{ - "path_to_id": { "/Mi Carpeta - admin/doc.pdf": "a1b2c3d4-..." }, - "id_to_path": { "a1b2c3d4-...": "/Mi Carpeta - admin/doc.pdf" }, - "version": 42 -} -``` - -### IdMappingOptimizer (Cache Layer) - -**File:** `src/infrastructure/services/id_mapping_optimizer.rs` - -Wraps **IdMappingService** with an in-memory TTL cache: - -| Parameter | Value | -|---|---| -| Max cache entries | 10,000 | -| TTL | 300 s (5 min) | -| Cleanup interval | 150 s (2.5 min) | -| Batch threshold | ≥ 20 queued items | -| Max concurrent batches | 2 (semaphore) | - -```rust -pub struct IdMappingOptimizer { - base_service: Arc, - path_to_id_cache: RwLock>, - id_to_path_cache: RwLock>, - stats: RwLock, - batch_limiter: Semaphore, - pending_batch: Mutex, -} -``` - -Lookup flow: check cache → if miss, queue request → trigger batch if ≥ 20 pending → fallback to **base_service** → update cache. - -On `update_path` / `remove_id`, cache entries are invalidated first, then delegated. - -Used only for folder ID mapping. File ID mapping uses the base **IdMappingService** directly. - ---- - -## Path Service - -**File:** `src/infrastructure/services/path_service.rs` - -```rust -pub struct PathService { - root_path: PathBuf, // e.g., ./storage -} -``` - -### Path Resolution - -| Method | Description | -|---|---| -| `resolve_path(storage_path)` | Appends **StoragePath** segments to **root_path** → absolute `PathBuf` | -| `to_storage_path(physical_path)` | Strips **root_path** prefix → **StoragePath** (returns `None` if outside root) | -| `create_file_path(folder, name)` | Combines folder path + filename | -| `is_direct_child(parent, child)` | Check parent-child relationship | -| `is_in_root(path)` | Verify path is within storage root | - -### Path Validation - -`validate_path(path)` rejects: -- Empty path segments -- Segments containing dangerous characters: `\`, `:`, `*`, `?`, `"`, `<`, `>`, `|` -- Segments starting with `.` (exception: `.well-known` for WebDAV/CalDAV/CardDAV) - -### Trait Implementations - -- **StoragePort** -- `resolve_path()`, `ensure_directory()` (validates first, then `fs::create_dir_all`), `file_exists()`, `directory_exists()` -- **StorageMediator** -- Simplified stub variant. Folder lookup methods return `NotFound`. - ---- - -## Storage Mediator - -Bridges folder IDs to filesystem paths by combining folder repository, path service, and ID mapping. - -### StorageMediator Trait - -**File:** `src/application/services/storage_mediator.rs` - -```rust -#[async_trait] -pub trait StorageMediator: Send + Sync + 'static { - async fn get_folder_path(&self, folder_id: &str) -> StorageMediatorResult; - async fn get_folder_storage_path(&self, folder_id: &str) -> StorageMediatorResult; - async fn get_folder(&self, folder_id: &str) -> StorageMediatorResult; - async fn file_exists_at_path(&self, path: &Path) -> StorageMediatorResult; - async fn file_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult; - async fn folder_exists_at_path(&self, path: &Path) -> StorageMediatorResult; - async fn folder_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult; - fn resolve_path(&self, relative_path: &Path) -> PathBuf; - fn resolve_storage_path(&self, storage_path: &StoragePath) -> PathBuf; - async fn ensure_directory(&self, path: &Path) -> StorageMediatorResult<()>; - async fn ensure_storage_directory(&self, storage_path: &StoragePath) -> StorageMediatorResult<()>; -} -``` - -### FileSystemStorageMediator - -```rust -pub struct FileSystemStorageMediator { - pub folder_storage_port: Arc, - pub path_service: Arc, - pub id_mapping: Arc, -} -``` - -Folder ID → filesystem path resolution: - -``` -folder_id ──► FolderStoragePort.get_folder(id) - ──► Folder entity - ──► IdMappingPort.get_path_by_id(folder.id()) - ──► StoragePath - ──► StoragePort.resolve_path(storage_path) - ──► PathBuf (absolute) -``` - -A **StubStorageMediator** also exists (returns `/tmp` paths) for DI bootstrap before the real mediator is available. - ---- - -## Session Management - -### Session Entity - -**File:** `src/domain/entities/session.rs` - -```rust -pub struct Session { - id: String, // UUID v4 - user_id: String, - refresh_token: String, - expires_at: DateTime, - ip_address: Option, - user_agent: Option, - created_at: DateTime, - revoked: bool, -} -``` - -Constructors: -- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` -- generates UUID, panics if **user_id** or **refresh_token** empty -- `Session::from_raw(...)` -- for DB reconstruction - -### SessionRepository (Domain Port) - -**File:** `src/domain/repositories/session_repository.rs` - -```rust -#[async_trait] -pub trait SessionRepository: Send + Sync + 'static { - async fn create_session(&self, session: Session) -> SessionRepositoryResult; - async fn get_session_by_id(&self, id: &str) -> SessionRepositoryResult; - async fn get_session_by_refresh_token(&self, token: &str) -> SessionRepositoryResult; - async fn get_sessions_by_user_id(&self, user_id: &str) -> SessionRepositoryResult>; - async fn revoke_session(&self, session_id: &str) -> SessionRepositoryResult<()>; - async fn revoke_all_user_sessions(&self, user_id: &str) -> SessionRepositoryResult; - async fn delete_expired_sessions(&self) -> SessionRepositoryResult; -} -``` - -### SessionStoragePort (Application Port) - -**File:** `src/application/ports/auth_ports.rs` - -```rust -#[async_trait] -pub trait SessionStoragePort: Send + Sync + 'static { - async fn create_session(&self, session: Session) -> Result; - async fn get_session_by_refresh_token(&self, token: &str) -> Result; - async fn revoke_session(&self, session_id: &str) -> Result<(), DomainError>; - async fn revoke_all_user_sessions(&self, user_id: &str) -> Result; -} -``` - -### SessionPgRepository (Infrastructure) - -**File:** `src/infrastructure/repositories/pg/session_pg_repository.rs` - -```rust -pub struct SessionPgRepository { - pool: Arc, -} -``` - -Implements both **SessionRepository** and **SessionStoragePort**. Uses `with_transaction()` helper for write operations. `create_session` also updates `auth.users.last_login_at` within the same transaction. - -### Database Schema - -```sql -CREATE TABLE IF NOT EXISTS auth.sessions ( - id VARCHAR(36) PRIMARY KEY, - user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, - refresh_token TEXT NOT NULL UNIQUE, - expires_at TIMESTAMP WITH TIME ZONE NOT NULL, - ip_address TEXT, - user_agent TEXT, - created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, - revoked BOOLEAN NOT NULL DEFAULT FALSE -); - --- Indexes -CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id); -CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token); -CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at); -CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked) - WHERE NOT revoked AND is_session_active(expires_at); -``` - -### Auth Service - -**File:** `src/application/services/auth_application_service.rs` - -**AuthApplicationService** orchestrates authentication using: -- **UserStoragePort** -- user CRUD -- **SessionStoragePort** -- session lifecycle -- **PasswordHasherPort** -- Argon2id hashing -- **TokenServicePort** -- JWT generation/validation -- `RwLock` -- hot-reloadable OIDC configuration -- `Mutex>` -- in-flight OIDC login states - -Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**. - ---- - -## File Use Case Factory - -**File:** `src/application/services/file_use_case_factory.rs` - -```rust -pub trait FileUseCaseFactory: Send + Sync + 'static { - fn create_file_upload_use_case(&self) -> Arc; - fn create_file_retrieval_use_case(&self) -> Arc; - fn create_file_management_use_case(&self) -> Arc; -} -``` - -**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**. The main DI-wired services use `*_full()` constructors that inject write-behind cache, dedup, content cache, and transcode ports for full optimization. - -### File Operation Port Hierarchy - -| Port | Key Methods | -|---|---| -| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `WriteBehind` <256KB, `Buffered` 256KB-1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` | -| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (write-behind → content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` | -| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) | - ---- - -## Architecture Diagram - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Interfaces Layer │ -│ Axum Router → API Routes + Middleware (Auth, Compress) │ -└─────────────────────┬───────────────────────────────────────┘ - │ Arc -┌─────────────────────▼───────────────────────────────────────┐ -│ Application Layer │ -│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ -│ │ FileUpload │ │ FolderService│ │ AuthApplication │ │ -│ │ FileRetrieval│ │ SearchService│ │ AdminSettings │ │ -│ │ FileMgmt │ │ I18nService │ │ TrashService │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────────┬──────────┘ │ -│ │ Ports (traits) │ │ │ -└─────────┼────────────────┼─────────────────────┼────────────┘ - │ │ │ -┌─────────▼────────────────▼─────────────────────▼────────────┐ -│ Infrastructure Layer │ -│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ -│ │ FileFsRead/ │ │ IdMapping │ │ SessionPg │ │ -│ │ FileFsWrite │ │ + Optimizer │ │ UserPg │ │ -│ │ FolderFs │ │ PathService │ │ JwtTokenService │ │ -│ │ TrashFs │ │ StorageMed. │ │ Argon2Hasher │ │ -│ └────────────────┘ └──────────────┘ └──────────────────┘ │ -│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ -│ │ ContentCache │ │ Thumbnail │ │ WriteBehind │ │ -│ │ MetadataCache │ │ Transcode │ │ BufferPool │ │ -│ │ BufferPool │ │ Dedup │ │ Compression │ │ -│ └────────────────┘ └──────────────┘ └──────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ -┌─────────────────────▼───────────────────────────────────────┐ -│ Domain Layer │ -│ Entities: File, Folder, Session, User, Calendar, Contact │ -│ Value Objects: StoragePath │ -│ Repository Traits: SessionRepository, ... │ -│ Domain Errors │ -└─────────────────────────────────────────────────────────────┘ -``` +# 01 - Internal Architecture + +OxiCloud follows a **hexagonal (ports & adapters) architecture** organized in four layers: + +``` +Domain → Application → Infrastructure → Interfaces +``` + +All cross-layer dependencies point inward via trait-based ports. The DI container (**AppServiceFactory**) wires concrete implementations at startup. + +--- + +## Storage Model: 100% Blob Storage + +OxiCloud uses a **100% blob storage model** where: + +- **File metadata** (name, folder, size, user, timestamps, trash status) is stored in **PostgreSQL** (`storage.files` table). +- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob`. +- **Folder structure** is purely virtual — represented as rows in `storage.folders` (no filesystem directories per user). +- **Trash** is a soft-delete flag (`is_trashed`, `trashed_at`) on files and folders, exposed via `storage.trash_items` VIEW. + +There are no filesystem-based ID mappings, no `folder_ids.json`/`file_ids.json`, and no storage mediator. + +--- + +## Dependency Injection Container + +### AppServiceFactory + +**File:** `src/common/di.rs` + +```rust +pub struct AppServiceFactory { + storage_path: PathBuf, + locales_path: PathBuf, + config: AppConfig, +} +``` + +Initialization order in `build_app_state()`: + +1. **Core services** — path, content cache, thumbnail, chunked upload, transcode, dedup, compression +2. **Repository services** — `FolderDbRepository`, `FileBlobReadRepository`, `FileBlobWriteRepository`, `TrashDbRepository` (all PgPool-backed) +3. **Trash service** (if **enable_trash** enabled) +4. **Application services** — folder, file upload/retrieval/management, search, i18n +5. **Share service** (if **enable_file_sharing** enabled) +6. **DB-dependent services** — favorites, recent, storage usage, auth (via **auth_factory**) +7. **Preload** translations +8. **ZIP service** (needs file retrieval + folder service, wired last) +9. **Assemble AppState** + admin settings + CalDAV/CardDAV + +### AppState (Global State) + +```rust +pub struct AppState { + pub core: CoreServices, + pub repositories: RepositoryServices, + pub applications: ApplicationServices, + pub db_pool: Option>, + pub auth_service: Option, + pub admin_settings_service: Option>, + pub trash_service: Option>, + pub share_service: Option>, + pub favorites_service: Option>, + pub recent_service: Option>, + pub storage_usage_service: Option>, + pub calendar_service: Option>, + pub contact_service: Option>, + pub calendar_use_case: Option>, + pub addressbook_use_case: Option>, + pub contact_use_case: Option>, +} +``` + +Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `with_trash_service()` → ... → `for_routing()`. The `Default` impl uses stubs from `crate::common::stubs`. + +### Service Groups + +```rust +pub struct CoreServices { + pub path_service: Arc, + pub file_content_cache: Arc, + pub thumbnail_service: Arc, + pub chunked_upload_service: Arc, + pub image_transcode_service: Arc, + pub dedup_service: Arc, + pub compression_service: Arc, + pub zip_service: Arc, + pub config: AppConfig, +} + +pub struct RepositoryServices { + pub folder_repository: Arc, + pub folder_repo_concrete: Arc, + pub file_read_repository: Arc, + pub file_write_repository: Arc, + pub i18n_repository: Arc, + pub trash_repository: Option>, +} + +pub struct ApplicationServices { + pub folder_service_concrete: Arc, + pub folder_service: Arc, + pub file_upload_service: Arc, + pub file_retrieval_service: Arc, + pub file_management_service: Arc, + pub file_use_case_factory: Arc, + pub i18n_service: Arc, + pub trash_service: Option>, + pub search_service: Option>, + pub share_service: Option>, + pub favorites_service: Option>, + pub recent_service: Option>, +} + +pub struct AuthServices { + pub token_service: Arc, + pub auth_application_service: Arc, +} +``` + +--- + +## Database Schema (Storage) + +All file and folder metadata lives in the `storage` PostgreSQL schema: + +```sql +CREATE SCHEMA IF NOT EXISTS storage; + +CREATE TABLE storage.folders ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name TEXT NOT NULL, + parent_id UUID REFERENCES storage.folders(id) ON DELETE CASCADE, + user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id), + is_trashed BOOLEAN NOT NULL DEFAULT FALSE, + trashed_at TIMESTAMPTZ, + original_parent_id UUID, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE TABLE storage.files ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + name TEXT NOT NULL, + folder_id UUID NOT NULL REFERENCES storage.folders(id) ON DELETE CASCADE, + user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id), + blob_hash TEXT NOT NULL, + size BIGINT NOT NULL DEFAULT 0, + mime_type TEXT NOT NULL DEFAULT 'application/octet-stream', + is_trashed BOOLEAN NOT NULL DEFAULT FALSE, + trashed_at TIMESTAMPTZ, + original_folder_id UUID, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE OR REPLACE VIEW storage.trash_items AS + SELECT id, name, 'file' AS item_type, folder_id AS parent_id, + user_id, size, mime_type, trashed_at, created_at + FROM storage.files WHERE is_trashed = TRUE + UNION ALL + SELECT id, name, 'folder' AS item_type, parent_id, + user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at + FROM storage.folders WHERE is_trashed = TRUE; +``` + +--- + +## Repository Layer (Infrastructure) + +All repositories use **PgPool** for metadata and **DedupService** for blob content. + +### FolderDbRepository + +**File:** `src/infrastructure/repositories/pg/folder_db_repository.rs` + +```rust +pub struct FolderDbRepository { + pool: Option>, +} +``` + +Implements `FolderRepository`. Uses recursive CTEs for path building, unique constraints for name dedup within parent, and soft-delete flags for trash operations. + +Key methods: `create_folder`, `get_folder`, `get_folder_by_path`, `list_folders`, `rename_folder`, `move_folder`, `delete_folder`, `move_to_trash`, `restore_from_trash`, `create_home_folder`, `get_folder_user_id`. + +`new_stub()` creates a pool-less instance for `AppState::default()`. + +### FileBlobReadRepository + +**File:** `src/infrastructure/repositories/pg/file_blob_read_repository.rs` + +```rust +pub struct FileBlobReadRepository { + pool: Arc, + dedup: Arc, + folder_repo: Arc, +} +``` + +Implements `FileReadPort`. Reads metadata from `storage.files` and content from blob store via `dedup.read_blob()` / `read_blob_bytes()`. + +Key methods: `get_file`, `list_files`, `get_file_content`, `get_file_stream`, `get_file_range_stream`, `get_file_mmap`, `get_file_path`, `get_parent_folder_id`. + +### FileBlobWriteRepository + +**File:** `src/infrastructure/repositories/pg/file_blob_write_repository.rs` + +```rust +pub struct FileBlobWriteRepository { + pool: Arc, + dedup: Arc, + folder_repo: Arc, +} +``` + +Implements `FileWritePort`. Stores content via `dedup.store_bytes()` (returns hash), then INSERTs metadata into `storage.files`. + +Key methods: `save_file`, `save_file_from_stream`, `move_file`, `rename_file`, `delete_file`, `update_file_content`, `move_to_trash`, `restore_from_trash`, `delete_file_permanently`. + +### TrashDbRepository + +**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs` + +```rust +pub struct TrashDbRepository { + pool: Arc, + retention_days: u32, +} +``` + +Implements `TrashRepository`. Reads from `storage.trash_items` VIEW. `clear_trash` DELETEs rows where `is_trashed = TRUE`. `get_expired_items` checks `trashed_at` against the configured retention period. + +--- + +## Path Service + +**File:** `src/infrastructure/services/path_service.rs` + +```rust +pub struct PathService { + root_path: PathBuf, // e.g., ./storage +} +``` + +Used for resolving storage root paths (blob storage directory, thumbnail paths, etc.). Not used for per-user folder resolution — that is handled by `FolderDbRepository` via PostgreSQL. + +### StoragePath (Domain Value Object) + +**File:** `src/domain/services/path_service.rs` + +```rust +#[derive(Debug, Clone, PartialEq, Eq, Default)] +pub struct StoragePath { + segments: Vec, +} +``` + +| Method | Description | +|---|---| +| `root()` | Empty path (storage root) | +| `from_string(path)` | Parse from `/`-delimited string | +| `join(segment)` | Append a segment | +| `file_name()` | Last segment | +| `parent()` | All segments except last | +| `to_string()` | Join segments with `/` | + +### Trait Implementations + +- **StoragePort** — `resolve_path()`, `ensure_directory()`, `file_exists()`, `directory_exists()` + +--- + +## Session Management + +### Session Entity + +**File:** `src/domain/entities/session.rs` + +```rust +pub struct Session { + id: String, // UUID v4 + user_id: String, + refresh_token: String, + expires_at: DateTime, + ip_address: Option, + user_agent: Option, + created_at: DateTime, + revoked: bool, +} +``` + +Constructors: +- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` — generates UUID, panics if **user_id** or **refresh_token** empty +- `Session::from_raw(...)` — for DB reconstruction + +### SessionRepository (Domain Port) + +**File:** `src/domain/repositories/session_repository.rs` + +```rust +#[async_trait] +pub trait SessionRepository: Send + Sync + 'static { + async fn create_session(&self, session: Session) -> SessionRepositoryResult; + async fn get_session_by_id(&self, id: &str) -> SessionRepositoryResult; + async fn get_session_by_refresh_token(&self, token: &str) -> SessionRepositoryResult; + async fn get_sessions_by_user_id(&self, user_id: &str) -> SessionRepositoryResult>; + async fn revoke_session(&self, session_id: &str) -> SessionRepositoryResult<()>; + async fn revoke_all_user_sessions(&self, user_id: &str) -> SessionRepositoryResult; + async fn delete_expired_sessions(&self) -> SessionRepositoryResult; +} +``` + +### SessionStoragePort (Application Port) + +**File:** `src/application/ports/auth_ports.rs` + +```rust +#[async_trait] +pub trait SessionStoragePort: Send + Sync + 'static { + async fn create_session(&self, session: Session) -> Result; + async fn get_session_by_refresh_token(&self, token: &str) -> Result; + async fn revoke_session(&self, session_id: &str) -> Result<(), DomainError>; + async fn revoke_all_user_sessions(&self, user_id: &str) -> Result; +} +``` + +### SessionPgRepository (Infrastructure) + +**File:** `src/infrastructure/repositories/pg/session_pg_repository.rs` + +```rust +pub struct SessionPgRepository { + pool: Arc, +} +``` + +Implements both **SessionRepository** and **SessionStoragePort**. Uses `with_transaction()` helper for write operations. `create_session` also updates `auth.users.last_login_at` within the same transaction. + +### Database Schema + +```sql +CREATE TABLE IF NOT EXISTS auth.sessions ( + id VARCHAR(36) PRIMARY KEY, + user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, + refresh_token TEXT NOT NULL UNIQUE, + expires_at TIMESTAMP WITH TIME ZONE NOT NULL, + ip_address TEXT, + user_agent TEXT, + created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP, + revoked BOOLEAN NOT NULL DEFAULT FALSE +); + +CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id); +CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token); +CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at); +CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked) + WHERE NOT revoked AND is_session_active(expires_at); +``` + +### Auth Service + +**File:** `src/application/services/auth_application_service.rs` + +**AuthApplicationService** orchestrates authentication using: +- **UserStoragePort** — user CRUD +- **SessionStoragePort** — session lifecycle +- **PasswordHasherPort** — Argon2id hashing +- **TokenServicePort** — JWT generation/validation +- `RwLock` — hot-reloadable OIDC configuration +- `Mutex>` — in-flight OIDC login states + +Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**. + +--- + +## File Use Case Factory + +**File:** `src/application/services/file_use_case_factory.rs` + +```rust +pub trait FileUseCaseFactory: Send + Sync + 'static { + fn create_file_upload_use_case(&self) -> Arc; + fn create_file_retrieval_use_case(&self) -> Arc; + fn create_file_management_use_case(&self) -> Arc; +} +``` + +**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**. + +### File Operation Port Hierarchy + +| Port | Key Methods | +|---|---| +| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `Buffered` <1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` | +| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` | +| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) | + +--- + +## Architecture Diagram + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Interfaces Layer │ +│ Axum Router → API Routes + Middleware (Auth, Compress) │ +└─────────────────────┬───────────────────────────────────────┘ + │ Arc +┌─────────────────────▼───────────────────────────────────────┐ +│ Application Layer │ +│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ +│ │ FileUpload │ │ FolderService│ │ AuthApplication │ │ +│ │ FileRetrieval│ │ SearchService│ │ AdminSettings │ │ +│ │ FileMgmt │ │ I18nService │ │ TrashService │ │ +│ └──────┬───────┘ └──────┬───────┘ └──────────┬──────────┘ │ +│ │ Ports (traits) │ │ │ +└─────────┼────────────────┼─────────────────────┼────────────┘ + │ │ │ +┌─────────▼────────────────▼─────────────────────▼────────────┐ +│ Infrastructure Layer │ +│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ +│ │ FileBlobRead │ │ PathService │ │ SessionPg │ │ +│ │ FileBlobWrite │ │ DedupService │ │ UserPg │ │ +│ │ FolderDb │ │ Thumbnail │ │ JwtTokenService │ │ +│ │ TrashDb │ │ Transcode │ │ Argon2Hasher │ │ +│ └────────────────┘ └──────────────┘ └──────────────────┘ │ +│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ +│ │ ContentCache │ │ Compression │ │ ChunkedUpload │ │ +│ │ BufferPool │ │ ZipService │ │ ShareFsRepo │ │ +│ └────────────────┘ └──────────────┘ └──────────────────┘ │ +└─────────────────────────────────────────────────────────────┘ + │ +┌─────────────────────▼───────────────────────────────────────┐ +│ Domain Layer │ +│ Entities: File, Folder, Session, User, Calendar, Contact │ +│ Value Objects: StoragePath │ +│ Repository Traits: FolderRepository, TrashRepository, ... │ +│ Domain Errors │ +└─────────────────────────────────────────────────────────────┘ +``` + +### Data Flow: File Upload + +``` +HTTP Request (multipart) + → FileUploadService.smart_upload() + → FileBlobWriteRepository.save_file() / save_file_from_stream() + → DedupService.store_bytes() → .blobs/{prefix}/{hash}.blob + → INSERT INTO storage.files (name, folder_id, blob_hash, size, ...) + → 201 Created (FileDto) +``` + +### Data Flow: File Download + +``` +HTTP Request (GET /api/files/{id}/download) + → FileRetrievalService.get_file_optimized() + → ContentCache hit? → serve from RAM + → FileBlobReadRepository.get_file_content() / get_file_stream() + → SELECT blob_hash FROM storage.files WHERE id = $1 + → DedupService.read_blob(hash) → bytes from .blobs/ + → Optional WebP transcode → response +``` + +### Data Flow: Folder Operations + +``` +HTTP Request (POST /api/folders) + → FolderService.create_folder() + → FolderDbRepository.create_folder() + → INSERT INTO storage.folders (name, parent_id, user_id, ...) + → 201 Created (FolderDto) +``` diff --git a/doc/share-integration.md b/doc/share-integration.md index 1b00efcf..2b66ecd0 100644 --- a/doc/share-integration.md +++ b/doc/share-integration.md @@ -134,7 +134,7 @@ Handles: shared element validation, permission management, unique link/token gen ## Infrastructure -**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share links to the filesystem: +**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share link metadata to a local JSON file: ```rust pub struct ShareFsRepository { @@ -157,7 +157,9 @@ struct ShareRecord { } ``` -Stores shared links in a JSON file. Supports queries and updates, search by ID/token/user, and pagination. +Stores share link records in a JSON file. Supports queries and updates, search by ID/token/user, and pagination. + +> **Note:** This repository stores *share link metadata* only (tokens, permissions, expiration). The actual file/folder content is accessed via `FileReadPort` / `FolderStoragePort` which use the blob storage model (PostgreSQL metadata + DedupService blobs). ## API Handlers and Routes @@ -256,26 +258,30 @@ The service is instantiated via **AppServiceFactory** in `src/common/di.rs` and ```rust // In AppServiceFactory::create_share_service() -let share_service: Option> = if config.features.enable_file_sharing { - let share_repository = Arc::new(ShareFsRepository::new(Arc::new(config.clone()))); - let share_service = Arc::new(ShareService::new( - Arc::new(config.clone()), - share_repository, - file_read_repository.clone(), - folder_repository.clone(), - password_hasher.clone(), - )); - Some(share_service) -} else { - None -}; +pub fn create_share_service(&self, repos: &RepositoryServices) + -> Option> +{ + if !self.config.features.enable_file_sharing { + return None; + } -// Add to AppState -let app_state = AppState { - // ... - share_service: share_service.clone(), - // ... -}; + let share_repository = Arc::new(ShareFsRepository::new( + Arc::new(self.config.clone()) + )); + + let password_hasher: Arc = + Arc::new(Argon2PasswordHasher::new()); + + let service = Arc::new(ShareService::new( + Arc::new(self.config.clone()), + share_repository, + repos.file_read_repository.clone(), // FileBlobReadRepository + repos.folder_repository.clone(), // FolderDbRepository + password_hasher, + )); + + Some(service) +} ``` ## Workflows @@ -353,8 +359,9 @@ HTTP status code mapping: ## Technical Notes -- **Performance**: JSON file-based storage works for moderate volumes. For higher load, migrate to a database. -- **Scalability**: the design supports horizontal scaling via distributed or cloud-based repositories. +- **Share metadata** is stored in a local JSON file via `ShareFsRepository`. This is separate from the 100% blob storage model used for file content. +- **File/folder lookups** during share access go through `FileReadPort` / `FolderStoragePort`, which read metadata from PostgreSQL and content from the DedupService blob store. +- **Scalability**: for higher load, share metadata could be migrated to PostgreSQL using the same hexagonal architecture (implement `ShareRepository` with PgPool). - **Maintenance**: clear separation of concerns makes testing and maintenance straightforward. -The sharing feature is enabled by default in the current configuration. +The sharing feature is enabled via `OXICLOUD_ENABLE_FILE_SHARING` configuration flag. diff --git a/doc/trash-feature-summary.md b/doc/trash-feature-summary.md index 30a2dff7..afa5f456 100644 --- a/doc/trash-feature-summary.md +++ b/doc/trash-feature-summary.md @@ -1,79 +1,117 @@ -# 14 - Trash Feature - -Soft-delete for files and folders. Items go to a per-user trash bin instead of being permanently removed. Configurable retention period with automatic cleanup. - -## Architecture - -Follows the hexagonal architecture: - -1. **Domain Layer** (`/src/domain/`): - - Entities: **TrashedItem** representing files and folders in the trash - - Repository interfaces: **TrashRepository** defining trash management operations - -2. **Application Layer** (`/src/application/`): - - DTOs: **TrashedItemDto** for data transfer between layers - - Ports: **TrashUseCase** defining available operations - - Services: **TrashService** implementing the trash use cases - -3. **Infrastructure Layer** (`/src/infrastructure/`): - - Repositories: **TrashFsRepository** for filesystem-based trash storage - - Trash-related methods in existing repositories: `FileWriteRepository::move_to_trash()`, `FolderRepository::move_to_trash()`, etc. - - Services: **TrashCleanupService** for automatic cleanup of expired items - -4. **Interface Layer** (`/src/interfaces/`): - - API handlers: `trash_handler.rs` with HTTP endpoints for trash operations - - Routes: updated `routes.rs` to include trash endpoints - -## Key Features - -1. **Soft Deletion** -- files and folders move to trash, not immediately deleted -2. **Per-User Trash** -- each user has an isolated trash bin -3. **Retention Policy** -- items auto-delete after a configurable period -4. **Restoration** -- items can be restored to their original location -5. **Permanent Deletion** -- items can be permanently deleted before retention expires -6. **Empty Trash** -- wipe everything in the trash at once - -## API Endpoints - -- `GET /api/trash` or `GET /api/trash/` -- list all items in the user's trash -- `DELETE /api/trash/files/:id` -- move a file to trash -- `DELETE /api/trash/folders/:id` -- move a folder to trash -- `POST /api/trash/:id/restore` -- restore an item to its original location -- `DELETE /api/trash/:id` -- permanently delete an item from trash -- `DELETE /api/trash/empty` -- empty the entire trash bin - -## Testing - -1. **Unit Tests** -- testing **TrashService**: - - Move files and folders to trash - - Restore items from trash - - Permanent deletion - - Empty trash operation - -2. **Integration Tests** -- Python script hitting the API endpoints: - - End-to-end testing of all trash operations - - Verification of move, list, restore, and delete behavior - -3. **Shell Script** -- for manual testing and demonstration - -## Configuration - -- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true) -- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**) - -## Implementation Details - -1. **Physical File Storage** -- when items are trashed, they physically move to a `.trash` directory -2. **Metadata Storage** -- trashed item info stored in a separate database table or file -3. **User Isolation** -- trash items are isolated by user ID -4. **Automatic Cleanup** -- a background job runs periodically to clean up expired items -5. **Transaction Safety** -- operations are atomic with proper error handling - -## Future Enhancements - -1. **Trash Quotas** -- limit trash storage per user -2. **Batch Operations** -- trash, restore, or delete multiple items at once -3. **Storage Optimization** -- deduplication for trashed items -4. **Version Control** -- track file versions when moving to trash -5. **Scheduled Cleanup** -- let users configure custom retention periods -6. **Trash Monitoring** -- metrics and alerts for trash usage and cleanup +# 14 - Trash Feature + +Soft-delete for files and folders. Items go to a per-user trash bin instead of being permanently removed. Configurable retention period with automatic cleanup. + +## Architecture + +Follows the hexagonal architecture: + +1. **Domain Layer** (`/src/domain/`): + - Entities: **TrashedItem** representing files and folders in the trash + - Repository interfaces: **TrashRepository** defining trash management operations + +2. **Application Layer** (`/src/application/`): + - DTOs: **TrashedItemDto** for data transfer between layers + - Ports: **TrashUseCase** defining available operations + - Services: **TrashService** implementing the trash use cases + +3. **Infrastructure Layer** (`/src/infrastructure/`): + - Repositories: **TrashDbRepository** (PostgreSQL) — reads from `storage.trash_items` VIEW, manages soft-delete flags + - Trash-related methods in file/folder repositories: `FileBlobWriteRepository::move_to_trash()`, `FolderDbRepository::move_to_trash()`, etc. + - Services: **TrashCleanupService** for automatic cleanup of expired items + +4. **Interface Layer** (`/src/interfaces/`): + - API handlers: `trash_handler.rs` with HTTP endpoints for trash operations + - Routes: updated `routes.rs` to include trash endpoints + +## Storage Model + +Trash uses a **soft-delete** model in PostgreSQL: + +- Files and folders have `is_trashed` (BOOLEAN) and `trashed_at` (TIMESTAMPTZ) columns +- When an item is trashed, `is_trashed` is set to `TRUE` and `trashed_at` records the timestamp +- `original_parent_id` / `original_folder_id` stores the original location for restore +- The `storage.trash_items` VIEW provides a unified list of all trashed items (files + folders) +- No physical file movement occurs — blob content stays at `.blobs/{prefix}/{hash}.blob` +- Permanent deletion removes the DB row and decrements the blob reference counter + +```sql +CREATE OR REPLACE VIEW storage.trash_items AS + SELECT id, name, 'file' AS item_type, folder_id AS parent_id, + user_id, size, mime_type, trashed_at, created_at + FROM storage.files WHERE is_trashed = TRUE + UNION ALL + SELECT id, name, 'folder' AS item_type, parent_id, + user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at + FROM storage.folders WHERE is_trashed = TRUE; +``` + +## Key Features + +1. **Soft Deletion** — files and folders are flagged as trashed, not immediately deleted +2. **Per-User Trash** — each user has an isolated trash bin (filtered by `user_id`) +3. **Retention Policy** — items auto-delete after a configurable period +4. **Restoration** — items can be restored to their original location via `original_parent_id`/`original_folder_id` +5. **Permanent Deletion** — items can be permanently deleted before retention expires (removes DB row + decrements blob ref) +6. **Empty Trash** — wipe everything in the trash at once + +## API Endpoints + +- `GET /api/trash` or `GET /api/trash/` — list all items in the user's trash +- `DELETE /api/trash/files/:id` — move a file to trash +- `DELETE /api/trash/folders/:id` — move a folder to trash +- `POST /api/trash/:id/restore` — restore an item to its original location +- `DELETE /api/trash/:id` — permanently delete an item from trash +- `DELETE /api/trash/empty` — empty the entire trash bin + +## Implementation Details + +### TrashDbRepository + +**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs` + +```rust +pub struct TrashDbRepository { + pool: Arc, + retention_days: u32, +} +``` + +Key methods: +- `get_trash_items(user_id)` — SELECT from `storage.trash_items` WHERE `user_id = $1` +- `clear_trash(user_id)` — DELETE from `storage.files` and `storage.folders` WHERE `is_trashed = TRUE AND user_id = $1` +- `get_expired_items()` — finds items where `trashed_at + retention_days < NOW()` + +### TrashService + +**File:** `src/application/services/trash_service.rs` + +Constructor: `TrashService::new(trash_repo, file_read, file_write, folder_repo, retention_days)` + +Orchestrates trash operations by delegating to the appropriate repository: +- Moving a file to trash → `FileBlobWriteRepository::move_to_trash()` +- Moving a folder to trash → `FolderDbRepository::move_to_trash()` +- Permanent deletion → removes DB row + calls `DedupService::decrement_ref()` to clean blob if unreferenced + +### TrashCleanupService + +**File:** `src/infrastructure/services/trash_cleanup_service.rs` + +Background job that runs every 24 hours to permanently delete items past the retention period. + +## Testing + +1. **Unit Tests** — testing **TrashService**: + - Move files and folders to trash + - Restore items from trash + - Permanent deletion + - Empty trash operation + +2. **Integration Tests** — Python script hitting the API endpoints: + - End-to-end testing of all trash operations + - Verification of move, list, restore, and delete behavior + +## Configuration + +- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true) +- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**)