docs: update documentation to reflect 100% blob storage model
Rewrite documentation to match the new architecture where all file metadata lives in PostgreSQL and content is stored as content-addressed blobs via DedupService. Updated files: - internal-architecture.md: complete rewrite — new DB schema, blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated DI container, service groups, architecture diagram, data flows - file-system-safety.md: repurposed as storage-safety.md — covers PostgreSQL ACID guarantees + DedupService atomic writes - caching-architecture.md: updated repo references to blob repos, removed write-behind cache section, updated upload/download flows - trash-feature-summary.md: rewritten for soft-delete model (is_trashed flag, trash_items VIEW, TrashDbRepository) - share-integration.md: clarified ShareFsRepository scope, updated DI snippet, added blob storage context note - deduplication.md: updated DI snippet (dedup injected into repos) - deployment.md: updated feature matrix (file storage requires DB) - important-delta-sync-implementation.md: updated DI references Removed legacy references: IdMappingPort, StorageMediator, WriteBehindCache, FsFileRepository, FsFolderRepository, TrashFsRepository, folder_ids.json, file_ids.json.
This commit is contained in:
Dionisio
+274
-370
@@ -1,370 +1,274 @@
|
|||||||
# 04 - Caching Architecture
|
# 04 - Caching Architecture
|
||||||
|
|
||||||
OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads.
|
OxiCloud uses a multi-layer caching system spanning HTTP-level caching down to kernel-level memory mapping. Covers both uploads and downloads.
|
||||||
|
|
||||||
## Cache Layers Summary
|
## Cache Layers Summary
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────────────────────────────────────────────┐
|
┌─────────────────────────────────────────────────────┐
|
||||||
│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints
|
│ Layer 0: HTTP Cache Middleware (ETag + 304) │ All endpoints
|
||||||
├─────────────────────────────────────────────────────┤
|
├─────────────────────────────────────────────────────┤
|
||||||
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
|
│ Layer 1: File Content Cache (LRU, <10MB files) │ Downloads
|
||||||
├─────────────────────────────────────────────────────┤
|
├─────────────────────────────────────────────────────┤
|
||||||
│ Layer 2: MMAP (memmap2, 10-100MB files) │ Downloads
|
│ Layer 2: MMAP (memmap2, 10-100MB blobs) │ Downloads
|
||||||
├─────────────────────────────────────────────────────┤
|
├─────────────────────────────────────────────────────┤
|
||||||
│ Layer 3: Streaming (FramedRead, ≥100MB files) │ Downloads
|
│ Layer 3: Streaming (FramedRead, ≥100MB blobs) │ Downloads
|
||||||
├─────────────────────────────────────────────────────┤
|
├─────────────────────────────────────────────────────┤
|
||||||
│ Layer 4: File Metadata Cache (adaptive TTL) │ All file ops
|
│ Layer 4: Buffer Pool (reusable I/O buffers) │ Compression
|
||||||
├─────────────────────────────────────────────────────┤
|
└─────────────────────────────────────────────────────┘
|
||||||
│ Layer 5: Write-Behind Cache (<256KB uploads) │ Uploads
|
```
|
||||||
├─────────────────────────────────────────────────────┤
|
|
||||||
│ Layer 6: Buffer Pool (reusable I/O buffers) │ Compression
|
> **Note:** File metadata (name, size, MIME type, folder) is served from PostgreSQL — no separate filesystem metadata cache is needed.
|
||||||
└─────────────────────────────────────────────────────┘
|
|
||||||
```
|
---
|
||||||
|
|
||||||
---
|
## Layer 0: HTTP Cache Middleware
|
||||||
|
|
||||||
## Layer 0: HTTP Cache Middleware
|
**File**: `src/interfaces/middleware/cache.rs`
|
||||||
|
|
||||||
**File**: `src/interfaces/middleware/cache.rs`
|
Generic HTTP caching layer applied to API endpoints.
|
||||||
|
|
||||||
Generic HTTP caching layer applied to API endpoints.
|
| Parameter | Value |
|
||||||
|
|---|---|
|
||||||
| Parameter | Value |
|
| Max entries | 1,000 |
|
||||||
|---|---|
|
| Default max-age | 60 seconds |
|
||||||
| Max entries | 1,000 |
|
| Eviction | LRU (oldest 10% when full) |
|
||||||
| Default max-age | 60 seconds |
|
| Cleanup | Background task every 5 minutes |
|
||||||
| Eviction | LRU (oldest 10% when full) |
|
|
||||||
| Cleanup | Background task every 5 minutes |
|
Features:
|
||||||
|
- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`)
|
||||||
Features:
|
- `Cache-Control` header injection
|
||||||
- ETag-based conditional requests (`If-None-Match` → `304 Not Modified`)
|
- Implements Tower `Layer` + `Service` traits for Axum integration
|
||||||
- `Cache-Control` header injection
|
- Per-request key: method + URI
|
||||||
- Implements Tower `Layer` + `Service` traits for Axum integration
|
|
||||||
- Per-request key: method + URI
|
---
|
||||||
|
|
||||||
---
|
## Layer 1: File Content Cache (Download Tier 1)
|
||||||
|
|
||||||
## Layer 1: File Content Cache (Download Tier 1)
|
**File**: `src/infrastructure/services/file_content_cache.rs`
|
||||||
|
|
||||||
**File**: `src/infrastructure/services/file_content_cache.rs`
|
In-memory LRU cache for small files, served directly from RAM.
|
||||||
|
|
||||||
In-memory LRU cache for small files, served directly from RAM.
|
| Parameter | Value |
|
||||||
|
|---|---|
|
||||||
| Parameter | Value |
|
| Max file size | 10 MB per file |
|
||||||
|---|---|
|
| Max total cache size | 512 MB |
|
||||||
| Max file size | 10 MB per file |
|
| Max entries | 10,000 |
|
||||||
| Max total cache size | 512 MB |
|
| Structure | `lru::LruCache<String, CacheEntry>` |
|
||||||
| Max entries | 10,000 |
|
| Latency | ~0.1ms |
|
||||||
| Structure | `lru::LruCache<String, CacheEntry>` |
|
|
||||||
| Latency | ~0.1ms |
|
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
|
||||||
|
|
||||||
**CacheEntry**: `{ data: Bytes, etag: String, content_type: String, size: usize }`
|
Methods:
|
||||||
|
- `should_cache(size)` — checks if file fits in cache
|
||||||
Methods:
|
- `get(file_id)` → `Option<(Bytes, String, String)>` — returns (data, etag, content_type)
|
||||||
- `should_cache(size)` -- checks if file fits in cache
|
- `put(file_id, content, etag, content_type)` — inserts with LRU eviction
|
||||||
- `get(file_id)` → `Option<(Bytes, String, String)>` -- returns (data, etag, content_type)
|
- `invalidate(file_id)`, `clear()`
|
||||||
- `put(file_id, content, etag, content_type)` -- inserts with LRU eviction
|
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
|
||||||
- `invalidate(file_id)`, `clear()`
|
|
||||||
- `stats()` → `CacheStats { current_size_bytes, max_size_bytes, hits, misses, hit_rate_percent }`
|
Port: implements **ContentCachePort** trait.
|
||||||
|
|
||||||
Port: implements **ContentCachePort** trait.
|
---
|
||||||
|
|
||||||
---
|
## Layer 2: MMAP (Download Tier 2)
|
||||||
|
|
||||||
## Layer 2: MMAP (Download Tier 2)
|
**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||||
|
|
||||||
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs`
|
Memory-mapped I/O for medium blobs using `memmap2`.
|
||||||
|
|
||||||
Memory-mapped I/O for medium files using `memmap2`.
|
| Parameter | Value |
|
||||||
|
|---|---|
|
||||||
| Parameter | Value |
|
| File range | 10 MB - 100 MB |
|
||||||
|---|---|
|
| Implementation | `memmap2::Mmap` via `spawn_blocking` |
|
||||||
| File range | 10 MB - 100 MB |
|
| Latency | ~1-5ms |
|
||||||
| Implementation | `memmap2::Mmap` via `spawn_blocking` |
|
|
||||||
| Latency | ~1-5ms |
|
The blob file (`.blobs/{prefix}/{hash}.blob`) is memory-mapped and its contents copied to `Bytes`. Benefits from kernel page cache for frequently accessed blobs.
|
||||||
|
|
||||||
Current implementation copies mmap'd data to `Bytes` (`Bytes::copy_from_slice(&mmap[..])`). Not true zero-copy, but still benefits from kernel page cache.
|
---
|
||||||
|
|
||||||
---
|
## Layer 3: Streaming (Download Tier 3)
|
||||||
|
|
||||||
## Layer 3: Streaming (Download Tier 3)
|
**File**: `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||||
|
|
||||||
**File**: `src/infrastructure/repositories/file_fs_read_repository.rs`
|
Chunked streaming for large blobs using tokio-util codecs.
|
||||||
|
|
||||||
Chunked streaming for large files using tokio-util codecs.
|
| Parameter | Value |
|
||||||
|
|---|---|
|
||||||
| Parameter | Value |
|
| File range | ≥100 MB |
|
||||||
|---|---|
|
| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) |
|
||||||
| File range | ≥100 MB |
|
| Implementation | `FramedRead` + `BytesCodec` |
|
||||||
| Chunk size | 1 MB (configurable via **ResourceConfig.chunk_size_bytes**) |
|
| RAM usage | Near zero (one chunk at a time) |
|
||||||
| Implementation | `FramedRead` + `BytesCodec` |
|
|
||||||
| RAM usage | Near zero (one chunk at a time) |
|
---
|
||||||
|
|
||||||
---
|
## Layer 4: Buffer Pool
|
||||||
|
|
||||||
## Layer 4: File Metadata Cache
|
**File**: `src/infrastructure/services/buffer_pool.rs`
|
||||||
|
|
||||||
**File**: `src/infrastructure/services/file_metadata_cache.rs`
|
Reusable byte buffer pool to reduce allocation pressure during compression operations.
|
||||||
|
|
||||||
Caches filesystem metadata (existence, size, MIME type, timestamps) to avoid repeated `stat()` calls.
|
| Parameter | Value |
|
||||||
|
|---|---|
|
||||||
| Parameter | Value |
|
| Buffer size | 64 KB |
|
||||||
|---|---|
|
| Max buffers | 100 |
|
||||||
| Default file TTL | 60 seconds |
|
| Buffer TTL | 60 seconds |
|
||||||
| Default directory TTL | 120 seconds |
|
| Concurrency control | `tokio::sync::Semaphore` |
|
||||||
| Max entries | 10,000 |
|
|
||||||
| Adaptive TTL multiplier | 5x for popular entries (≥10 accesses) |
|
Features:
|
||||||
| LRU eviction | Frees 10% capacity when full |
|
- `get_buffer()` — borrows a buffer (blocks if pool exhausted)
|
||||||
| Cleanup | Background task runs periodically |
|
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
|
||||||
|
- Expired buffers are cleaned periodically via `start_cleaner()`
|
||||||
**CachedMetadata:**
|
- Tracks stats: gets, hits, misses, returns, evictions, waits
|
||||||
```rust
|
|
||||||
pub struct FileMetadata {
|
---
|
||||||
pub path: PathBuf,
|
|
||||||
pub exists: bool,
|
## Configuration
|
||||||
pub entry_type: CacheEntryType, // File | Directory | Unknown
|
|
||||||
pub size: Option<u64>,
|
All cache-related config in `src/common/config.rs`:
|
||||||
pub mime_type: Option<String>,
|
|
||||||
pub created_at: Option<u64>,
|
```rust
|
||||||
pub modified_at: Option<u64>,
|
pub struct ResourceConfig {
|
||||||
pub last_access: Instant,
|
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
|
||||||
pub expires_at: Instant,
|
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
|
||||||
pub access_count: usize,
|
pub max_in_memory_file_size_mb: u64, // 50 MB
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**Adaptive TTL**: entries accessed ≥10 times get 5x the configured TTL, keeping frequently accessed file metadata in cache longer.
|
## Download Flow
|
||||||
|
|
||||||
Port: implements **MetadataCachePort** trait.
|
```
|
||||||
|
Request → ETag check (304?) → Range request (206?)
|
||||||
---
|
→ file size < 10MB? → Tier 1: LRU cache (RAM)
|
||||||
|
→ file size < 100MB? → Tier 2: MMAP (kernel page cache on blob file)
|
||||||
## Layer 5: Write-Behind Cache
|
→ file size ≥ 100MB → Tier 3: Streaming (chunked from blob file)
|
||||||
|
```
|
||||||
**File**: `src/infrastructure/services/write_behind_cache.rs`
|
|
||||||
|
In all tiers, metadata (file name, size, MIME type) comes from a PostgreSQL `SELECT` on `storage.files`. Content is read from the DedupService blob at `.blobs/{prefix}/{hash}.blob`.
|
||||||
Buffers small uploads in RAM and confirms immediately. Flushes to disk asynchronously.
|
|
||||||
|
---
|
||||||
| Parameter | Value |
|
|
||||||
|---|---|
|
## Range Requests (HTTP 206 Partial Content)
|
||||||
| Max file size | 1 MB per file |
|
|
||||||
| Max total cache | 100 MB |
|
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||||
| Max pending duration | 30 seconds |
|
|
||||||
| Flush interval | 100 ms |
|
**Crate**: `http-range-header = "0.4"` for parsing.
|
||||||
| Write strategy | Atomic (temp file + rename) |
|
|
||||||
|
### Request Processing Flow
|
||||||
Architecture:
|
|
||||||
- `put_pending(file_id, content, target_path)` stores bytes in `HashMap<String, PendingWrite>`
|
```
|
||||||
- Background `flush_worker` processes **FlushCommands** via `mpsc` channel
|
Range header present?
|
||||||
- Periodic checker force-flushes entries older than 30 seconds
|
├─ parse_range_header(range_str)
|
||||||
- `get_pending(file_id)` serves reads while data is still in RAM (before flush)
|
│ ├─ Parse OK → ranges.validate(file_size)
|
||||||
|
│ │ ├─ Valid → take first range → get_file_range_stream(start, end+1)
|
||||||
Port: implements **WriteBehindCachePort** trait.
|
│ │ │ ├─ Stream OK → 206 Partial Content
|
||||||
|
│ │ │ └─ Stream Err → fall through to normal download (200)
|
||||||
Statistics:
|
│ │ └─ Invalid → 416 Range Not Satisfiable
|
||||||
```rust
|
│ └─ Parse Err → fall through to normal download (200)
|
||||||
pub struct WriteBehindStatsDto {
|
└─ No Range header → normal 3-tier download
|
||||||
pub pending_count: usize,
|
```
|
||||||
pub pending_bytes: usize,
|
|
||||||
pub total_writes: u64,
|
### Response Headers (206)
|
||||||
pub total_bytes_written: u64,
|
|
||||||
pub cache_hits: u64,
|
| Header | Value |
|
||||||
pub avg_flush_time_us: u64,
|
|---|---|
|
||||||
}
|
| `Content-Type` | File MIME type |
|
||||||
```
|
| `Content-Range` | `bytes {start}-{end}/{total_size}` |
|
||||||
|
| `Content-Length` | Range length (end - start + 1) |
|
||||||
---
|
| `Accept-Ranges` | `bytes` |
|
||||||
|
| `ETag` | `"{file_id}-{modified_at}"` |
|
||||||
## Layer 6: Buffer Pool
|
| `Cache-Control` | `private, max-age=3600, must-revalidate` |
|
||||||
|
|
||||||
**File**: `src/infrastructure/services/buffer_pool.rs`
|
### 416 Range Not Satisfiable
|
||||||
|
|
||||||
Reusable byte buffer pool to reduce allocation pressure during compression operations.
|
Returned when `ranges.validate(file_size)` fails:
|
||||||
|
```
|
||||||
| Parameter | Value |
|
HTTP/1.1 416 Range Not Satisfiable
|
||||||
|---|---|
|
Content-Range: bytes */12345
|
||||||
| Buffer size | 64 KB |
|
```
|
||||||
| Max buffers | 100 |
|
|
||||||
| Buffer TTL | 60 seconds |
|
### Blob File Seek Implementation
|
||||||
| Concurrency control | `tokio::sync::Semaphore` |
|
|
||||||
|
`get_file_range_stream()` at the repository level:
|
||||||
Features:
|
|
||||||
- `get_buffer()` -- borrows a buffer (blocks if pool exhausted)
|
1. Resolves blob path from `blob_hash` via DedupService
|
||||||
- **BorrowedBuffer** auto-returns to pool on `Drop` via `tokio::spawn`
|
2. Opens the blob file with `TokioFile::open()`
|
||||||
- Expired buffers are cleaned periodically via `start_cleaner()`
|
3. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
|
||||||
- Tracks stats: gets, hits, misses, returns, evictions, waits
|
4. Limits read to `range_length` via `fh.take(range_length)`
|
||||||
|
5. Wraps in `FramedRead` + `BytesCodec`
|
||||||
---
|
|
||||||
|
Adaptive chunk size:
|
||||||
## Configuration
|
|
||||||
|
| Range size | Chunk size |
|
||||||
All cache-related config in `src/common/config.rs`:
|
|---|---|
|
||||||
|
| ≤ 1 MB | 8 KB |
|
||||||
```rust
|
| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) |
|
||||||
pub struct CacheConfig {
|
|
||||||
pub file_ttl_ms: u64, // default: 60,000 (1 min)
|
### Tier Interaction
|
||||||
pub directory_ttl_ms: u64, // default: 120,000 (2 min)
|
|
||||||
pub max_entries: usize, // default: 10,000
|
Range requests **bypass all download tiers** (LRU, MMAP). They always use direct blob file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
|
||||||
}
|
|
||||||
|
### Limitations
|
||||||
pub struct ResourceConfig {
|
|
||||||
pub large_file_threshold_mb: u64, // 100 MB (mmap→streaming boundary)
|
- **Multipart ranges not supported**: only the first range in a multi-range request is served.
|
||||||
pub chunk_size_bytes: usize, // 1 MB (streaming chunk size)
|
- **`If-Range` not handled**: no conditional range support.
|
||||||
pub max_in_memory_file_size_mb: u64, // 50 MB
|
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
|
||||||
}
|
|
||||||
```
|
---
|
||||||
|
|
||||||
## Download Flow
|
## Upload Flow
|
||||||
|
|
||||||
```
|
```
|
||||||
Request → ETag check (304?) → Range request (206?)
|
Request → file size < 1MB? → Buffered write (sync to blob store)
|
||||||
→ file size < 10MB? → Tier 1: LRU cache (RAM)
|
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to blob store)
|
||||||
→ file size < 100MB? → Tier 2: MMAP (kernel page cache)
|
```
|
||||||
→ file size ≥ 100MB → Tier 3: Streaming (chunked)
|
|
||||||
```
|
### Upload Strategy Selection
|
||||||
|
|
||||||
---
|
**File**: `src/application/services/file_upload_service.rs`
|
||||||
|
|
||||||
## Range Requests (HTTP 206 Partial Content)
|
```rust
|
||||||
|
pub enum UploadStrategy {
|
||||||
**Files**: `src/interfaces/api/handlers/file_handler.rs`, `src/infrastructure/repositories/file_fs_read_repository.rs`
|
Buffered, // < 1 MB — collect bytes, write to blob store
|
||||||
|
Streaming, // ≥ 1 MB — chunk-by-chunk write via save_file_from_stream
|
||||||
**Crate**: `http-range-header = "0.4"` for parsing.
|
}
|
||||||
|
```
|
||||||
### Request Processing Flow
|
|
||||||
|
| Constant | Value |
|
||||||
```
|
|---|---|
|
||||||
Range header present?
|
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
|
||||||
├─ parse_range_header(range_str)
|
|
||||||
│ ├─ Parse OK → ranges.validate(file_size)
|
### Handler-Level Buffering
|
||||||
│ │ ├─ Valid → take first range → get_file_range_stream(start, end+1)
|
|
||||||
│ │ │ ├─ Stream OK → 206 Partial Content
|
Upload handlers buffer the multipart body in RAM as `Vec<Bytes>` before calling the service layer:
|
||||||
│ │ │ └─ Stream Err → fall through to normal download (200)
|
|
||||||
│ │ └─ Invalid → 416 Range Not Satisfiable
|
```rust
|
||||||
│ └─ Parse Err → fall through to normal download (200)
|
let mut chunks: Vec<Bytes> = Vec::new();
|
||||||
└─ No Range header → normal 3-tier download
|
while let Some(chunk) = field.chunk().await {
|
||||||
```
|
chunks.push(chunk);
|
||||||
|
}
|
||||||
### Response Headers (206)
|
upload_service.smart_upload(..., chunks, total_size).await
|
||||||
|
```
|
||||||
| Header | Value |
|
|
||||||
|---|---|
|
### Buffered Path (< 1 MB)
|
||||||
| `Content-Type` | File MIME type |
|
|
||||||
| `Content-Range` | `bytes {start}-{end}/{total_size}` |
|
Uses `save_file()` — all bytes are passed to `FileBlobWriteRepository`, which calls `DedupService.store_bytes()` to compute hash and store the blob, then INSERTs metadata into `storage.files`.
|
||||||
| `Content-Length` | Range length (end - start + 1) |
|
|
||||||
| `Accept-Ranges` | `bytes` |
|
### Streaming Path (≥ 1 MB)
|
||||||
| `ETag` | `"{file_id}-{modified_at}"` |
|
|
||||||
| `Cache-Control` | `private, max-age=3600, must-revalidate` |
|
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
|
||||||
|
|
||||||
### 416 Range Not Satisfiable
|
```rust
|
||||||
|
let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
|
||||||
Returned when `ranges.validate(file_size)` fails:
|
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
|
||||||
```
|
```
|
||||||
HTTP/1.1 416 Range Not Satisfiable
|
|
||||||
Content-Range: bytes */12345
|
`FileBlobWriteRepository.save_file_from_stream()` collects the stream, stores via DedupService, and INSERTs metadata.
|
||||||
```
|
|
||||||
|
### Dedup Integration
|
||||||
### File Seek Implementation
|
|
||||||
|
Deduplication is handled at the **repository layer** (not the service layer) for all upload strategies. `FileBlobWriteRepository` always calls `DedupService.store_bytes()` which:
|
||||||
`get_file_range_stream()` at the repository level:
|
|
||||||
|
1. Computes SHA-256 hash of content
|
||||||
```rust
|
2. Checks if blob already exists (dedup hit → increment ref count, skip write)
|
||||||
async fn get_file_range_stream(
|
3. If new → atomic write to `.blobs/{prefix}/{hash}.blob`
|
||||||
&self, id: &str, start: u64, end: Option<u64>,
|
4. Returns the hash for storage in `storage.files.blob_hash`
|
||||||
) -> Result<Box<dyn Stream<...> + Send>, DomainError>
|
|
||||||
```
|
|
||||||
|
|
||||||
1. Opens the file with `TokioFile::open()`
|
|
||||||
2. Seeks to `start` via `fh.seek(SeekFrom::Start(start))`
|
|
||||||
3. Limits read to `range_length` via `fh.take(range_length)`
|
|
||||||
4. Wraps in `FramedRead` + `BytesCodec`
|
|
||||||
|
|
||||||
Adaptive chunk size:
|
|
||||||
|
|
||||||
| Range size | Chunk size |
|
|
||||||
|---|---|
|
|
||||||
| ≤ 1 MB | 8 KB |
|
|
||||||
| > 1 MB | 1 MB (from **ResourceConfig.chunk_size_bytes**) |
|
|
||||||
|
|
||||||
### Tier Interaction
|
|
||||||
|
|
||||||
Range requests **bypass all download tiers** (LRU, MMAP, write-behind). They always use direct file seek + streaming. On stream creation error, the handler falls through to the normal `get_file_optimized()` 3-tier path.
|
|
||||||
|
|
||||||
### Limitations
|
|
||||||
|
|
||||||
- **Multipart ranges not supported**: only the first range in a multi-range request is served. Additional ranges are ignored.
|
|
||||||
- **`If-Range` not handled**: no conditional range support.
|
|
||||||
- **`If-Modified-Since` not handled**: only `If-None-Match` (ETag) is checked.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Upload Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
Request → file size < 256KB? → Write-behind cache (instant 201, async flush)
|
|
||||||
→ file size < 1MB? → Buffered write (sync)
|
|
||||||
→ file size ≥ 1MB → Streaming write (chunk-by-chunk to temp + rename)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Upload Strategy Selection
|
|
||||||
|
|
||||||
**File**: `src/application/services/file_upload_service.rs`
|
|
||||||
|
|
||||||
```rust
|
|
||||||
pub enum UploadStrategy {
|
|
||||||
WriteBehind, // < 256 KB — instant response, async disk write
|
|
||||||
Buffered, // 256 KB – 1 MB — sync write to final path
|
|
||||||
Streaming, // ≥ 1 MB — chunk-by-chunk write to temp file + rename
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
| Constant | Value |
|
|
||||||
|---|---|
|
|
||||||
| `WRITE_BEHIND_THRESHOLD` | 256 KB |
|
|
||||||
| `STREAMING_UPLOAD_THRESHOLD` | 1 MB |
|
|
||||||
|
|
||||||
### Handler-Level Buffering
|
|
||||||
|
|
||||||
Both upload handlers (`upload_file` and `upload_file_with_cache`) buffer the **entire multipart body in RAM** as `Vec<Bytes>` before calling the service layer:
|
|
||||||
|
|
||||||
```rust
|
|
||||||
let mut chunks: Vec<Bytes> = Vec::new();
|
|
||||||
while let Some(chunk) = field.chunk().await {
|
|
||||||
chunks.push(chunk);
|
|
||||||
}
|
|
||||||
// All bytes are now in RAM
|
|
||||||
upload_service.smart_upload(..., chunks, total_size).await
|
|
||||||
```
|
|
||||||
|
|
||||||
The "streaming" in `UploadStrategy::Streaming` refers to the **service→repository** path, not the HTTP-body→disk path. By the time `save_file_from_stream()` is called, data is already in memory.
|
|
||||||
|
|
||||||
### Streaming Path (≥ 1 MB): Service → Repository
|
|
||||||
|
|
||||||
`smart_upload()` converts the in-memory `Vec<Bytes>` into a `futures::stream::iter()` and passes it to `save_file_from_stream()`:
|
|
||||||
|
|
||||||
```rust
|
|
||||||
let chunk_stream = stream::iter(chunks.into_iter().map(|c| Ok(c)));
|
|
||||||
self.file_write.save_file_from_stream(name, folder_id, content_type, chunk_stream).await
|
|
||||||
```
|
|
||||||
|
|
||||||
**`save_file_from_stream()` implementation** (`file_fs_write_repository.rs`):
|
|
||||||
|
|
||||||
1. Resolves target path + generates unique name if collision
|
|
||||||
2. Creates temp file: `{target_path}.tmp.upload`
|
|
||||||
3. Iterates stream, writing each chunk with `fh.write_all(&chunk)`
|
|
||||||
4. Calls `fh.flush()` + `fh.sync_all()` for durability
|
|
||||||
5. Atomic rename: `fs::rename(temp_path, final_path)`
|
|
||||||
6. Post-write: ID mapping, cache invalidation, metadata update
|
|
||||||
|
|
||||||
### Buffered Path (256 KB - 1 MB)
|
|
||||||
|
|
||||||
Uses `save_file()` -- writes all bytes directly to the **final path** (no temp file). For larger content, writes in chunks of **ResourceConfig.chunk_size_bytes** (1 MB).
|
|
||||||
|
|
||||||
### Write-Behind Path (< 256 KB)
|
|
||||||
|
|
||||||
See **Layer 5** above. Instant `201`, background flush within 30 seconds.
|
|
||||||
|
|
||||||
### Dedup Pre-Check
|
|
||||||
|
|
||||||
Runs for **all upload strategies** before writing. Re-combines all chunks into a single `Vec<u8>` for hash computation, which means data is temporarily duplicated in RAM during dedup processing.
|
|
||||||
|
|||||||
@@ -285,9 +285,12 @@ pub struct CoreServices {
|
|||||||
// ...
|
// ...
|
||||||
}
|
}
|
||||||
|
|
||||||
// Injected into application services:
|
// Injected into blob repositories (which handle dedup internally):
|
||||||
FileUploadService::new_full(... core.dedup_service.clone())
|
FileBlobReadRepository::new(pool, core.dedup_service.clone(), folder_repo)
|
||||||
FileManagementService::new_full(... core.dedup_service.clone())
|
FileBlobWriteRepository::new(pool, core.dedup_service.clone(), folder_repo)
|
||||||
|
|
||||||
|
// Also injected into FileManagementService for ref cleanup on delete:
|
||||||
|
FileManagementService::new_full(write, read, trash, core.dedup_service.clone())
|
||||||
```
|
```
|
||||||
|
|
||||||
## Persistence
|
## Persistence
|
||||||
|
|||||||
+4
-4
@@ -174,17 +174,17 @@ Hardcoded defaults in `src/common/config.rs`:
|
|||||||
|
|
||||||
| Feature | Requires DB | Requires Auth | Feature Flag |
|
| Feature | Requires DB | Requires Auth | Feature Flag |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| File storage | No | No | Always on |
|
| File storage | Yes | No | Always on |
|
||||||
| Authentication | Yes | -- | `OXICLOUD_ENABLE_AUTH` |
|
| Authentication | Yes | -- | `OXICLOUD_ENABLE_AUTH` |
|
||||||
| OIDC / SSO | Yes | Yes | `OXICLOUD_OIDC_ENABLED` |
|
| OIDC / SSO | Yes | Yes | `OXICLOUD_OIDC_ENABLED` |
|
||||||
| File sharing | Yes | Yes | `OXICLOUD_ENABLE_FILE_SHARING` |
|
| File sharing | Yes | Yes | `OXICLOUD_ENABLE_FILE_SHARING` |
|
||||||
| Trash | No | No | `OXICLOUD_ENABLE_TRASH` |
|
| Trash | Yes | No | `OXICLOUD_ENABLE_TRASH` |
|
||||||
| Search | No | No | `OXICLOUD_ENABLE_SEARCH` |
|
| Search | Yes | No | `OXICLOUD_ENABLE_SEARCH` |
|
||||||
| Favorites | Yes | Yes | Always on (when DB available) |
|
| Favorites | Yes | Yes | Always on (when DB available) |
|
||||||
| Recent items | Yes | Yes | Always on (when DB available) |
|
| Recent items | Yes | Yes | Always on (when DB available) |
|
||||||
| Storage quotas | Yes | Yes | `OXICLOUD_ENABLE_USER_STORAGE_QUOTAS` |
|
| Storage quotas | Yes | Yes | `OXICLOUD_ENABLE_USER_STORAGE_QUOTAS` |
|
||||||
| Admin panel | Yes | Yes | Always on (when auth enabled) |
|
| Admin panel | Yes | Yes | Always on (when auth enabled) |
|
||||||
| WebDAV | No | Optional | Always on |
|
| WebDAV | Yes | Optional | Always on |
|
||||||
| CalDAV | Yes | Yes | Always on (when DB available) |
|
| CalDAV | Yes | Yes | Always on (when DB available) |
|
||||||
| CardDAV | Yes | Yes | Always on (when DB available) |
|
| CardDAV | Yes | Yes | Always on (when DB available) |
|
||||||
| Deduplication | No | No | Always on |
|
| Deduplication | No | No | Always on |
|
||||||
|
|||||||
+125
-156
@@ -1,156 +1,125 @@
|
|||||||
# 03 - File System Safety
|
# 03 - Storage Safety
|
||||||
|
|
||||||
OxiCloud ensures data integrity and durability during file operations through atomic writes, fsync, and directory synchronization. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss.
|
OxiCloud ensures data integrity and durability through a combination of PostgreSQL transactional guarantees and atomic blob writes. The goal: writes either complete fully or not at all, data reaches persistent storage, and the system recovers from crashes or power loss.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## The Problem: Buffered I/O
|
## Storage Model
|
||||||
|
|
||||||
Standard filesystem operations use buffered I/O by default:
|
OxiCloud uses a **100% blob storage model**:
|
||||||
|
|
||||||
```rust
|
- **Metadata** (file names, folder hierarchy, sizes, MIME types, trash status) lives in **PostgreSQL** — protected by ACID transactions.
|
||||||
// This operation may not immediately persist to disk
|
- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob` — protected by atomic writes and fsync.
|
||||||
fs::write(path, content)
|
|
||||||
```
|
---
|
||||||
|
|
||||||
When an application writes data, the OS typically:
|
## PostgreSQL Safety (Metadata)
|
||||||
|
|
||||||
1. Accepts the write into memory buffers
|
All file and folder metadata operations use PostgreSQL transactions:
|
||||||
2. Acknowledges completion to the application
|
|
||||||
3. Schedules the actual disk write for later
|
- **Single-row operations** (INSERT, UPDATE, DELETE) are inherently atomic.
|
||||||
|
- **Multi-step operations** (e.g., move file: UPDATE folder_id + UPDATE path) use explicit transactions via `sqlx`.
|
||||||
A crash during that window means data loss -- the data exists only in memory buffers that haven't been flushed.
|
- **Foreign key constraints** prevent orphaned records (e.g., files referencing non-existent folders).
|
||||||
|
- **Unique constraints** prevent duplicate names within the same parent folder.
|
||||||
---
|
- **Soft-delete** for trash (`is_trashed = TRUE`) preserves data until explicit permanent deletion.
|
||||||
|
|
||||||
## OxiCloud's Approach
|
The `storage.trash_items` VIEW provides a unified read interface over trashed files and folders without duplicating data.
|
||||||
|
|
||||||
All safety mechanisms live in the **FileSystemUtils** service.
|
---
|
||||||
|
|
||||||
### Atomic Write Pattern
|
## Blob Storage Safety (Content)
|
||||||
|
|
||||||
Files are written using write-then-rename:
|
### DedupService Atomic Writes
|
||||||
|
|
||||||
```rust
|
**File:** `src/infrastructure/services/dedup_service.rs`
|
||||||
/// Writes data to a file with fsync to ensure durability
|
|
||||||
/// Uses a safe atomic write pattern: write to temp file, fsync, rename
|
When storing file content, DedupService uses the following pattern:
|
||||||
pub async fn atomic_write<P: AsRef<Path>>(path: P, contents: &[u8]) -> Result<(), IoError>
|
|
||||||
```
|
1. **Hash computation** — SHA-256 hash of content determines the blob path
|
||||||
|
2. **Deduplication check** — if a blob with the same hash exists, only increment the reference counter (no write needed)
|
||||||
Steps:
|
3. **Atomic write** — if new content:
|
||||||
1. Write to a temporary file in the same directory
|
- Write to a temporary file (`.blob.tmp`)
|
||||||
2. Call `fsync` to ensure data is on disk
|
- Call `fsync` to ensure data reaches persistent storage
|
||||||
3. Atomically rename the temp file to the target file
|
- Atomically rename temp file to final path (`.blobs/{prefix}/{hash}.blob`)
|
||||||
4. Sync the parent directory to ensure the rename is persisted
|
4. **Reference counting** — track how many files reference each blob
|
||||||
|
|
||||||
### Directory Synchronization
|
This ensures that a blob either fully exists or doesn't — no partial writes.
|
||||||
|
|
||||||
```rust
|
### FileSystemUtils
|
||||||
/// Creates directories with fsync
|
|
||||||
pub async fn create_dir_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
**File:** `src/infrastructure/services/file_system_utils.rs`
|
||||||
```
|
|
||||||
|
Low-level utilities used internally by DedupService and other infrastructure services:
|
||||||
Directories are created, their entries persisted to disk, and parent directories synchronized too.
|
|
||||||
|
```rust
|
||||||
### Rename and Delete Operations
|
/// Atomic write: temp file → fsync → rename
|
||||||
|
pub async fn atomic_write<P: AsRef<Path>>(path: P, contents: &[u8]) -> Result<(), IoError>
|
||||||
```rust
|
|
||||||
/// Renames a file or directory with proper syncing
|
/// Directory creation with fsync
|
||||||
pub async fn rename_with_sync<P: AsRef<Path>, Q: AsRef<Path>>(from: P, to: Q) -> Result<(), IoError>
|
pub async fn create_dir_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
||||||
|
|
||||||
/// Removes a file with directory syncing
|
/// Rename with directory sync
|
||||||
pub async fn remove_file_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
pub async fn rename_with_sync<P, Q>(from: P, to: Q) -> Result<(), IoError>
|
||||||
```
|
|
||||||
|
/// Delete with directory sync
|
||||||
Both complete the operation itself, then update and sync the parent directory entry.
|
pub async fn remove_file_with_sync<P: AsRef<Path>>(path: P) -> Result<(), IoError>
|
||||||
|
```
|
||||||
---
|
|
||||||
|
### fsync Guarantees
|
||||||
## Implementation Details
|
|
||||||
|
- `sync_all()` on written files ensures data and metadata reach the physical storage device
|
||||||
### fsync on Files
|
- Directory entries are synced after create/rename/delete operations
|
||||||
|
- Prevents data loss during crashes or power failures between OS buffer flush and disk write
|
||||||
```rust
|
|
||||||
// Write file content
|
---
|
||||||
file.write_all(contents).await?;
|
|
||||||
|
## Transaction Flow: File Upload
|
||||||
// Ensure data is synced to disk
|
|
||||||
file.flush().await?;
|
```
|
||||||
file.sync_all().await?;
|
1. DedupService.store_bytes(content)
|
||||||
```
|
→ Compute SHA-256 hash
|
||||||
|
→ Check if blob exists (dedup hit → increment ref, return hash)
|
||||||
`sync_all()` instructs the OS to flush data and metadata to the physical storage device.
|
→ Write to .blobs/{prefix}/{hash}.blob.tmp
|
||||||
|
→ fsync + rename → .blobs/{prefix}/{hash}.blob
|
||||||
### fsync on Directories
|
|
||||||
|
2. FileBlobWriteRepository.save_file()
|
||||||
```rust
|
→ BEGIN TRANSACTION
|
||||||
// Sync a directory to ensure its contents (entries) are durable
|
→ INSERT INTO storage.files (name, folder_id, blob_hash, size, ...)
|
||||||
async fn sync_directory<P: AsRef<Path>>(path: P) -> Result<(), IoError> {
|
→ COMMIT
|
||||||
let dir_file = OpenOptions::new().read(true).open(path).await?;
|
```
|
||||||
dir_file.sync_all().await
|
|
||||||
}
|
If step 1 fails, no metadata is written. If step 2 fails, the blob exists but is unreferenced (cleaned up by garbage collection). Data is never in an inconsistent state.
|
||||||
```
|
|
||||||
|
## Transaction Flow: File Deletion
|
||||||
Required after any operation that modifies directory entries (create, rename, delete).
|
|
||||||
|
```
|
||||||
---
|
1. FileBlobWriteRepository.delete_file_permanently()
|
||||||
|
→ BEGIN TRANSACTION
|
||||||
## Usage in the Codebase
|
→ DELETE FROM storage.files WHERE id = $1 (captures blob_hash first)
|
||||||
|
→ COMMIT
|
||||||
### File Write Repository
|
|
||||||
|
2. DedupService.decrement_ref(blob_hash)
|
||||||
```rust
|
→ Decrement reference counter
|
||||||
// Write the file to disk using atomic write with fsync
|
→ If counter reaches 0, delete the blob file
|
||||||
tokio::time::timeout(
|
```
|
||||||
self.config.timeouts.file_write_timeout(),
|
|
||||||
FileSystemUtils::atomic_write(&abs_path, &content)
|
If step 2 fails, an unreferenced blob may remain on disk (occupies space but is not a correctness issue). Future garbage collection can clean these up.
|
||||||
).await
|
|
||||||
```
|
---
|
||||||
|
|
||||||
### File Move Operations
|
## Benefits
|
||||||
|
|
||||||
```rust
|
1. **ACID transactions** — metadata operations are atomic, consistent, isolated, and durable
|
||||||
// Move the file physically with fsync
|
2. **Content-addressable storage** — identical content is stored once, referenced by hash
|
||||||
time::timeout(
|
3. **Crash resilience** — atomic blob writes + PostgreSQL WAL ensure recovery
|
||||||
self.config.timeouts.file_timeout(),
|
4. **No partial writes** — temp file + rename pattern guarantees all-or-nothing
|
||||||
FileSystemUtils::rename_with_sync(&old_abs_path, &new_abs_path)
|
5. **Referential integrity** — foreign keys prevent orphaned metadata
|
||||||
).await
|
|
||||||
```
|
---
|
||||||
|
|
||||||
### Directory Creation
|
## Performance Considerations
|
||||||
|
|
||||||
```rust
|
- PostgreSQL connection pooling (`sqlx::PgPool`) amortizes connection overhead
|
||||||
// Ensure the parent directory exists with proper syncing
|
- Dedup hash computation is CPU-bound but avoids unnecessary disk writes for duplicate content
|
||||||
self.ensure_parent_directory(&abs_path).await?;
|
- Blob fsync adds latency vs. buffered writes, but ensures durability for critical user data
|
||||||
|
- Content cache (in-memory LRU) serves repeat reads without disk or DB access
|
||||||
// Implementation uses FileSystemUtils
|
|
||||||
async fn ensure_parent_directory(&self, abs_path: &PathBuf) -> FileRepositoryResult<()> {
|
|
||||||
if let Some(parent) = abs_path.parent() {
|
|
||||||
time::timeout(
|
|
||||||
self.config.timeouts.dir_timeout(),
|
|
||||||
FileSystemUtils::create_dir_with_sync(parent)
|
|
||||||
).await
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Benefits
|
|
||||||
|
|
||||||
1. **Data durability** -- critical data is synced to persistent storage
|
|
||||||
2. **Crash resilience** -- recovery from unexpected failures without data loss
|
|
||||||
3. **Consistency** -- file operations maintain a consistent filesystem state
|
|
||||||
4. **Atomic operations** -- file writes appear as all-or-nothing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Performance Considerations
|
|
||||||
|
|
||||||
Syncing to disk costs more than buffered writes. OxiCloud mitigates this by:
|
|
||||||
|
|
||||||
1. Applying these measures only to critical operations
|
|
||||||
2. Using timeouts to prevent indefinite blocking
|
|
||||||
3. Implementing parallel processing for large files
|
|
||||||
|
|
||||||
The tradeoff favors safety for critical data while maintaining good performance for most operations.
|
|
||||||
|
|||||||
@@ -462,7 +462,7 @@ src/
|
|||||||
│ └── delta_sync_handler.rs # NUEVO: Endpoints API
|
│ └── delta_sync_handler.rs # NUEVO: Endpoints API
|
||||||
│
|
│
|
||||||
└── common/
|
└── common/
|
||||||
└── di.rs # Añadir: delta_sync_service a CoreServices
|
└── di.rs # Añadir: delta_sync_service a AppState
|
||||||
```
|
```
|
||||||
|
|
||||||
### Main service (delta_sync_service.rs)
|
### Main service (delta_sync_service.rs)
|
||||||
@@ -1133,7 +1133,7 @@ thiserror = "1.0" # Para errores tipados (probablemente ya existe)
|
|||||||
- [ ] Implement **generate_delta()**
|
- [ ] Implement **generate_delta()**
|
||||||
- [ ] Implement **apply_delta()**
|
- [ ] Implement **apply_delta()**
|
||||||
- [ ] Create handler and API endpoints
|
- [ ] Create handler and API endpoints
|
||||||
- [ ] Integrate into DI (**CoreServices**)
|
- [ ] Integrate into DI (**AppState**)
|
||||||
- [ ] Add routes in `routes.rs`
|
- [ ] Add routes in `routes.rs`
|
||||||
- [ ] Integrate with upload (automatic indexing)
|
- [ ] Integrate with upload (automatic indexing)
|
||||||
- [ ] Integrate with delete (signature cleanup)
|
- [ ] Integrate with delete (signature cleanup)
|
||||||
|
|||||||
+474
-477
@@ -1,477 +1,474 @@
|
|||||||
# 01 - Internal Architecture
|
# 01 - Internal Architecture
|
||||||
|
|
||||||
OxiCloud follows a **hexagonal (ports & adapters) architecture** organized in four layers:
|
OxiCloud follows a **hexagonal (ports & adapters) architecture** organized in four layers:
|
||||||
|
|
||||||
```
|
```
|
||||||
Domain → Application → Infrastructure → Interfaces
|
Domain → Application → Infrastructure → Interfaces
|
||||||
```
|
```
|
||||||
|
|
||||||
All cross-layer dependencies point inward via trait-based ports. The DI container (**AppServiceFactory**) wires concrete implementations at startup.
|
All cross-layer dependencies point inward via trait-based ports. The DI container (**AppServiceFactory**) wires concrete implementations at startup.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Dependency Injection Container
|
## Storage Model: 100% Blob Storage
|
||||||
|
|
||||||
### AppServiceFactory
|
OxiCloud uses a **100% blob storage model** where:
|
||||||
|
|
||||||
**File:** `src/common/di.rs`
|
- **File metadata** (name, folder, size, user, timestamps, trash status) is stored in **PostgreSQL** (`storage.files` table).
|
||||||
|
- **File content** is stored as content-addressed blobs via **DedupService** at `.blobs/{prefix}/{hash}.blob`.
|
||||||
```rust
|
- **Folder structure** is purely virtual — represented as rows in `storage.folders` (no filesystem directories per user).
|
||||||
pub struct AppServiceFactory {
|
- **Trash** is a soft-delete flag (`is_trashed`, `trashed_at`) on files and folders, exposed via `storage.trash_items` VIEW.
|
||||||
storage_path: PathBuf,
|
|
||||||
locales_path: PathBuf,
|
There are no filesystem-based ID mappings, no `folder_ids.json`/`file_ids.json`, and no storage mediator.
|
||||||
config: AppConfig,
|
|
||||||
}
|
---
|
||||||
```
|
|
||||||
|
## Dependency Injection Container
|
||||||
Initialization order in `build_app_state()`:
|
|
||||||
|
### AppServiceFactory
|
||||||
1. **Core services** -- path, caches, ID mapping, thumbnail, write-behind, chunked upload, transcode, dedup, compression
|
|
||||||
2. **Repository services** -- folder repo (stub mediator first), then **FileSystemStorageMediator** (real), file repos, metadata cache, buffer pool
|
**File:** `src/common/di.rs`
|
||||||
3. **Trash service** (if **enable_trash** enabled)
|
|
||||||
4. **Application services** -- folder, file upload/retrieval/management, search, i18n
|
```rust
|
||||||
5. **Share service** (if **enable_file_sharing** enabled)
|
pub struct AppServiceFactory {
|
||||||
6. **DB-dependent services** -- favorites, recent, storage usage, auth (via **auth_factory**)
|
storage_path: PathBuf,
|
||||||
7. **Preload** translations + metadata cache
|
locales_path: PathBuf,
|
||||||
8. **ZIP service** (needs file retrieval + folder service, wired last)
|
config: AppConfig,
|
||||||
9. **Assemble AppState** + admin settings + CalDAV/CardDAV
|
}
|
||||||
|
```
|
||||||
### AppState (Global State)
|
|
||||||
|
Initialization order in `build_app_state()`:
|
||||||
```rust
|
|
||||||
pub struct AppState {
|
1. **Core services** — path, content cache, thumbnail, chunked upload, transcode, dedup, compression
|
||||||
pub core: CoreServices,
|
2. **Repository services** — `FolderDbRepository`, `FileBlobReadRepository`, `FileBlobWriteRepository`, `TrashDbRepository` (all PgPool-backed)
|
||||||
pub repositories: RepositoryServices,
|
3. **Trash service** (if **enable_trash** enabled)
|
||||||
pub applications: ApplicationServices,
|
4. **Application services** — folder, file upload/retrieval/management, search, i18n
|
||||||
pub db_pool: Option<Arc<PgPool>>,
|
5. **Share service** (if **enable_file_sharing** enabled)
|
||||||
pub auth_service: Option<AuthServices>,
|
6. **DB-dependent services** — favorites, recent, storage usage, auth (via **auth_factory**)
|
||||||
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
7. **Preload** translations
|
||||||
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
8. **ZIP service** (needs file retrieval + folder service, wired last)
|
||||||
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
9. **Assemble AppState** + admin settings + CalDAV/CardDAV
|
||||||
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
|
||||||
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
### AppState (Global State)
|
||||||
pub storage_usage_service: Option<Arc<dyn StorageUsagePort>>,
|
|
||||||
pub calendar_service: Option<Arc<dyn StorageUseCase>>,
|
```rust
|
||||||
pub contact_service: Option<Arc<dyn StorageUseCase>>,
|
pub struct AppState {
|
||||||
pub calendar_use_case: Option<Arc<dyn CalendarUseCase>>,
|
pub core: CoreServices,
|
||||||
pub addressbook_use_case: Option<Arc<dyn AddressBookUseCase>>,
|
pub repositories: RepositoryServices,
|
||||||
pub contact_use_case: Option<Arc<dyn ContactUseCase>>,
|
pub applications: ApplicationServices,
|
||||||
}
|
pub db_pool: Option<Arc<PgPool>>,
|
||||||
```
|
pub auth_service: Option<AuthServices>,
|
||||||
|
pub admin_settings_service: Option<Arc<AdminSettingsService>>,
|
||||||
Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `with_trash_service()` → ... → `for_routing()`. The `Default` impl uses stubs from `crate::common::stubs`.
|
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
||||||
|
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
||||||
### Service Groups
|
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
||||||
|
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
||||||
```rust
|
pub storage_usage_service: Option<Arc<dyn StorageUsagePort>>,
|
||||||
pub struct CoreServices {
|
pub calendar_service: Option<Arc<dyn StorageUseCase>>,
|
||||||
pub path_service: Arc<PathService>,
|
pub contact_service: Option<Arc<dyn StorageUseCase>>,
|
||||||
pub file_content_cache: Arc<dyn ContentCachePort>,
|
pub calendar_use_case: Option<Arc<dyn CalendarUseCase>>,
|
||||||
pub id_mapping_service: Arc<dyn IdMappingPort>, // folder IDs
|
pub addressbook_use_case: Option<Arc<dyn AddressBookUseCase>>,
|
||||||
pub file_id_mapping_service: Arc<IdMappingService>, // file IDs (concrete)
|
pub contact_use_case: Option<Arc<dyn ContactUseCase>>,
|
||||||
pub id_mapping_optimizer: Arc<IdMappingOptimizer>,
|
}
|
||||||
pub thumbnail_service: Arc<dyn ThumbnailPort>,
|
```
|
||||||
pub write_behind_cache: Arc<dyn WriteBehindCachePort>,
|
|
||||||
pub chunked_upload_service: Arc<dyn ChunkedUploadPort>,
|
Builder pattern: `new()` → `with_database()` → `with_auth_services()` → `with_trash_service()` → ... → `for_routing()`. The `Default` impl uses stubs from `crate::common::stubs`.
|
||||||
pub image_transcode_service: Arc<dyn ImageTranscodePort>,
|
|
||||||
pub dedup_service: Arc<dyn DedupPort>,
|
### Service Groups
|
||||||
pub compression_service: Arc<dyn CompressionPort>,
|
|
||||||
pub zip_service: Arc<dyn ZipPort>,
|
```rust
|
||||||
pub config: AppConfig,
|
pub struct CoreServices {
|
||||||
}
|
pub path_service: Arc<PathService>,
|
||||||
|
pub file_content_cache: Arc<dyn ContentCachePort>,
|
||||||
pub struct RepositoryServices {
|
pub thumbnail_service: Arc<dyn ThumbnailPort>,
|
||||||
pub folder_repository: Arc<dyn FolderStoragePort>,
|
pub chunked_upload_service: Arc<dyn ChunkedUploadPort>,
|
||||||
pub file_read_repository: Arc<dyn FileReadPort>,
|
pub image_transcode_service: Arc<dyn ImageTranscodePort>,
|
||||||
pub file_write_repository: Arc<dyn FileWritePort>,
|
pub dedup_service: Arc<dyn DedupPort>,
|
||||||
pub i18n_repository: Arc<dyn I18nService>,
|
pub compression_service: Arc<dyn CompressionPort>,
|
||||||
pub storage_mediator: Arc<dyn StorageMediator>,
|
pub zip_service: Arc<dyn ZipPort>,
|
||||||
pub metadata_cache: Arc<FileMetadataCache>,
|
pub config: AppConfig,
|
||||||
pub trash_repository: Option<Arc<dyn TrashRepository>>,
|
}
|
||||||
}
|
|
||||||
|
pub struct RepositoryServices {
|
||||||
pub struct ApplicationServices {
|
pub folder_repository: Arc<dyn FolderStoragePort>,
|
||||||
pub folder_service_concrete: Arc<FolderService>,
|
pub folder_repo_concrete: Arc<FolderDbRepository>,
|
||||||
pub folder_service: Arc<dyn FolderUseCase>,
|
pub file_read_repository: Arc<dyn FileReadPort>,
|
||||||
pub file_upload_service: Arc<dyn FileUploadUseCase>,
|
pub file_write_repository: Arc<dyn FileWritePort>,
|
||||||
pub file_retrieval_service: Arc<dyn FileRetrievalUseCase>,
|
pub i18n_repository: Arc<dyn I18nService>,
|
||||||
pub file_management_service: Arc<dyn FileManagementUseCase>,
|
pub trash_repository: Option<Arc<dyn TrashRepository>>,
|
||||||
pub file_use_case_factory: Arc<dyn FileUseCaseFactory>,
|
}
|
||||||
pub i18n_service: Arc<I18nApplicationService>,
|
|
||||||
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
pub struct ApplicationServices {
|
||||||
pub search_service: Option<Arc<dyn SearchUseCase>>,
|
pub folder_service_concrete: Arc<FolderService>,
|
||||||
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
pub folder_service: Arc<dyn FolderUseCase>,
|
||||||
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
pub file_upload_service: Arc<dyn FileUploadUseCase>,
|
||||||
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
pub file_retrieval_service: Arc<dyn FileRetrievalUseCase>,
|
||||||
}
|
pub file_management_service: Arc<dyn FileManagementUseCase>,
|
||||||
|
pub file_use_case_factory: Arc<dyn FileUseCaseFactory>,
|
||||||
pub struct AuthServices {
|
pub i18n_service: Arc<I18nApplicationService>,
|
||||||
pub token_service: Arc<dyn TokenServicePort>,
|
pub trash_service: Option<Arc<dyn TrashUseCase>>,
|
||||||
pub auth_application_service: Arc<AuthApplicationService>,
|
pub search_service: Option<Arc<dyn SearchUseCase>>,
|
||||||
}
|
pub share_service: Option<Arc<dyn ShareUseCase>>,
|
||||||
```
|
pub favorites_service: Option<Arc<dyn FavoritesUseCase>>,
|
||||||
|
pub recent_service: Option<Arc<dyn RecentItemsUseCase>>,
|
||||||
---
|
}
|
||||||
|
|
||||||
## ID Mapping System
|
pub struct AuthServices {
|
||||||
|
pub token_service: Arc<dyn TokenServicePort>,
|
||||||
Maps bidirectionally between **filesystem StoragePaths** and **UUID identifiers**. Two separate instances exist: one for folders (`folder_ids.json`), one for files (`file_ids.json`).
|
pub auth_application_service: Arc<AuthApplicationService>,
|
||||||
|
}
|
||||||
### StoragePath (Domain Value Object)
|
```
|
||||||
|
|
||||||
**File:** `src/domain/services/path_service.rs`
|
---
|
||||||
|
|
||||||
```rust
|
## Database Schema (Storage)
|
||||||
#[derive(Debug, Clone, PartialEq, Eq, Default)]
|
|
||||||
pub struct StoragePath {
|
All file and folder metadata lives in the `storage` PostgreSQL schema:
|
||||||
segments: Vec<String>, // e.g., ["Mi Carpeta - admin", "file.txt"]
|
|
||||||
}
|
```sql
|
||||||
```
|
CREATE SCHEMA IF NOT EXISTS storage;
|
||||||
|
|
||||||
| Method | Description |
|
CREATE TABLE storage.folders (
|
||||||
|---|---|
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||||
| `root()` | Empty path (storage root) |
|
name TEXT NOT NULL,
|
||||||
| `from_string(path)` | Parse from `/`-delimited string |
|
parent_id UUID REFERENCES storage.folders(id) ON DELETE CASCADE,
|
||||||
| `join(segment)` | Append a segment |
|
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id),
|
||||||
| `file_name()` | Last segment |
|
is_trashed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||||
| `parent()` | All segments except last |
|
trashed_at TIMESTAMPTZ,
|
||||||
| `to_string()` | Join segments with `/` |
|
original_parent_id UUID,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
### IdMappingPort (Application Port)
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
|
);
|
||||||
**File:** `src/application/ports/outbound.rs`
|
|
||||||
|
CREATE TABLE storage.files (
|
||||||
```rust
|
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||||
#[async_trait]
|
name TEXT NOT NULL,
|
||||||
pub trait IdMappingPort: Send + Sync + 'static {
|
folder_id UUID NOT NULL REFERENCES storage.folders(id) ON DELETE CASCADE,
|
||||||
async fn get_or_create_id(&self, path: &StoragePath) -> Result<String, DomainError>;
|
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id),
|
||||||
async fn get_path_by_id(&self, id: &str) -> Result<StoragePath, DomainError>;
|
blob_hash TEXT NOT NULL,
|
||||||
async fn update_path(&self, id: &str, new_path: &StoragePath) -> Result<(), DomainError>;
|
size BIGINT NOT NULL DEFAULT 0,
|
||||||
async fn remove_id(&self, id: &str) -> Result<(), DomainError>;
|
mime_type TEXT NOT NULL DEFAULT 'application/octet-stream',
|
||||||
async fn save_changes(&self) -> Result<(), DomainError>;
|
is_trashed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||||
// Default impls for PathBuf variants:
|
trashed_at TIMESTAMPTZ,
|
||||||
async fn get_file_path(&self, file_id: &str) -> Result<PathBuf, DomainError>;
|
original_folder_id UUID,
|
||||||
async fn update_file_path(&self, file_id: &str, new_path: &PathBuf) -> Result<(), DomainError>;
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
}
|
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||||
```
|
);
|
||||||
|
|
||||||
### IdMappingService (Base Implementation)
|
CREATE OR REPLACE VIEW storage.trash_items AS
|
||||||
|
SELECT id, name, 'file' AS item_type, folder_id AS parent_id,
|
||||||
**File:** `src/infrastructure/services/id_mapping_service.rs`
|
user_id, size, mime_type, trashed_at, created_at
|
||||||
|
FROM storage.files WHERE is_trashed = TRUE
|
||||||
```rust
|
UNION ALL
|
||||||
pub struct IdMappingService {
|
SELECT id, name, 'folder' AS item_type, parent_id,
|
||||||
map_path: PathBuf, // e.g., storage/file_ids.json
|
user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at
|
||||||
id_map: RwLock<IdMap>,
|
FROM storage.folders WHERE is_trashed = TRUE;
|
||||||
save_mutex: Mutex<()>,
|
```
|
||||||
timeouts: TimeoutConfig,
|
|
||||||
pending_save: RwLock<bool>,
|
---
|
||||||
}
|
|
||||||
|
## Repository Layer (Infrastructure)
|
||||||
struct IdMap {
|
|
||||||
path_to_id: HashMap<String, String>,
|
All repositories use **PgPool** for metadata and **DedupService** for blob content.
|
||||||
id_to_path: HashMap<String, String>,
|
|
||||||
version: u32,
|
### FolderDbRepository
|
||||||
}
|
|
||||||
```
|
**File:** `src/infrastructure/repositories/pg/folder_db_repository.rs`
|
||||||
|
|
||||||
Operations:
|
```rust
|
||||||
- `get_or_create_id(path)` -- Read-lock first (cache hit). Write-lock on miss, generates `Uuid::new_v4()`.
|
pub struct FolderDbRepository {
|
||||||
- `save_pending_changes()` -- Atomic write: serialize → write `.tmp` file → rename over original (with retry).
|
pool: Option<Arc<PgPool>>,
|
||||||
- `new(map_path)` -- Loads from JSON, rebuilds inverse map if inconsistent.
|
}
|
||||||
|
```
|
||||||
Persistence format (`storage/file_ids.json`):
|
|
||||||
```json
|
Implements `FolderRepository`. Uses recursive CTEs for path building, unique constraints for name dedup within parent, and soft-delete flags for trash operations.
|
||||||
{
|
|
||||||
"path_to_id": { "/Mi Carpeta - admin/doc.pdf": "a1b2c3d4-..." },
|
Key methods: `create_folder`, `get_folder`, `get_folder_by_path`, `list_folders`, `rename_folder`, `move_folder`, `delete_folder`, `move_to_trash`, `restore_from_trash`, `create_home_folder`, `get_folder_user_id`.
|
||||||
"id_to_path": { "a1b2c3d4-...": "/Mi Carpeta - admin/doc.pdf" },
|
|
||||||
"version": 42
|
`new_stub()` creates a pool-less instance for `AppState::default()`.
|
||||||
}
|
|
||||||
```
|
### FileBlobReadRepository
|
||||||
|
|
||||||
### IdMappingOptimizer (Cache Layer)
|
**File:** `src/infrastructure/repositories/pg/file_blob_read_repository.rs`
|
||||||
|
|
||||||
**File:** `src/infrastructure/services/id_mapping_optimizer.rs`
|
```rust
|
||||||
|
pub struct FileBlobReadRepository {
|
||||||
Wraps **IdMappingService** with an in-memory TTL cache:
|
pool: Arc<PgPool>,
|
||||||
|
dedup: Arc<dyn DedupPort>,
|
||||||
| Parameter | Value |
|
folder_repo: Arc<FolderDbRepository>,
|
||||||
|---|---|
|
}
|
||||||
| Max cache entries | 10,000 |
|
```
|
||||||
| TTL | 300 s (5 min) |
|
|
||||||
| Cleanup interval | 150 s (2.5 min) |
|
Implements `FileReadPort`. Reads metadata from `storage.files` and content from blob store via `dedup.read_blob()` / `read_blob_bytes()`.
|
||||||
| Batch threshold | ≥ 20 queued items |
|
|
||||||
| Max concurrent batches | 2 (semaphore) |
|
Key methods: `get_file`, `list_files`, `get_file_content`, `get_file_stream`, `get_file_range_stream`, `get_file_mmap`, `get_file_path`, `get_parent_folder_id`.
|
||||||
|
|
||||||
```rust
|
### FileBlobWriteRepository
|
||||||
pub struct IdMappingOptimizer {
|
|
||||||
base_service: Arc<IdMappingService>,
|
**File:** `src/infrastructure/repositories/pg/file_blob_write_repository.rs`
|
||||||
path_to_id_cache: RwLock<HashMap<String, (String, Instant)>>,
|
|
||||||
id_to_path_cache: RwLock<HashMap<String, (String, Instant)>>,
|
```rust
|
||||||
stats: RwLock<OptimizerStats>,
|
pub struct FileBlobWriteRepository {
|
||||||
batch_limiter: Semaphore,
|
pool: Arc<PgPool>,
|
||||||
pending_batch: Mutex<BatchQueue>,
|
dedup: Arc<dyn DedupPort>,
|
||||||
}
|
folder_repo: Arc<FolderDbRepository>,
|
||||||
```
|
}
|
||||||
|
```
|
||||||
Lookup flow: check cache → if miss, queue request → trigger batch if ≥ 20 pending → fallback to **base_service** → update cache.
|
|
||||||
|
Implements `FileWritePort`. Stores content via `dedup.store_bytes()` (returns hash), then INSERTs metadata into `storage.files`.
|
||||||
On `update_path` / `remove_id`, cache entries are invalidated first, then delegated.
|
|
||||||
|
Key methods: `save_file`, `save_file_from_stream`, `move_file`, `rename_file`, `delete_file`, `update_file_content`, `move_to_trash`, `restore_from_trash`, `delete_file_permanently`.
|
||||||
Used only for folder ID mapping. File ID mapping uses the base **IdMappingService** directly.
|
|
||||||
|
### TrashDbRepository
|
||||||
---
|
|
||||||
|
**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs`
|
||||||
## Path Service
|
|
||||||
|
```rust
|
||||||
**File:** `src/infrastructure/services/path_service.rs`
|
pub struct TrashDbRepository {
|
||||||
|
pool: Arc<PgPool>,
|
||||||
```rust
|
retention_days: u32,
|
||||||
pub struct PathService {
|
}
|
||||||
root_path: PathBuf, // e.g., ./storage
|
```
|
||||||
}
|
|
||||||
```
|
Implements `TrashRepository`. Reads from `storage.trash_items` VIEW. `clear_trash` DELETEs rows where `is_trashed = TRUE`. `get_expired_items` checks `trashed_at` against the configured retention period.
|
||||||
|
|
||||||
### Path Resolution
|
---
|
||||||
|
|
||||||
| Method | Description |
|
## Path Service
|
||||||
|---|---|
|
|
||||||
| `resolve_path(storage_path)` | Appends **StoragePath** segments to **root_path** → absolute `PathBuf` |
|
**File:** `src/infrastructure/services/path_service.rs`
|
||||||
| `to_storage_path(physical_path)` | Strips **root_path** prefix → **StoragePath** (returns `None` if outside root) |
|
|
||||||
| `create_file_path(folder, name)` | Combines folder path + filename |
|
```rust
|
||||||
| `is_direct_child(parent, child)` | Check parent-child relationship |
|
pub struct PathService {
|
||||||
| `is_in_root(path)` | Verify path is within storage root |
|
root_path: PathBuf, // e.g., ./storage
|
||||||
|
}
|
||||||
### Path Validation
|
```
|
||||||
|
|
||||||
`validate_path(path)` rejects:
|
Used for resolving storage root paths (blob storage directory, thumbnail paths, etc.). Not used for per-user folder resolution — that is handled by `FolderDbRepository` via PostgreSQL.
|
||||||
- Empty path segments
|
|
||||||
- Segments containing dangerous characters: `\`, `:`, `*`, `?`, `"`, `<`, `>`, `|`
|
### StoragePath (Domain Value Object)
|
||||||
- Segments starting with `.` (exception: `.well-known` for WebDAV/CalDAV/CardDAV)
|
|
||||||
|
**File:** `src/domain/services/path_service.rs`
|
||||||
### Trait Implementations
|
|
||||||
|
```rust
|
||||||
- **StoragePort** -- `resolve_path()`, `ensure_directory()` (validates first, then `fs::create_dir_all`), `file_exists()`, `directory_exists()`
|
#[derive(Debug, Clone, PartialEq, Eq, Default)]
|
||||||
- **StorageMediator** -- Simplified stub variant. Folder lookup methods return `NotFound`.
|
pub struct StoragePath {
|
||||||
|
segments: Vec<String>,
|
||||||
---
|
}
|
||||||
|
```
|
||||||
## Storage Mediator
|
|
||||||
|
| Method | Description |
|
||||||
Bridges folder IDs to filesystem paths by combining folder repository, path service, and ID mapping.
|
|---|---|
|
||||||
|
| `root()` | Empty path (storage root) |
|
||||||
### StorageMediator Trait
|
| `from_string(path)` | Parse from `/`-delimited string |
|
||||||
|
| `join(segment)` | Append a segment |
|
||||||
**File:** `src/application/services/storage_mediator.rs`
|
| `file_name()` | Last segment |
|
||||||
|
| `parent()` | All segments except last |
|
||||||
```rust
|
| `to_string()` | Join segments with `/` |
|
||||||
#[async_trait]
|
|
||||||
pub trait StorageMediator: Send + Sync + 'static {
|
### Trait Implementations
|
||||||
async fn get_folder_path(&self, folder_id: &str) -> StorageMediatorResult<PathBuf>;
|
|
||||||
async fn get_folder_storage_path(&self, folder_id: &str) -> StorageMediatorResult<StoragePath>;
|
- **StoragePort** — `resolve_path()`, `ensure_directory()`, `file_exists()`, `directory_exists()`
|
||||||
async fn get_folder(&self, folder_id: &str) -> StorageMediatorResult<Folder>;
|
|
||||||
async fn file_exists_at_path(&self, path: &Path) -> StorageMediatorResult<bool>;
|
---
|
||||||
async fn file_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult<bool>;
|
|
||||||
async fn folder_exists_at_path(&self, path: &Path) -> StorageMediatorResult<bool>;
|
## Session Management
|
||||||
async fn folder_exists_at_storage_path(&self, storage_path: &StoragePath) -> StorageMediatorResult<bool>;
|
|
||||||
fn resolve_path(&self, relative_path: &Path) -> PathBuf;
|
### Session Entity
|
||||||
fn resolve_storage_path(&self, storage_path: &StoragePath) -> PathBuf;
|
|
||||||
async fn ensure_directory(&self, path: &Path) -> StorageMediatorResult<()>;
|
**File:** `src/domain/entities/session.rs`
|
||||||
async fn ensure_storage_directory(&self, storage_path: &StoragePath) -> StorageMediatorResult<()>;
|
|
||||||
}
|
```rust
|
||||||
```
|
pub struct Session {
|
||||||
|
id: String, // UUID v4
|
||||||
### FileSystemStorageMediator
|
user_id: String,
|
||||||
|
refresh_token: String,
|
||||||
```rust
|
expires_at: DateTime<Utc>,
|
||||||
pub struct FileSystemStorageMediator {
|
ip_address: Option<String>,
|
||||||
pub folder_storage_port: Arc<dyn FolderStoragePort>,
|
user_agent: Option<String>,
|
||||||
pub path_service: Arc<dyn StoragePort>,
|
created_at: DateTime<Utc>,
|
||||||
pub id_mapping: Arc<dyn IdMappingPort>,
|
revoked: bool,
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Folder ID → filesystem path resolution:
|
Constructors:
|
||||||
|
- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` — generates UUID, panics if **user_id** or **refresh_token** empty
|
||||||
```
|
- `Session::from_raw(...)` — for DB reconstruction
|
||||||
folder_id ──► FolderStoragePort.get_folder(id)
|
|
||||||
──► Folder entity
|
### SessionRepository (Domain Port)
|
||||||
──► IdMappingPort.get_path_by_id(folder.id())
|
|
||||||
──► StoragePath
|
**File:** `src/domain/repositories/session_repository.rs`
|
||||||
──► StoragePort.resolve_path(storage_path)
|
|
||||||
──► PathBuf (absolute)
|
```rust
|
||||||
```
|
#[async_trait]
|
||||||
|
pub trait SessionRepository: Send + Sync + 'static {
|
||||||
A **StubStorageMediator** also exists (returns `/tmp` paths) for DI bootstrap before the real mediator is available.
|
async fn create_session(&self, session: Session) -> SessionRepositoryResult<Session>;
|
||||||
|
async fn get_session_by_id(&self, id: &str) -> SessionRepositoryResult<Session>;
|
||||||
---
|
async fn get_session_by_refresh_token(&self, token: &str) -> SessionRepositoryResult<Session>;
|
||||||
|
async fn get_sessions_by_user_id(&self, user_id: &str) -> SessionRepositoryResult<Vec<Session>>;
|
||||||
## Session Management
|
async fn revoke_session(&self, session_id: &str) -> SessionRepositoryResult<()>;
|
||||||
|
async fn revoke_all_user_sessions(&self, user_id: &str) -> SessionRepositoryResult<u64>;
|
||||||
### Session Entity
|
async fn delete_expired_sessions(&self) -> SessionRepositoryResult<u64>;
|
||||||
|
}
|
||||||
**File:** `src/domain/entities/session.rs`
|
```
|
||||||
|
|
||||||
```rust
|
### SessionStoragePort (Application Port)
|
||||||
pub struct Session {
|
|
||||||
id: String, // UUID v4
|
**File:** `src/application/ports/auth_ports.rs`
|
||||||
user_id: String,
|
|
||||||
refresh_token: String,
|
```rust
|
||||||
expires_at: DateTime<Utc>,
|
#[async_trait]
|
||||||
ip_address: Option<String>,
|
pub trait SessionStoragePort: Send + Sync + 'static {
|
||||||
user_agent: Option<String>,
|
async fn create_session(&self, session: Session) -> Result<Session, DomainError>;
|
||||||
created_at: DateTime<Utc>,
|
async fn get_session_by_refresh_token(&self, token: &str) -> Result<Session, DomainError>;
|
||||||
revoked: bool,
|
async fn revoke_session(&self, session_id: &str) -> Result<(), DomainError>;
|
||||||
}
|
async fn revoke_all_user_sessions(&self, user_id: &str) -> Result<u64, DomainError>;
|
||||||
```
|
}
|
||||||
|
```
|
||||||
Constructors:
|
|
||||||
- `Session::new(user_id, refresh_token, ip_address, user_agent, expires_in_days)` -- generates UUID, panics if **user_id** or **refresh_token** empty
|
### SessionPgRepository (Infrastructure)
|
||||||
- `Session::from_raw(...)` -- for DB reconstruction
|
|
||||||
|
**File:** `src/infrastructure/repositories/pg/session_pg_repository.rs`
|
||||||
### SessionRepository (Domain Port)
|
|
||||||
|
```rust
|
||||||
**File:** `src/domain/repositories/session_repository.rs`
|
pub struct SessionPgRepository {
|
||||||
|
pool: Arc<PgPool>,
|
||||||
```rust
|
}
|
||||||
#[async_trait]
|
```
|
||||||
pub trait SessionRepository: Send + Sync + 'static {
|
|
||||||
async fn create_session(&self, session: Session) -> SessionRepositoryResult<Session>;
|
Implements both **SessionRepository** and **SessionStoragePort**. Uses `with_transaction()` helper for write operations. `create_session` also updates `auth.users.last_login_at` within the same transaction.
|
||||||
async fn get_session_by_id(&self, id: &str) -> SessionRepositoryResult<Session>;
|
|
||||||
async fn get_session_by_refresh_token(&self, token: &str) -> SessionRepositoryResult<Session>;
|
### Database Schema
|
||||||
async fn get_sessions_by_user_id(&self, user_id: &str) -> SessionRepositoryResult<Vec<Session>>;
|
|
||||||
async fn revoke_session(&self, session_id: &str) -> SessionRepositoryResult<()>;
|
```sql
|
||||||
async fn revoke_all_user_sessions(&self, user_id: &str) -> SessionRepositoryResult<u64>;
|
CREATE TABLE IF NOT EXISTS auth.sessions (
|
||||||
async fn delete_expired_sessions(&self) -> SessionRepositoryResult<u64>;
|
id VARCHAR(36) PRIMARY KEY,
|
||||||
}
|
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
|
||||||
```
|
refresh_token TEXT NOT NULL UNIQUE,
|
||||||
|
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
||||||
### SessionStoragePort (Application Port)
|
ip_address TEXT,
|
||||||
|
user_agent TEXT,
|
||||||
**File:** `src/application/ports/auth_ports.rs`
|
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
revoked BOOLEAN NOT NULL DEFAULT FALSE
|
||||||
```rust
|
);
|
||||||
#[async_trait]
|
|
||||||
pub trait SessionStoragePort: Send + Sync + 'static {
|
CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id);
|
||||||
async fn create_session(&self, session: Session) -> Result<Session, DomainError>;
|
CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token);
|
||||||
async fn get_session_by_refresh_token(&self, token: &str) -> Result<Session, DomainError>;
|
CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at);
|
||||||
async fn revoke_session(&self, session_id: &str) -> Result<(), DomainError>;
|
CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked)
|
||||||
async fn revoke_all_user_sessions(&self, user_id: &str) -> Result<u64, DomainError>;
|
WHERE NOT revoked AND is_session_active(expires_at);
|
||||||
}
|
```
|
||||||
```
|
|
||||||
|
### Auth Service
|
||||||
### SessionPgRepository (Infrastructure)
|
|
||||||
|
**File:** `src/application/services/auth_application_service.rs`
|
||||||
**File:** `src/infrastructure/repositories/pg/session_pg_repository.rs`
|
|
||||||
|
**AuthApplicationService** orchestrates authentication using:
|
||||||
```rust
|
- **UserStoragePort** — user CRUD
|
||||||
pub struct SessionPgRepository {
|
- **SessionStoragePort** — session lifecycle
|
||||||
pool: Arc<PgPool>,
|
- **PasswordHasherPort** — Argon2id hashing
|
||||||
}
|
- **TokenServicePort** — JWT generation/validation
|
||||||
```
|
- `RwLock<OidcState>` — hot-reloadable OIDC configuration
|
||||||
|
- `Mutex<HashMap<String, PendingOidcFlow>>` — in-flight OIDC login states
|
||||||
Implements both **SessionRepository** and **SessionStoragePort**. Uses `with_transaction()` helper for write operations. `create_session` also updates `auth.users.last_login_at` within the same transaction.
|
|
||||||
|
Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**.
|
||||||
### Database Schema
|
|
||||||
|
---
|
||||||
```sql
|
|
||||||
CREATE TABLE IF NOT EXISTS auth.sessions (
|
## File Use Case Factory
|
||||||
id VARCHAR(36) PRIMARY KEY,
|
|
||||||
user_id VARCHAR(36) NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE,
|
**File:** `src/application/services/file_use_case_factory.rs`
|
||||||
refresh_token TEXT NOT NULL UNIQUE,
|
|
||||||
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
|
```rust
|
||||||
ip_address TEXT,
|
pub trait FileUseCaseFactory: Send + Sync + 'static {
|
||||||
user_agent TEXT,
|
fn create_file_upload_use_case(&self) -> Arc<dyn FileUploadUseCase>;
|
||||||
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
fn create_file_retrieval_use_case(&self) -> Arc<dyn FileRetrievalUseCase>;
|
||||||
revoked BOOLEAN NOT NULL DEFAULT FALSE
|
fn create_file_management_use_case(&self) -> Arc<dyn FileManagementUseCase>;
|
||||||
);
|
}
|
||||||
|
```
|
||||||
-- Indexes
|
|
||||||
CREATE INDEX idx_sessions_user_id ON auth.sessions(user_id);
|
**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**.
|
||||||
CREATE INDEX idx_sessions_refresh_token ON auth.sessions(refresh_token);
|
|
||||||
CREATE INDEX idx_sessions_expires_at ON auth.sessions(expires_at);
|
### File Operation Port Hierarchy
|
||||||
CREATE INDEX idx_sessions_active ON auth.sessions(user_id, revoked)
|
|
||||||
WHERE NOT revoked AND is_session_active(expires_at);
|
| Port | Key Methods |
|
||||||
```
|
|---|---|
|
||||||
|
| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `Buffered` <1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` |
|
||||||
### Auth Service
|
| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` |
|
||||||
|
| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) |
|
||||||
**File:** `src/application/services/auth_application_service.rs`
|
|
||||||
|
---
|
||||||
**AuthApplicationService** orchestrates authentication using:
|
|
||||||
- **UserStoragePort** -- user CRUD
|
## Architecture Diagram
|
||||||
- **SessionStoragePort** -- session lifecycle
|
|
||||||
- **PasswordHasherPort** -- Argon2id hashing
|
```
|
||||||
- **TokenServicePort** -- JWT generation/validation
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
- `RwLock<OidcState>` -- hot-reloadable OIDC configuration
|
│ Interfaces Layer │
|
||||||
- `Mutex<HashMap<String, PendingOidcFlow>>` -- in-flight OIDC login states
|
│ Axum Router → API Routes + Middleware (Auth, Compress) │
|
||||||
|
└─────────────────────┬───────────────────────────────────────┘
|
||||||
Wired by `auth_factory.rs`: **UserPgRepository** + **SessionPgRepository** + **Argon2PasswordHasher** + **JwtTokenService** → **AuthApplicationService**.
|
│ Arc<AppState>
|
||||||
|
┌─────────────────────▼───────────────────────────────────────┐
|
||||||
---
|
│ Application Layer │
|
||||||
|
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │
|
||||||
## File Use Case Factory
|
│ │ FileUpload │ │ FolderService│ │ AuthApplication │ │
|
||||||
|
│ │ FileRetrieval│ │ SearchService│ │ AdminSettings │ │
|
||||||
**File:** `src/application/services/file_use_case_factory.rs`
|
│ │ FileMgmt │ │ I18nService │ │ TrashService │ │
|
||||||
|
│ └──────┬───────┘ └──────┬───────┘ └──────────┬──────────┘ │
|
||||||
```rust
|
│ │ Ports (traits) │ │ │
|
||||||
pub trait FileUseCaseFactory: Send + Sync + 'static {
|
└─────────┼────────────────┼─────────────────────┼────────────┘
|
||||||
fn create_file_upload_use_case(&self) -> Arc<dyn FileUploadUseCase>;
|
│ │ │
|
||||||
fn create_file_retrieval_use_case(&self) -> Arc<dyn FileRetrievalUseCase>;
|
┌─────────▼────────────────▼─────────────────────▼────────────┐
|
||||||
fn create_file_management_use_case(&self) -> Arc<dyn FileManagementUseCase>;
|
│ Infrastructure Layer │
|
||||||
}
|
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||||
```
|
│ │ FileBlobRead │ │ PathService │ │ SessionPg │ │
|
||||||
|
│ │ FileBlobWrite │ │ DedupService │ │ UserPg │ │
|
||||||
**AppFileUseCaseFactory** creates lightweight service instances with only **FileReadPort** / **FileWritePort**. The main DI-wired services use `*_full()` constructors that inject write-behind cache, dedup, content cache, and transcode ports for full optimization.
|
│ │ FolderDb │ │ Thumbnail │ │ JwtTokenService │ │
|
||||||
|
│ │ TrashDb │ │ Transcode │ │ Argon2Hasher │ │
|
||||||
### File Operation Port Hierarchy
|
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
||||||
|
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||||
| Port | Key Methods |
|
│ │ ContentCache │ │ Compression │ │ ChunkedUpload │ │
|
||||||
|---|---|
|
│ │ BufferPool │ │ ZipService │ │ ShareFsRepo │ │
|
||||||
| **FileUploadUseCase** | `upload_file()`, `smart_upload()` (returns **UploadStrategy**: `WriteBehind` <256KB, `Buffered` 256KB-1MB, `Streaming` ≥1MB), `create_file()`, `update_file()` |
|
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
||||||
| **FileRetrievalUseCase** | `get_file()`, `get_file_content()`, `get_file_stream()`, `get_file_optimized()` (write-behind → content-cache → WebP transcode → mmap → streaming), `get_file_range_stream()` |
|
└─────────────────────────────────────────────────────────────┘
|
||||||
| **FileManagementUseCase** | `move_file()`, `rename_file()`, `delete_file()`, `delete_with_cleanup()` (trash-first with dedup reference cleanup) |
|
│
|
||||||
|
┌─────────────────────▼───────────────────────────────────────┐
|
||||||
---
|
│ Domain Layer │
|
||||||
|
│ Entities: File, Folder, Session, User, Calendar, Contact │
|
||||||
## Architecture Diagram
|
│ Value Objects: StoragePath │
|
||||||
|
│ Repository Traits: FolderRepository, TrashRepository, ... │
|
||||||
```
|
│ Domain Errors │
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
└─────────────────────────────────────────────────────────────┘
|
||||||
│ Interfaces Layer │
|
```
|
||||||
│ Axum Router → API Routes + Middleware (Auth, Compress) │
|
|
||||||
└─────────────────────┬───────────────────────────────────────┘
|
### Data Flow: File Upload
|
||||||
│ Arc<AppState>
|
|
||||||
┌─────────────────────▼───────────────────────────────────────┐
|
```
|
||||||
│ Application Layer │
|
HTTP Request (multipart)
|
||||||
│ ┌──────────────┐ ┌──────────────┐ ┌─────────────────────┐ │
|
→ FileUploadService.smart_upload()
|
||||||
│ │ FileUpload │ │ FolderService│ │ AuthApplication │ │
|
→ FileBlobWriteRepository.save_file() / save_file_from_stream()
|
||||||
│ │ FileRetrieval│ │ SearchService│ │ AdminSettings │ │
|
→ DedupService.store_bytes() → .blobs/{prefix}/{hash}.blob
|
||||||
│ │ FileMgmt │ │ I18nService │ │ TrashService │ │
|
→ INSERT INTO storage.files (name, folder_id, blob_hash, size, ...)
|
||||||
│ └──────┬───────┘ └──────┬───────┘ └──────────┬──────────┘ │
|
→ 201 Created (FileDto)
|
||||||
│ │ Ports (traits) │ │ │
|
```
|
||||||
└─────────┼────────────────┼─────────────────────┼────────────┘
|
|
||||||
│ │ │
|
### Data Flow: File Download
|
||||||
┌─────────▼────────────────▼─────────────────────▼────────────┐
|
|
||||||
│ Infrastructure Layer │
|
```
|
||||||
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
HTTP Request (GET /api/files/{id}/download)
|
||||||
│ │ FileFsRead/ │ │ IdMapping │ │ SessionPg │ │
|
→ FileRetrievalService.get_file_optimized()
|
||||||
│ │ FileFsWrite │ │ + Optimizer │ │ UserPg │ │
|
→ ContentCache hit? → serve from RAM
|
||||||
│ │ FolderFs │ │ PathService │ │ JwtTokenService │ │
|
→ FileBlobReadRepository.get_file_content() / get_file_stream()
|
||||||
│ │ TrashFs │ │ StorageMed. │ │ Argon2Hasher │ │
|
→ SELECT blob_hash FROM storage.files WHERE id = $1
|
||||||
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
→ DedupService.read_blob(hash) → bytes from .blobs/
|
||||||
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
→ Optional WebP transcode → response
|
||||||
│ │ ContentCache │ │ Thumbnail │ │ WriteBehind │ │
|
```
|
||||||
│ │ MetadataCache │ │ Transcode │ │ BufferPool │ │
|
|
||||||
│ │ BufferPool │ │ Dedup │ │ Compression │ │
|
### Data Flow: Folder Operations
|
||||||
│ └────────────────┘ └──────────────┘ └──────────────────┘ │
|
|
||||||
└─────────────────────────────────────────────────────────────┘
|
```
|
||||||
│
|
HTTP Request (POST /api/folders)
|
||||||
┌─────────────────────▼───────────────────────────────────────┐
|
→ FolderService.create_folder()
|
||||||
│ Domain Layer │
|
→ FolderDbRepository.create_folder()
|
||||||
│ Entities: File, Folder, Session, User, Calendar, Contact │
|
→ INSERT INTO storage.folders (name, parent_id, user_id, ...)
|
||||||
│ Value Objects: StoragePath │
|
→ 201 Created (FolderDto)
|
||||||
│ Repository Traits: SessionRepository, ... │
|
```
|
||||||
│ Domain Errors │
|
|
||||||
└─────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|||||||
+31
-24
@@ -134,7 +134,7 @@ Handles: shared element validation, permission management, unique link/token gen
|
|||||||
|
|
||||||
## Infrastructure
|
## Infrastructure
|
||||||
|
|
||||||
**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share links to the filesystem:
|
**ShareFsRepository** (`src/infrastructure/repositories/share_fs_repository.rs`) persists share link metadata to a local JSON file:
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
pub struct ShareFsRepository {
|
pub struct ShareFsRepository {
|
||||||
@@ -157,7 +157,9 @@ struct ShareRecord {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Stores shared links in a JSON file. Supports queries and updates, search by ID/token/user, and pagination.
|
Stores share link records in a JSON file. Supports queries and updates, search by ID/token/user, and pagination.
|
||||||
|
|
||||||
|
> **Note:** This repository stores *share link metadata* only (tokens, permissions, expiration). The actual file/folder content is accessed via `FileReadPort` / `FolderStoragePort` which use the blob storage model (PostgreSQL metadata + DedupService blobs).
|
||||||
|
|
||||||
## API Handlers and Routes
|
## API Handlers and Routes
|
||||||
|
|
||||||
@@ -256,26 +258,30 @@ The service is instantiated via **AppServiceFactory** in `src/common/di.rs` and
|
|||||||
|
|
||||||
```rust
|
```rust
|
||||||
// In AppServiceFactory::create_share_service()
|
// In AppServiceFactory::create_share_service()
|
||||||
let share_service: Option<Arc<dyn ShareUseCase>> = if config.features.enable_file_sharing {
|
pub fn create_share_service(&self, repos: &RepositoryServices)
|
||||||
let share_repository = Arc::new(ShareFsRepository::new(Arc::new(config.clone())));
|
-> Option<Arc<dyn ShareUseCase>>
|
||||||
let share_service = Arc::new(ShareService::new(
|
{
|
||||||
Arc::new(config.clone()),
|
if !self.config.features.enable_file_sharing {
|
||||||
share_repository,
|
return None;
|
||||||
file_read_repository.clone(),
|
}
|
||||||
folder_repository.clone(),
|
|
||||||
password_hasher.clone(),
|
|
||||||
));
|
|
||||||
Some(share_service)
|
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
|
|
||||||
// Add to AppState
|
let share_repository = Arc::new(ShareFsRepository::new(
|
||||||
let app_state = AppState {
|
Arc::new(self.config.clone())
|
||||||
// ...
|
));
|
||||||
share_service: share_service.clone(),
|
|
||||||
// ...
|
let password_hasher: Arc<dyn PasswordHasherPort> =
|
||||||
};
|
Arc::new(Argon2PasswordHasher::new());
|
||||||
|
|
||||||
|
let service = Arc::new(ShareService::new(
|
||||||
|
Arc::new(self.config.clone()),
|
||||||
|
share_repository,
|
||||||
|
repos.file_read_repository.clone(), // FileBlobReadRepository
|
||||||
|
repos.folder_repository.clone(), // FolderDbRepository
|
||||||
|
password_hasher,
|
||||||
|
));
|
||||||
|
|
||||||
|
Some(service)
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Workflows
|
## Workflows
|
||||||
@@ -353,8 +359,9 @@ HTTP status code mapping:
|
|||||||
|
|
||||||
## Technical Notes
|
## Technical Notes
|
||||||
|
|
||||||
- **Performance**: JSON file-based storage works for moderate volumes. For higher load, migrate to a database.
|
- **Share metadata** is stored in a local JSON file via `ShareFsRepository`. This is separate from the 100% blob storage model used for file content.
|
||||||
- **Scalability**: the design supports horizontal scaling via distributed or cloud-based repositories.
|
- **File/folder lookups** during share access go through `FileReadPort` / `FolderStoragePort`, which read metadata from PostgreSQL and content from the DedupService blob store.
|
||||||
|
- **Scalability**: for higher load, share metadata could be migrated to PostgreSQL using the same hexagonal architecture (implement `ShareRepository` with PgPool).
|
||||||
- **Maintenance**: clear separation of concerns makes testing and maintenance straightforward.
|
- **Maintenance**: clear separation of concerns makes testing and maintenance straightforward.
|
||||||
|
|
||||||
The sharing feature is enabled by default in the current configuration.
|
The sharing feature is enabled via `OXICLOUD_ENABLE_FILE_SHARING` configuration flag.
|
||||||
|
|||||||
+117
-79
@@ -1,79 +1,117 @@
|
|||||||
# 14 - Trash Feature
|
# 14 - Trash Feature
|
||||||
|
|
||||||
Soft-delete for files and folders. Items go to a per-user trash bin instead of being permanently removed. Configurable retention period with automatic cleanup.
|
Soft-delete for files and folders. Items go to a per-user trash bin instead of being permanently removed. Configurable retention period with automatic cleanup.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
Follows the hexagonal architecture:
|
Follows the hexagonal architecture:
|
||||||
|
|
||||||
1. **Domain Layer** (`/src/domain/`):
|
1. **Domain Layer** (`/src/domain/`):
|
||||||
- Entities: **TrashedItem** representing files and folders in the trash
|
- Entities: **TrashedItem** representing files and folders in the trash
|
||||||
- Repository interfaces: **TrashRepository** defining trash management operations
|
- Repository interfaces: **TrashRepository** defining trash management operations
|
||||||
|
|
||||||
2. **Application Layer** (`/src/application/`):
|
2. **Application Layer** (`/src/application/`):
|
||||||
- DTOs: **TrashedItemDto** for data transfer between layers
|
- DTOs: **TrashedItemDto** for data transfer between layers
|
||||||
- Ports: **TrashUseCase** defining available operations
|
- Ports: **TrashUseCase** defining available operations
|
||||||
- Services: **TrashService** implementing the trash use cases
|
- Services: **TrashService** implementing the trash use cases
|
||||||
|
|
||||||
3. **Infrastructure Layer** (`/src/infrastructure/`):
|
3. **Infrastructure Layer** (`/src/infrastructure/`):
|
||||||
- Repositories: **TrashFsRepository** for filesystem-based trash storage
|
- Repositories: **TrashDbRepository** (PostgreSQL) — reads from `storage.trash_items` VIEW, manages soft-delete flags
|
||||||
- Trash-related methods in existing repositories: `FileWriteRepository::move_to_trash()`, `FolderRepository::move_to_trash()`, etc.
|
- Trash-related methods in file/folder repositories: `FileBlobWriteRepository::move_to_trash()`, `FolderDbRepository::move_to_trash()`, etc.
|
||||||
- Services: **TrashCleanupService** for automatic cleanup of expired items
|
- Services: **TrashCleanupService** for automatic cleanup of expired items
|
||||||
|
|
||||||
4. **Interface Layer** (`/src/interfaces/`):
|
4. **Interface Layer** (`/src/interfaces/`):
|
||||||
- API handlers: `trash_handler.rs` with HTTP endpoints for trash operations
|
- API handlers: `trash_handler.rs` with HTTP endpoints for trash operations
|
||||||
- Routes: updated `routes.rs` to include trash endpoints
|
- Routes: updated `routes.rs` to include trash endpoints
|
||||||
|
|
||||||
## Key Features
|
## Storage Model
|
||||||
|
|
||||||
1. **Soft Deletion** -- files and folders move to trash, not immediately deleted
|
Trash uses a **soft-delete** model in PostgreSQL:
|
||||||
2. **Per-User Trash** -- each user has an isolated trash bin
|
|
||||||
3. **Retention Policy** -- items auto-delete after a configurable period
|
- Files and folders have `is_trashed` (BOOLEAN) and `trashed_at` (TIMESTAMPTZ) columns
|
||||||
4. **Restoration** -- items can be restored to their original location
|
- When an item is trashed, `is_trashed` is set to `TRUE` and `trashed_at` records the timestamp
|
||||||
5. **Permanent Deletion** -- items can be permanently deleted before retention expires
|
- `original_parent_id` / `original_folder_id` stores the original location for restore
|
||||||
6. **Empty Trash** -- wipe everything in the trash at once
|
- The `storage.trash_items` VIEW provides a unified list of all trashed items (files + folders)
|
||||||
|
- No physical file movement occurs — blob content stays at `.blobs/{prefix}/{hash}.blob`
|
||||||
## API Endpoints
|
- Permanent deletion removes the DB row and decrements the blob reference counter
|
||||||
|
|
||||||
- `GET /api/trash` or `GET /api/trash/` -- list all items in the user's trash
|
```sql
|
||||||
- `DELETE /api/trash/files/:id` -- move a file to trash
|
CREATE OR REPLACE VIEW storage.trash_items AS
|
||||||
- `DELETE /api/trash/folders/:id` -- move a folder to trash
|
SELECT id, name, 'file' AS item_type, folder_id AS parent_id,
|
||||||
- `POST /api/trash/:id/restore` -- restore an item to its original location
|
user_id, size, mime_type, trashed_at, created_at
|
||||||
- `DELETE /api/trash/:id` -- permanently delete an item from trash
|
FROM storage.files WHERE is_trashed = TRUE
|
||||||
- `DELETE /api/trash/empty` -- empty the entire trash bin
|
UNION ALL
|
||||||
|
SELECT id, name, 'folder' AS item_type, parent_id,
|
||||||
## Testing
|
user_id, 0 AS size, NULL AS mime_type, trashed_at, created_at
|
||||||
|
FROM storage.folders WHERE is_trashed = TRUE;
|
||||||
1. **Unit Tests** -- testing **TrashService**:
|
```
|
||||||
- Move files and folders to trash
|
|
||||||
- Restore items from trash
|
## Key Features
|
||||||
- Permanent deletion
|
|
||||||
- Empty trash operation
|
1. **Soft Deletion** — files and folders are flagged as trashed, not immediately deleted
|
||||||
|
2. **Per-User Trash** — each user has an isolated trash bin (filtered by `user_id`)
|
||||||
2. **Integration Tests** -- Python script hitting the API endpoints:
|
3. **Retention Policy** — items auto-delete after a configurable period
|
||||||
- End-to-end testing of all trash operations
|
4. **Restoration** — items can be restored to their original location via `original_parent_id`/`original_folder_id`
|
||||||
- Verification of move, list, restore, and delete behavior
|
5. **Permanent Deletion** — items can be permanently deleted before retention expires (removes DB row + decrements blob ref)
|
||||||
|
6. **Empty Trash** — wipe everything in the trash at once
|
||||||
3. **Shell Script** -- for manual testing and demonstration
|
|
||||||
|
## API Endpoints
|
||||||
## Configuration
|
|
||||||
|
- `GET /api/trash` or `GET /api/trash/` — list all items in the user's trash
|
||||||
- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true)
|
- `DELETE /api/trash/files/:id` — move a file to trash
|
||||||
- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**)
|
- `DELETE /api/trash/folders/:id` — move a folder to trash
|
||||||
|
- `POST /api/trash/:id/restore` — restore an item to its original location
|
||||||
## Implementation Details
|
- `DELETE /api/trash/:id` — permanently delete an item from trash
|
||||||
|
- `DELETE /api/trash/empty` — empty the entire trash bin
|
||||||
1. **Physical File Storage** -- when items are trashed, they physically move to a `.trash` directory
|
|
||||||
2. **Metadata Storage** -- trashed item info stored in a separate database table or file
|
## Implementation Details
|
||||||
3. **User Isolation** -- trash items are isolated by user ID
|
|
||||||
4. **Automatic Cleanup** -- a background job runs periodically to clean up expired items
|
### TrashDbRepository
|
||||||
5. **Transaction Safety** -- operations are atomic with proper error handling
|
|
||||||
|
**File:** `src/infrastructure/repositories/pg/trash_db_repository.rs`
|
||||||
## Future Enhancements
|
|
||||||
|
```rust
|
||||||
1. **Trash Quotas** -- limit trash storage per user
|
pub struct TrashDbRepository {
|
||||||
2. **Batch Operations** -- trash, restore, or delete multiple items at once
|
pool: Arc<PgPool>,
|
||||||
3. **Storage Optimization** -- deduplication for trashed items
|
retention_days: u32,
|
||||||
4. **Version Control** -- track file versions when moving to trash
|
}
|
||||||
5. **Scheduled Cleanup** -- let users configure custom retention periods
|
```
|
||||||
6. **Trash Monitoring** -- metrics and alerts for trash usage and cleanup
|
|
||||||
|
Key methods:
|
||||||
|
- `get_trash_items(user_id)` — SELECT from `storage.trash_items` WHERE `user_id = $1`
|
||||||
|
- `clear_trash(user_id)` — DELETE from `storage.files` and `storage.folders` WHERE `is_trashed = TRUE AND user_id = $1`
|
||||||
|
- `get_expired_items()` — finds items where `trashed_at + retention_days < NOW()`
|
||||||
|
|
||||||
|
### TrashService
|
||||||
|
|
||||||
|
**File:** `src/application/services/trash_service.rs`
|
||||||
|
|
||||||
|
Constructor: `TrashService::new(trash_repo, file_read, file_write, folder_repo, retention_days)`
|
||||||
|
|
||||||
|
Orchestrates trash operations by delegating to the appropriate repository:
|
||||||
|
- Moving a file to trash → `FileBlobWriteRepository::move_to_trash()`
|
||||||
|
- Moving a folder to trash → `FolderDbRepository::move_to_trash()`
|
||||||
|
- Permanent deletion → removes DB row + calls `DedupService::decrement_ref()` to clean blob if unreferenced
|
||||||
|
|
||||||
|
### TrashCleanupService
|
||||||
|
|
||||||
|
**File:** `src/infrastructure/services/trash_cleanup_service.rs`
|
||||||
|
|
||||||
|
Background job that runs every 24 hours to permanently delete items past the retention period.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
1. **Unit Tests** — testing **TrashService**:
|
||||||
|
- Move files and folders to trash
|
||||||
|
- Restore items from trash
|
||||||
|
- Permanent deletion
|
||||||
|
- Empty trash operation
|
||||||
|
|
||||||
|
2. **Integration Tests** — Python script hitting the API endpoints:
|
||||||
|
- End-to-end testing of all trash operations
|
||||||
|
- Verification of move, list, restore, and delete behavior
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
- **OXICLOUD_ENABLE_TRASH**: enable/disable the trash feature via **FeaturesConfig** (default: true)
|
||||||
|
- **OXICLOUD_TRASH_RETENTION_DAYS**: days to keep items before automatic deletion (default: 30, via **StorageConfig**)
|
||||||
|
|||||||
Reference in New Issue
Block a user