feat: thumbnail dedup — store thumbnails by blob_hash instead of file_id (#233)
Thumbnails are now keyed by blob_hash on disk so identical files share
a single set of thumbnails (icon/preview/large). For 4000 duplicate
files with the same content, this reduces thumbnail storage from 12,000
files to just 3.
Changes:
- get_thumbnail_path() keys by blob_hash instead of file_id
- get_thumbnail(), get_cached_thumbnail(), generate_all_sizes_background()
accept blob_hash parameter for disk dedup
- generate_all_sizes_background() fast path: if blob-hash thumbnails
already exist on disk, skip image processing entirely and just
populate moka cache for the new file_id
- delete_thumbnails() only invalidates moka cache (shared disk
thumbnails must not be deleted when one file is removed)
- delete_blob_thumbnails() added for GC; garbage_collect() now cleans
up orphaned thumbnail files alongside blob files
- External thumbnails (video frames) stored as ext-{file_id}.jpg
since they are client-generated and not dedup-able
- ThumbnailPort trait updated with blob_hash parameters
- All handler call sites updated (file_handler, preview_handler)
- Tests updated for new signatures
This commit is contained in:
@@ -68,18 +68,27 @@ pub trait ThumbnailPort: Send + Sync + 'static {
|
||||
|
||||
/// Get a thumbnail, generating it on-demand if needed.
|
||||
///
|
||||
/// Returns the thumbnail bytes in WebP format.
|
||||
/// `blob_hash` is the content hash used as the disk storage key
|
||||
/// (dedup: identical blobs share one set of thumbnails).
|
||||
async fn get_thumbnail(
|
||||
&self,
|
||||
file_id: &str,
|
||||
blob_hash: &str,
|
||||
size: ThumbnailSize,
|
||||
original_path: &Path,
|
||||
) -> Result<Bytes, DomainError>;
|
||||
|
||||
/// Generate all thumbnail sizes for a file in the background.
|
||||
///
|
||||
/// Called after file upload to pre-generate thumbnails.
|
||||
fn generate_all_sizes_background(self: Arc<Self>, file_id: String, original_path: PathBuf);
|
||||
/// `blob_hash` is the content hash used as the disk storage key.
|
||||
/// If thumbnails already exist for this hash, only the moka cache
|
||||
/// is populated (zero CPU for image processing).
|
||||
fn generate_all_sizes_background(
|
||||
self: Arc<Self>,
|
||||
file_id: String,
|
||||
blob_hash: String,
|
||||
original_path: PathBuf,
|
||||
);
|
||||
|
||||
/// Delete all thumbnails for a file.
|
||||
async fn delete_thumbnails(&self, file_id: &str) -> Result<(), DomainError>;
|
||||
@@ -87,9 +96,14 @@ pub trait ThumbnailPort: Send + Sync + 'static {
|
||||
/// Try to get a cached thumbnail without generating one.
|
||||
///
|
||||
/// Returns `None` if no cached thumbnail exists on disk or in memory.
|
||||
/// Used for non-image file types (videos) where thumbnails are
|
||||
/// generated client-side and uploaded.
|
||||
async fn get_cached_thumbnail(&self, file_id: &str, size: ThumbnailSize) -> Option<Bytes>;
|
||||
/// `blob_hash` is used to locate the file on disk. If `None`, only
|
||||
/// the in-memory moka cache is checked.
|
||||
async fn get_cached_thumbnail(
|
||||
&self,
|
||||
file_id: &str,
|
||||
blob_hash: Option<&str>,
|
||||
size: ThumbnailSize,
|
||||
) -> Option<Bytes>;
|
||||
|
||||
/// Store an externally-generated thumbnail (e.g. client-side video frame).
|
||||
///
|
||||
|
||||
Reference in New Issue
Block a user