Instant upload: register already-owned content by hash, zero bytes on the wire
Phase 0 of the delta-sync plan. Re-uploading a file the user already has (another device, a restore, a duplicate) used to transfer every byte just for the server to discard them as a dedup hit. The frontend now computes the file's BLAKE3 locally and, on a hit, registers the file with a single ~150-byte metadata call. Server — POST /api/files/by-hash: - All checks live in the application service per the AuthZ rule: Create permission on the target folder via the authorization engine, hash ownership via the existing user-scoped query (a non-owned hash returns 404 — same shape as "no such blob" — and emits an instant_upload.rejected audit event), quota on the logical size. - On success: one ref_count bump + the existing save_file_with_blob row registration (compensation included); is_new_blob=false so lifecycle hooks skip thumbnail regeneration. ~10 ms warm. - The storage-usage service is now built before the application services and injected, instead of only living on AppState. Client — WASM BLAKE3 + worker: - wasm/oxicloud-hash: the exact same blake3 crate the server uses, compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match server content addresses bit for bit. Built by scripts/build-wasm.sh; the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no npm dependencies, no wasm toolchain needed for regular builds. - static/js/workers/hashWorker.js streams the File in 8 MiB slices off the main thread (constant RAM at any file size). - features/files/instantUpload.js orchestrates: threshold (8 MiB — below it the round-trips cost more than the bytes), user-scoped /api/dedup/check, by-hash registration, and silent fallback to the normal byte upload on any miss, race or unsupported environment. Wired into both uploadFiles and uploadFolderEntries. - biome.json vendors exclusion fixed to cover nested directories (previous vendors were .mjs and never matched the *.js include). Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms warm with a 151-byte request (vs 20,971,873 bytes for the byte upload); the copy downloads byte-identical and the manifest ref_count goes 1→2; a second user probing the same hash gets exists:false and 404 plus the audit line; duplicate name → 409, malformed hash → 400; worker and wasm are served with correct MIME (application/wasm). https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
+20
-5
@@ -439,6 +439,7 @@ impl AppServiceFactory {
|
||||
repos: &RepositoryServices,
|
||||
trash_service: Option<Arc<TrashService>>,
|
||||
authz: &Arc<PgAclEngine>,
|
||||
storage_usage: &Arc<StorageUsageService>,
|
||||
) -> ApplicationServices {
|
||||
// Main services
|
||||
let folder_service = Arc::new(FolderService::new(
|
||||
@@ -452,7 +453,12 @@ impl AppServiceFactory {
|
||||
repos.file_read_repository.clone(),
|
||||
)
|
||||
.with_content_cache(core.file_content_cache.clone())
|
||||
.with_file_lifecycle_hook(core.file_lifecycle.clone()),
|
||||
.with_file_lifecycle_hook(core.file_lifecycle.clone())
|
||||
.with_instant_upload(
|
||||
authz.clone(),
|
||||
core.dedup_service.clone(),
|
||||
storage_usage.clone(),
|
||||
),
|
||||
);
|
||||
|
||||
let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache(
|
||||
@@ -733,9 +739,19 @@ impl AppServiceFactory {
|
||||
.create_trash_service(&repos, &core, &authorization)
|
||||
.await;
|
||||
|
||||
// 3c. Storage usage / quota service (needed by the instant-upload
|
||||
// path inside the application services, and re-exposed on AppState
|
||||
// for the handler-side quota checks of the byte-upload paths).
|
||||
let storage_usage = self.create_storage_usage_service(&repos, &pool, &maintenance_pool);
|
||||
|
||||
// 4. Application services (with trash + authz already wired)
|
||||
let mut apps =
|
||||
self.create_application_services(&core, &repos, trash_service.clone(), &authorization);
|
||||
let mut apps = self.create_application_services(
|
||||
&core,
|
||||
&repos,
|
||||
trash_service.clone(),
|
||||
&authorization,
|
||||
&storage_usage,
|
||||
);
|
||||
|
||||
// 5. Share service
|
||||
let share_service = self.create_share_service(&repos, &pool, &authorization);
|
||||
@@ -775,8 +791,7 @@ impl AppServiceFactory {
|
||||
recent_service = Some(recent.clone());
|
||||
apps.recent_service = Some(recent);
|
||||
|
||||
storage_usage_service =
|
||||
Some(self.create_storage_usage_service(&repos, &pool, &maintenance_pool));
|
||||
storage_usage_service = Some(storage_usage.clone());
|
||||
|
||||
self.start_tree_etag_flush_job(&maintenance_pool);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user