Delta-upload client: FastCDC in WASM + overlapped worker pipeline

Phase 2 — the client side of "upload only what changed", closing the delta-sync plan. WASM (wasm/oxicloud-hash): DeltaChunker adds incremental FastCDC with the server's exact crate and parameters (64K/256K/1M) next to the BLAKE3 hasher. The incremental split is provably identical to a single pass: every chunk except the last ends on a content/max-size condition whose decision window was fully buffered, so only the tail is provisional and re-examined as slices arrive. A mirror test — the client twin of the server's stream≡slice test — chunks 4 MiB of xorshift noise with adversarial slice sizes (7 B … 8 MiB) and requires boundary-for-boundary equality with one FastCDC pass. Vendored artifacts rebuilt (55 KB wasm). Worker (static/js/workers/deltaWorker.js): the full protocol off the main thread with OVERLAPPED stages — 8 MiB file slices feed the chunker while earlier batches (256 hashes) negotiate and their missing chunks upload through a 2-deep PUT pool (≤8 MiB framed bodies, bytes re-sliced from the File at send time, never hoarded). Commit handles 409 still_missing by uploading exactly the named hashes and retrying. Orchestrator (features/files/deltaUpload.js): threshold (8 MiB), worker lifecycle + size-scaled timeout, progress relay to the upload bell, conclusive-outcome mapping (201/200, 507 quota, 409 name conflict) and silent fallback to the byte upload for everything else. Wired into uploadFiles and uploadFolderEntries, which now surface one batch summary of the bytes dedup saved. This subsumes the whole-file instant-upload module — a fully-known file negotiates to nothing missing and the commit short-circuits on possession — so instantUpload.js and hashWorker.js are removed (the /api/dedup/check and /api/files/by-hash endpoints remain for API clients). Verified end-to-end against PostgreSQL 16 — the cross-boundary proof the whole design hangs on, in both directions: a 24 MB file byte- uploaded (server-side CDC) then edited and delta-negotiated with WASM-computed chunks reported missing 1/74 (boundaries bit-identical), synced with 344 KB on the wire vs 24 MB (98.6% saved) and downloaded byte-identical; inversely, a file created via delta then byte-uploaded as identical content produced a server-side manifest DEDUP HIT with the same content_hash. Insertion at the head of the file (the adversarial CDC case) still negotiated missing 1/74. Chunk+hash throughput ≈275 MB/s in V8 with SIMD128. https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
2026-06-11 15:44:44 +00:00
parent 44967da7f1
commit 5d034b0d09
11 changed files with 885 additions and 306 deletions
@@ -12,7 +12,7 @@ import { i18n } from '../../core/i18n.js';
 import { notifications } from '../../core/notifications.js';
 import { invalidateFolderMeta } from '../../model/filesModel.js';
 import { triggerBrowserDownload } from '../../utils/download.js';
-import { tryInstantUpload } from './instantUpload.js';
+import { formatSavedSummary, tryDeltaUpload } from './deltaUpload.js';

 /**
 * @typedef {Object} BatchResult
@@ -392,6 +392,7 @@ const fileOps = {
            let uploadedCount = 0;
            let successCount = 0;
            let quotaStop = false;
+            let savedBytesTotal = 0;

            const targetFolderId = app.currentPath || app.userHomeFolderId;

@@ -405,12 +406,19 @@ const fileOps = {
                if (quotaStop) return;
                const file = readableFiles[idx];

-                // ── Instant upload: when the server already has this exact
-                // content for this user, register it by hash — zero bytes
-                // on the wire. Any miss/failure falls back to a byte upload.
-                /** @type {UploadAnswer | null} */
-                let result = await tryInstantUpload(file, targetFolderId);
+                // ── Delta upload: chunk + hash locally (worker/WASM) and
+                // transfer only what the server doesn't already have for
+                // this user. Any miss/failure falls back to a byte upload.
+                /** @type {UploadAnswer & { savedBytes?: number } | null} */
+                let result = await tryDeltaUpload(file, targetFolderId, (pct) => {
+                    if (batchId) {
+                        try {
+                            notifications.updateFile(batchId, file.name, pct, 'uploading');
+                        } catch (_) {}
+                    }
+                });
                if (result) {
+                    savedBytesTotal += result.savedBytes || 0;
                    if (batchId) {
                        try {
                            notifications.updateFile(batchId, file.name, 100, result.ok ? 'done' : 'error');
@@ -492,6 +500,14 @@ const fileOps = {

            // All done
            this._finishUploadToast(successCount, totalFiles);
+            if (savedBytesTotal > 0 && notifications) {
+                notifications.addNotification({
+                    icon: 'fa-bolt',
+                    iconClass: 'upload',
+                    title: i18n?.getCurrentLocale?.()?.startsWith('es') ? 'Subida delta' : 'Delta upload',
+                    text: formatSavedSummary(savedBytesTotal, i18n?.getCurrentLocale?.() || 'en')
+                });
+            }

            // Refresh storage usage display
            try {
@@ -643,6 +659,7 @@ const fileOps = {
            let uploadedCount = 0;
            let successCount = 0;
            let quotaStop = false;
+            let savedBytesTotal = 0;

            // ── Concurrent upload with limited parallelism ──────────
            // FIFOs are pre-caught by the 0-byte arrayBuffer guard,
@@ -672,13 +689,14 @@ const fileOps = {
                    const parentPath = parts.slice(0, -1).join('/');
                    const targetFolderId = folderMap.get(parentPath) || currentFolderId;

-                    // ── Instant upload (zero bytes on the wire) ──
+                    // ── Delta upload (only changed bytes on the wire) ──
                    // Same fallback contract as uploadFiles: a null result
                    // means "do the byte upload". The shared accounting
                    // after this try block handles both outcomes.
-                    const instant = await tryInstantUpload(file, targetFolderId);
-                    if (instant) {
-                        result = instant;
+                    const delta = await tryDeltaUpload(file, targetFolderId);
+                    if (delta) {
+                        result = delta;
+                        savedBytesTotal += delta.savedBytes || 0;
                    } else {
                        // ── FIFO/pipe guard (0-byte files only) ──
                        // Named pipes (runit supervise/control) report size=0
@@ -773,6 +791,14 @@ const fileOps = {
            await Promise.all(workers);

            this._finishUploadToast(successCount, totalFiles);
+            if (savedBytesTotal > 0 && notifications) {
+                notifications.addNotification({
+                    icon: 'fa-bolt',
+                    iconClass: 'upload',
+                    title: i18n?.getCurrentLocale?.()?.startsWith('es') ? 'Subida delta' : 'Delta upload',
+                    text: formatSavedSummary(savedBytesTotal, i18n?.getCurrentLocale?.() || 'en')
+                });
+            }

            try {
                await refreshUserData();