Oxicloud

cjw/Oxicloud

Fork 0

Commit Graph

Author	SHA1	Message	Date
Claude	82ee7da0d2	perf: serve ranges from RAM cache, stream ZIPs, overlap ingest settle, O(1) chunk gate Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change gated by a before/after in examples/bench_round2.rs — an AFTER that did not beat its BEFORE was to be rolled back; none needed it): - Range requests (REST/DAV/shares) answered from the moka content cache for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice. 256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us). - Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster). Content-Length dropped (size unknown up front). - NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM per-session counter (lazy rebuild on cold start). 1,000-chunk upload gate cost: 33.1s -> 0.09s cumulative. - Delta download + commit-verify now use the CDC path's buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open latency 440ms -> 51ms; order preserved. - CDC ingest settles batches on a spawned task (depth-1 pipeline) so the source stream keeps flowing during PG pin + backend writes; rollback ledger shared + lock-serialized so compensation stays exact on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s. OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch). - Frontend: instant-upload BLAKE3 hashing moved off the main thread to a bounded Web Worker pool (File handles by reference); vitest gate asserts the pool beats sequential (first gate draft posting buffers was 2.6x slower and was rewritten — copies dominated). Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544 integration tests green; 270 frontend tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w	2026-07-16 16:50:07 +00:00
Claude	ed9a204e49	Delta download: file manifest + user-scoped chunk fetch for sync clients Phase 3 of the delta-sync plan — the inverse direction, so a future client app holding an older local version can fetch only what changed: - GET /api/files/{id}/manifest returns the file's chunk recipe ({file_hash, total_size, chunks}). Owner-scoped like the rest of the delta surface (Read permission through the authz engine first, then the chunk layer's possession standard; shared files use the regular download endpoints). A manifest is immutable for a given file_hash, so it is served with ETag = file_hash and If-None-Match answers 304 — polling sync clients pay one header round-trip per unchanged file. Legacy pre-CDC blobs are presented as a single-chunk manifest of themselves, so clients need no special case. - POST /api/files/delta/download streams the requested chunks as [u32 BE length][bytes] frames in request order — the same wire format the upload direction uses. Entitlement is the same possession rule as negotiate/commit (chunks reachable through the caller's own files); anything else returns 404 {not_available} — deliberately indistinguishable from "never existed" — with a delta_download.rejected audit event. Batches are bounded by the chunk_max_bytes budget; Content-Length is exact (sizes come from the dedup index) and peak RAM is one backend read frame. Both endpoints share the delta rate limiter. New DedupService primitives: manifest_chunk_list (with legacy fallback), chunk_sizes, chunk_stream. OpenAPI regenerated; protocol doc gains the download section; types.js maps the new wire shapes (plus the delta-upload typedefs that a container reset had silently dropped from a previous commit). Verified end-to-end against PostgreSQL 16 with a simulated two-device sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits → 2 chunks); device B diffed the manifest against its WASM-chunked local copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1% saved) and rebuilt the file byte-identical with the BLAKE3 verifying. If-None-Match revalidation returned 304; a second user got 404 on both the manifest and the chunk batch (with the not_available list and audit lines); an unknown hash was indistinguishable from a denied one; an empty hash list returned 400. https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS	2026-06-11 16:38:02 +00:00
Claude	44967da7f1	Delta-upload protocol: negotiate chunks by hash, upload only what changed Phase 1 of the delta-sync plan, server side. The CDC store already shares unchanged chunks between file versions after the bytes arrive; these three stateless endpoints move that detection to the client, so editing a few bytes of a large file uploads ~1 MiB instead of the whole file: - POST /api/files/delta/negotiate — given the file's chunk hashes, answer which ones the caller must upload. User-scoped and purely advisory. - PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes] frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request). Every hash is recomputed server-side; chunks land as ref_count=0 orphans that a commit pins or the periodic GC sweeps — no session table. - POST /api/files/delta/commit — pin one reference per distinct chunk with a single UPDATE…RETURNING restricted to chunks the caller is entitled to (reachable through their own non-trashed files, or unreferenced orphans); anything else returns 409 {still_missing} for the client to upload and retry. The pinned sequence is then RE-READ and the whole-file BLAKE3 recomputed before any manifest exists — a declared file_hash is never trusted, because a forged manifest would poison future whole-file dedup hits for other users. The manifest accounting is shared with the byte path (attach_manifest, extracted from store_from_stream); the file row is created (201) or its content swapped by file_id (200). Owners of the exact file_hash short-circuit to a pure reference bump. Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment probes were sequential scans), claimable/pin/release/store-loose/verify primitives on DedupService, update-by-id with Update-permission AuthZ on FileUploadService, per-caller rate limiter (240/min), audit events with stable reasons (rate_limited, chunk_verification_failed, file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing parser unit tests and a PG-gated integration suite covering the entitlement matrix (owned/foreign/orphan/unknown), orphan registration and the verification read. Verified end-to-end against PostgreSQL 16 with a node client hashing via the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks; a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the wire vs 24 MB (98.9% saved), downloading byte-identical. A second user probing the same chunks got nothing (negotiate: all missing; commit: 409 with all 96 still withheld); a forged file_hash returned 400 plus the audit line; a commit referencing one never-uploaded chunk returned 409 naming exactly that hash; the GIN index serves containment probes (Bitmap Index Scan) once the planner favors it. https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS	2026-06-11 14:34:02 +00:00

Author

SHA1

Message

Date

Claude

82ee7da0d2

perf: serve ranges from RAM cache, stream ZIPs, overlap ingest settle, O(1) chunk gate

Round 2 of benchmark-gated optimizations (benches/ROUND2.md; every change
gated by a before/after in examples/bench_round2.rs — an AFTER that did
not beat its BEFORE was to be rolled back; none needed it):

- Range requests (REST/DAV/shares) answered from the moka content cache
  for sub-10MB files: PG resolve + open/seek/read -> Bytes::slice.
  256KiB seeks: 1,730/s -> 3.7M/s (p50 552us -> 0.15us).
- Streaming folder/share ZIPs via tokio duplex: TTFB no longer scales
  with archive size (326ms -> 0.4ms on 192MiB corpus; total also faster).
  Content-Length dropped (size unknown up front).
- NC chunked-upload per-PUT gate: O(k) directory scan+stat -> in-RAM
  per-session counter (lazy rebuild on cold start). 1,000-chunk upload
  gate cost: 33.1s -> 0.09s cumulative.
- Delta download + commit-verify now use the CDC path's
  buffered(read_prefetch) read-ahead: 64-chunk drain at 5ms open
  latency 440ms -> 51ms; order preserved.
- CDC ingest settles batches on a spawned task (depth-1 pipeline) so
  the source stream keeps flowing during PG pin + backend writes;
  rollback ledger shared + lock-serialized so compensation stays exact
  on cancellation. 512MiB paced ingest: 60-69 -> 74-75 MB/s.
  OXICLOUD_INGEST_OVERLAP=0 restores inline settling (ops/bench hatch).
- Frontend: instant-upload BLAKE3 hashing moved off the main thread to
  a bounded Web Worker pool (File handles by reference); vitest gate
  asserts the pool beats sequential (first gate draft posting buffers
  was 2.6x slower and was rewritten — copies dominated).

Validation: cargo fmt + clippy -D warnings clean; 514 unit + 544
integration tests green; 270 frontend tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w

2026-07-16 16:50:07 +00:00

Claude

ed9a204e49

Delta download: file manifest + user-scoped chunk fetch for sync clients

Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:

- GET /api/files/{id}/manifest returns the file's chunk recipe
  ({file_hash, total_size, chunks}). Owner-scoped like the rest of the
  delta surface (Read permission through the authz engine first, then
  the chunk layer's possession standard; shared files use the regular
  download endpoints). A manifest is immutable for a given file_hash,
  so it is served with ETag = file_hash and If-None-Match answers 304 —
  polling sync clients pay one header round-trip per unchanged file.
  Legacy pre-CDC blobs are presented as a single-chunk manifest of
  themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
  [u32 BE length][bytes] frames in request order — the same wire format
  the upload direction uses. Entitlement is the same possession rule as
  negotiate/commit (chunks reachable through the caller's own files);
  anything else returns 404 {not_available} — deliberately
  indistinguishable from "never existed" — with a
  delta_download.rejected audit event. Batches are bounded by the
  chunk_max_bytes budget; Content-Length is exact (sizes come from the
  dedup index) and peak RAM is one backend read frame.

Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).

Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS

2026-06-11 16:38:02 +00:00

Claude

44967da7f1

Delta-upload protocol: negotiate chunks by hash, upload only what changed

Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:

- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
  which ones the caller must upload. User-scoped and purely advisory.
- PUT  /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
  frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
  Every hash is recomputed server-side; chunks land as ref_count=0 orphans
  that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
  a single UPDATE…RETURNING restricted to chunks the caller is entitled
  to (reachable through their own non-trashed files, or unreferenced
  orphans); anything else returns 409 {still_missing} for the client to
  upload and retry. The pinned sequence is then RE-READ and the whole-file
  BLAKE3 recomputed before any manifest exists — a declared file_hash is
  never trusted, because a forged manifest would poison future whole-file
  dedup hits for other users. The manifest accounting is shared with the
  byte path (attach_manifest, extracted from store_from_stream); the file
  row is created (201) or its content swapped by file_id (200). Owners of
  the exact file_hash short-circuit to a pure reference bump.

Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.

Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS

2026-06-11 14:34:02 +00:00

3 Commits