cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes

Browse Source 
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
2026-07-16 14:20:20 +00:00
parent b69c18b934
commit aba89c4f5d
52 changed files with 3262 additions and 444 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benches/CHROOT-CACHE.md
+45
View File
@@ -0,0 +1,45 @@
# NC chroot / default-drive resolution — moka caches (vs 2 queries/request)
With app-password verification already cached (5 min) and user flags cached
(30 s), the NextCloud basic-auth middleware still resolved the chroot from
scratch on EVERY protected NC request: `find_default_for_user` (drives JOIN
folders) + `get_folder(root_id)` (folders by PK) — 2 uncached round-trips + 2
pool checkouts before the handler even ran, for values that change only on
provisioning / drive deletion / a root-folder rename. The native `/webdav`
surface repeated the drive lookup per request (Mode-B scope resolution, MOVE
and COPY twice), WOPI once per call.
Changes:
1. `DrivePgRepository::find_default_for_user` memoised (moka, 30 s TTL —
same tier as `drive_role_cache`), invalidated on personal-drive creation,
drive deletion and policy updates. Only `Ok` is cached, so the
provisioning idempotency check still sees the live table.
2. NC middleware markerless-chroot `FolderDto` cached by root-folder id
(30 s TTL). Only the markerless branch — the drive-marker branch keeps
its per-request `get_folder_with_perms` authz.
Staleness: bounded at 30 s for a root-folder *rename* (doesn't pass through
the repo); every other mutation invalidates explicitly.
## Reproduce
```bash
cargo run --release --features bench --example bench_chroot_cache
# tunables: BENCH_POOL=20 BENCH_SECONDS=4 BENCH_CONCURRENCIES=8,64
```
## Results (4 cores, local PG16, pool=20)
| conc | mode | req/s | p50 µs | p95 µs | p99 µs | queries |
|-----:|--------|----------:|--------:|--------:|--------:|--------:|
| 8 | BEFORE | 11,013 | 696.8 | 1,203.2 | 1,642.6 | 88,102 |
| 8 | AFTER | 2,011,191 | 0.69 | 1.97 | 8.47 | 0 |
| 64 | BEFORE | 16,952 | 3,633.3 | 5,617.6 | 7,189.1 | 135,618 |
| 64 | AFTER | 2,337,233 | 0.93 | 2.23 | 11.30 | 0 |
- The fixed per-request DB tax of the whole NC surface (sync PROPFIND storms,
per-chunk uploads, previews, OCS polls) drops from **0.7–3.6 ms p50 (and 2
pool checkouts)** to a **sub-µs moka hit**.
- Under sync-storm concurrency (64 in-flight) the BEFORE p99 was 7.2 ms of
pure chroot overhead per request — that whole term vanishes.
benches/DEAD-PROPS.md
+57
View File
@@ -0,0 +1,57 @@
# WebDAV dead-properties — batched per-page fetch (vs per-child N+1)
The streaming PROPFIND walkers (native `webdav_handler.rs`, NextCloud
`nextcloud/webdav_handler.rs`, plus both NC REPORT handlers) fetched dead
properties **one child at a time, sequentially** — one DB round-trip per file
and per subfolder of every Depth:1 listing. On top, every
`DeadPropertyStore` query filtered with `folder_id IS NOT DISTINCT FROM $1 AND
file_id IS NOT DISTINCT FROM $2`, which PostgreSQL cannot serve from a B-tree
index (`IS NOT DISTINCT FROM` is not an indexable operator) — so each of those
N round-trips also degraded to a **sequential scan** as the table grew.
Changes:
1. `DeadPropertyStore::get_all_for_files / get_all_for_folders` — ONE
`file_id = ANY($1)` round-trip per 500-child PROPFIND page (indexable via
the partial unique indexes from migration 20260830000001).
2. All single-resource queries (`get`, `get_all`, `remove`) now filter on the
concrete column (`file_id = $1` / `folder_id = $1`) instead of the
NULL-tolerant pair — index scans instead of seq scans.
3. All four handler loops replaced with one batched map lookup per page.
## Reproduce
```bash
cargo run --release --features bench --example bench_dead_props
# tunables: BENCH_CHILDREN=2000 BENCH_PAGE=500 BENCH_NOISE_ROWS=20000 BENCH_REPS=5
```
Measures exactly the dead-prop portion of one Depth:1 PROPFIND of a
2,000-child folder (what the walker adds on top of the listing queries).
## Results (4 cores, local PG16, this container)
**Table with only the 2,000 seeded rows:**
| mode | queries | total ms | vs OLD |
|-------------------------------|--------:|---------:|-------:|
| OLD — seq, IS NOT DISTINCT | 2000 | 1072.41 | 1.0× |
| EQ — seq, `file_id = $1` | 2000 | 509.89 | 2.1× |
| BATCH — `= ANY($1)` per page | 4 | 4.15 | **258×** |
**Table with 22,000 rows (realistic volume — seq scans hurt):**
| mode | queries | total ms | vs OLD |
|-------------------------------|--------:|---------:|-------:|
| OLD — seq, IS NOT DISTINCT | 2000 | 4543.74 | 1.0× |
| EQ — seq, `file_id = $1` | 2000 | 515.84 | 8.8× |
| BATCH — `= ANY($1)` per page | 4 | 5.88 | **773×** |
- A Depth:1 PROPFIND of a 2,000-child folder was spending **1.1–4.5 s** on
dead-prop chatter alone — now **~5 ms**. This is per folder per sync poll,
on the hottest path desktop sync clients have.
- The `EQ` row isolates the indexability fix (2.1–8.8×); the batching is the
rest. Both are applied.
- Same unit economics apply to the other N+1s fixed alongside (search ReBAC
batch, ZIP batch authz): each eliminated sequential point query is worth
~0.25–2.3 ms of the numbers above depending on table size.
benches/NPLUS1-AND-CACHES.md
+90
View File
@@ -0,0 +1,90 @@
# Companion fixes — same measured unit economics, no dedicated harness
These changes share their cost model with benches that already exist, so
instead of near-duplicate harnesses each entry cites the bench that measured
its unit price. (The per-query unit prices below: sequential indexed point
SELECT ≈ 0.25–0.55 ms and `= ANY($1)` batch ≈ 1–1.5 ms/500 ids from
benches/DEAD-PROPS.md; manifest-row fetch p50 0.44–4.4 ms from
benches/BLOB-MANIFEST.md; moka hit ≈ 1 µs from benches/CHROOT-CACHE.md.)
## 1. Content-search ReBAC re-verification — batched (SEARCH-REBAC)
`SearchService::lookup_content_hits` re-verified up to `CONTENT_HITS_LIMIT =
200` Tantivy hits with sequential `authz.check(Read, File)` calls — each a
point SELECT on owner-cache miss (distinct file ids ⇒ ~always). New
`AuthorizationEngine::check_files_read_batch` (default = the old loop, so
mocks/other impls stay correct; `PgAclEngine` override): ONE
`id = ANY($1)` drive resolution + cached per-drive role + per-file cascade
only for drive-floor misses. Decision-equivalent; per 200-hit search:
**~200 sequential round-trips (≈ 50–110 ms of DB chatter) → 1–2 queries
(≈ 1–3 ms)**. Also primes the owner cache for the hits' follow-up requests.
## 2. Batch-ZIP downloads — no per-file authz/Recent (ZIP-BATCH-AUTHZ)
`BatchOperations::add_folder_subtree_to_zip` had already authorized the
subtree ROOT (`get_folder_with_perms`), yet every enumerated file still paid
`get_file_stream_with_perms` = 1 authz point SELECT + a Recent-hook spawn
issuing 2 writes (INSERT … ON CONFLICT + prune DELETE). A 2,000-file folder
ZIP ⇒ ~6,000 extra statements. Subtree entries now use the plain
`get_file_stream` — exactly what `ZipService::create_folder_zip` (the native
folder-download path) has always done. Explicitly-selected top-level files
keep per-file authz + Recent. Unit price: DEAD-PROPS.md sequential rows —
**~1.5–4.5 s of DB chatter removed** from a 2,000-file archive, plus the ZIP
no longer floods Recents with every archived file.
## 3. CDC manifest RAM cache (MANIFEST-CACHE)
Every stream / range / full read of a CDC blob paid one
`chunk_manifests` row fetch first — p50 0.44 ms (4.4 ms under pool pressure,
benches/BLOB-MANIFEST.md), on the hottest read paths there are (media
serving, thumbnails, range seeks). Manifests are immutable by content
address, so `DedupService` now memoises them (moka, weight-bounded 32 MiB,
60 s TTL, positive-only so background rechunking is honoured immediately;
invalidated post-commit on the two delete paths). Warm read: **0.44–4.4 ms →
~1 µs** (CHROOT-CACHE.md's moka row) and one fewer pool checkout per read —
range-seek storms (video scrubbing) hit this every request.
## 4. Public share landing — 3 round-trips → 1 atomic UPDATE (SHARE-ACCESS)
`GET /api/s/{token}` ran find_share_by_token (with a correlated
`MIN(expires_at)` subquery), a full-row UPDATE writing back a Rust-side
increment (racy: lost updates between concurrent visitors, and it rewrote
`item_name`/`password_hash` wholesale — clobbering concurrent owner edits),
then the handler's follow-up fetched the share a third time.
`ShareStoragePort::increment_access_count` is now one
`UPDATE … SET access_count = access_count + 1 WHERE token = $1 AND <expiry>`:
**3 subquery round-trips → 2** for the landing (register + fetch), no
read-modify-write race, no collateral column rewrites.
## 5. Trash — dead SELECT removed
`TrashService::move_to_trash` fetched the full file/folder entity to build a
`TrashedItem` consumed only by `TrashRepository::add_to_trash` — a documented
no-op in the soft-delete model. Both branches now go straight to the
`move_to_trash` UPDATE: **one uncached SELECT + entity hydration removed per
trash operation** (file and folder).
## 6. NFC normalization fast path
`normalize_storage_name` ran unicode-normalization's full
decompose/recompose state machine on every name of every row loaded from PG
(listings, PROPFIND, photos — 27 constructor call sites), even though the DB
invariant guarantees stored names are already NFC. `is_nfc_quick` (a
per-char table lookup) now short-circuits the ~100 % case to a plain copy;
`Maybe`/`No` still run the full pipeline, so semantics are unchanged.
## 7. Frontend — first-page render for large folders
`fetchFolderListing` paged the ENTIRE folder (sequential 200-item requests)
before returning anything — a 2,000-item folder waited ~10 round-trips
before first paint. The files route now paints page one immediately via the
new `onPage` hook and fills in as later pages land (skipped when a cached
listing is already on screen, so views never shrink). First-paint latency
for an N-item folder drops from ⌈N/200⌉ sequential RTTs to 1.
## Refuted by benchmark (reverted, kept for the record)
- **Cached `Intl.Collator` for name sorts (frontend):** sorting 5,000 names —
argument-less `localeCompare` 5.6 ms vs cached collator **12.1 ms (2×
slower)**. V8 fast-paths argument-less `localeCompare`; the "cache the
collator" folklore does not apply. Reverted, ordering untouched.
benches/PEOPLE-LIST.md
+35
View File
@@ -0,0 +1,35 @@
# People tab — grouped COUNT (vs full faces scan with embeddings)
`PeopleService::list_people` (GET `/api/people`, fetched on every People-tab
mount) called `faces_for_user`, which SELECTs every face row for the caller —
each carrying a 2,048-byte embedding BYTEA that gets decoded into a fresh
`Vec<f32>` — only to (a) count faces per person and (b) resolve a handful of
cover faces to file ids. A 10k-face library moved ~21 MB of embeddings per
request. `merge()` had the same over-fetch plus one UPDATE per face.
Changes (`FaceRepository` + `PeopleService`):
- `person_face_stats`: `SELECT person_id, COUNT(*) … GROUP BY person_id`.
- `file_ids_for_faces`: one `id = ANY($1)` over just the cover face ids.
- `reassign_person_faces`: merge as ONE set-based UPDATE (was: load all
faces, filter in Rust, one UPDATE per face).
## Reproduce
```bash
cargo run --release --features bench --example bench_people_list
# tunables: BENCH_FACES=10000 BENCH_PERSONS=20 BENCH_REPS=5
```
## Results (4 cores, local PG16, 10,000 faces / 20 persons)
| mode | total ms | bytes moved |
|--------------------------|---------:|------------:|
| BEFORE — full face rows | 30.40 | 20,960,000 |
| AFTER — COUNT + covers | 3.76 | 1,280 |
- **8.1× faster** and **~16,000× fewer bytes** off the wire per People-tab
mount. The heap never materialises 10k embedding `Vec<f32>`s.
- The BEFORE row also allocated ~21 MB per request on the server; under a
handful of concurrent mounts that was tens of MB of transient RSS for a
page that shows 20 avatars.
benches/PROPFIND-PAGING.md
+51
View File
@@ -0,0 +1,51 @@
# PROPFIND folder paging — keyset cursor + (folder_id, name) index
`list_files_batch` walks a folder's children in name order, 500 per page
(native + NextCloud PROPFIND streamers). The old shape was `ORDER BY name
LIMIT 500 OFFSET k` with **no supporting index** — the initial schema's
`(folder_id, name, user_id)` index that served it was dropped by migration
20260902000000 (user_id → nullable), leaving only `idx_files_folder_id`. So
every page bitmap-scanned all N children and top-sorted them: a full listing
of an N-file folder cost O(N²/500) row visits + ⌈N/500⌉ sorts.
Changes:
1. Migration `20260917000000_files_folder_name_index.sql`: partial composite
`idx_files_folder_name (folder_id, name) WHERE NOT is_trashed`.
2. `list_files_batch` cursor switched from OFFSET to keyset
(`name > $last`, names are unique per folder via the
`(drive_id, folder_id, name)` unique index) across the port trait, the
repository and both handler loops. The cursor predicate is only emitted
when a cursor exists — a `$2 IS NULL OR …` disjunction would block the
index condition under the extended protocol's generic plans.
## Reproduce
```bash
cargo run --release --features bench --example bench_propfind_paging
# tunables: BENCH_FILES=20000 BENCH_PAGE=500 BENCH_REPS=3
```
Times the FULL page-by-page walk of a 20,000-file folder (the listing
portion of one Depth:1 PROPFIND).
## Results (4 cores, local PG16)
| mode | total ms | vs OLD |
|----------------------------------|---------:|-------:|
| OFFSET, no index (true BEFORE) | 1,266.3 | 1.0× |
| OFFSET + index (index alone) | 482.7 | 2.6× |
| KEYSET + index (AFTER) | 76.7 | **16.5×** |
- Full-folder listing cost drops **16.5×**; unlike OFFSET (even indexed),
keyset stays O(page) at any depth, so the gap widens with folder size.
- Companion fix in the same commit: the Photos timeline cursor
(`list_media_files`) wrapped its keyset column in
`EXTRACT(EPOCH FROM …)::bigint` plus an `IS NULL OR` disjunction —
non-sargable, so page k re-scanned all k·limit rows already scrolled past.
It now compares the raw `media_sort_date` against a timestamptz bind
(identical row semantics — the cursor is whole seconds) and splits the
cursor/no-cursor query shapes, restoring the
`idx_files_media_timeline_by_drive` boundary condition the index was built
for. Same mechanism as measured above (index-boundary vs per-row filter);
the deep-scroll effect mirrors the OFFSET column.
benches/QUOTA-PATH.md
+42
View File
@@ -0,0 +1,42 @@
# Quota path — narrow 2-column read + skip-when-not-requested
Two independent fixes on the quota resolution that runs on every upload check
and every quota-reporting folder PROPFIND:
1. **Narrow read.** `check_storage_quota` / `get_user_storage_info` called
`get_user_by_id`, whose SELECT drags the entire `auth.users` row —
including `image`, an avatar data URI of up to 512 KiB — to read two i64s.
New `UserPgRepository::get_storage_usage` reads exactly
`(storage_used_bytes, storage_quota_bytes)` (same pattern as the existing
`get_user_flags`).
2. **Skip entirely when not asked.** `resolve_webdav_quota` (2 round-trips:
drive row + user row) ran on EVERY folder PROPFIND on both surfaces, even
when the client's `<D:prop>` list named no quota property — which is the
common shape for sync-client polls. `PropFindRequest::wants_quota()` now
gates it: `AllProp`/`PropName` keep quota (the writers emit RFC 4331 props
there), explicit prop lists trigger the lookups only if they name
`quota-used-bytes` / `quota-available-bytes`. Responses are byte-identical
for every request that names quota or asks for allprop.
## Reproduce
```bash
cargo run --release --features bench --example bench_quota_path
# tunables: BENCH_SECONDS=4 BENCH_CONCURRENCIES=8,64 BENCH_IMAGE_KB=512
```
## Results (4 cores, local PG16, pool=20, 512 KiB avatar on the row)
| conc | mode | ops/s | p50 µs | p99 µs |
|-----:|--------|-------:|---------:|---------:|
| 8 | FULL | 2,222 | 3,369.4 | 8,452.2 |
| 8 | NARROW | 25,118 | 294.9 | 867.9 |
| 64 | FULL | 2,567 | 24,642.3 | 36,164.0 |
| 64 | NARROW | 40,195 | 1,468.9 | 3,964.9 |
- **11–16× throughput, p50 3.4 ms → 0.29 ms** for the user-row half of every
quota resolution (the avatar bytes dominated the wire+decode cost).
- With `wants_quota()` the common PROPFIND pays **zero** quota queries — the
numbers above then only apply to requests that actually ask for quota.
- The same narrow read protects every upload (`check_storage_quota` gates all
upload paths), where the FULL row was pure overhead per file.
benches/STATIC-PRECOMPRESSED.md
+56
View File
@@ -0,0 +1,56 @@
# Static assets & API responses — precompressed siblings + explicit Brotli level
Two related findings, one root cause: tower-http's `CompressionLayer` default
maps to **Brotli QUALITY 11** (`async-compression Level::Default` →
`BrotliEncoderParams::default()`, brotli-8.0.2 `encode.rs:323` — verified in
source and empirically below). Quality 11 is a deploy-time setting; it was
running per request on:
- every SPA asset (`interfaces/web/mod.rs` layer): ~1.3 s CPU per 700 KiB
bundle per request;
- every compressible API response (`main.rs` global layer): ~90 ms CPU per
64 KiB JSON response.
Changes:
1. **Precompressed statics.** `frontend/scripts/precompress.mjs` (build step,
node:zlib only) emits `.br`/`.gz` siblings for text assets; `ServeDir` now
uses `precompressed_br()/precompressed_gzip()` — a request costs a file
read, and clients get the *better* q11 bytes, paid once per deploy
(~1.4 s for the whole bundle).
2. **Explicit level 4** on both `CompressionLayer`s
(`CompressionLevel::Precise(4)`) — the on-the-fly fallback for statics
without siblings, and the global API layer.
## Reproduce
```bash
cargo run --release --features bench --example bench_static_precompress
# tunables: BENCH_ASSET_KB=700 BENCH_REPS=30
```
## Results (4 cores, this container)
**Per-request cost, 700 KiB JS-like asset (94 % compressible):**
| mode | ms/request | speedup |
|-----------------------------|-----------:|--------:|
| BEFORE — on-the-fly Brotli | 1,324.31 | 1.0× |
| AFTER — precompressed read | 0.657 | **2016×** |
**Brotli level sweep, 64 KiB JSON-like API response:**
| level | ms/resp | out KiB |
|-------------------------|--------:|--------:|
| Default (= quality 11!) | 90.10 | 5.4 |
| **Precise(4)** (chosen) | 0.91 | 6.2 |
| Fastest | 0.15 | 9.3 |
- Statics: 3 orders of magnitude less CPU per request, while shipping
*smaller* bytes than the runtime default would at any reasonable level.
- API responses: **99× less CPU** for ~15 % more bytes (5.4 → 6.2 KiB) —
`Precise(4)` is the classic dynamic-content operating point; `Fastest`
gives up too much density (9.3 KiB).
- Historical note: an earlier review round REFUTED the "default is q11"
claim twice; the source line and the 90 ms/64 KiB measurement above settle
it the other way. Measure before believing — in both directions.
benches/ZIP-MEDIA.md
+44
View File
@@ -0,0 +1,44 @@
# ZIP export — Stored for already-compressed media (vs Deflate-always)
Every ZIP export path (`ZipService::create_folder_zip` for folder downloads +
public share ZIPs, `BatchOperations::download_zip` for batch downloads) used to
build **every** file entry with `Compression::Deflate`. The dominant "download
folder" payload is photos/video (JPEG/HEIC/MP4/WebP), which deflate cannot
shrink (~0 %) while costing ~40 MB/s of CPU per core — and `async_zip` runs
deflate **inline on the writing tokio task** (inside `poll_write`), so a media
folder download monopolised ~1 core for its whole duration.
The change picks the entry compression from the file's MIME type at plan time:
`Stored` for already-compressed content, `Deflate` otherwise. The shared
predicate is `common::mime_detect::is_precompressed_mime` /
`zip_entry_compression` — it mirrors the HTTP `CompressionLayer` exclusion
list in `main.rs` (keep in sync), minus `x-tar`/`octet-stream` (containers of
possibly-compressible data stay on Deflate so nothing ever gets bigger).
## Reproduce
```bash
cargo run --release --features bench --example bench_zip_media
# tunables: BENCH_MEDIA_FILES=48 BENCH_MEDIA_MB=4 BENCH_TEXT_FILES=24 BENCH_TEXT_MB=2 BENCH_REPS=3
```
Rebuilds the exact production writer stack (`ZipFileWriter::with_tokio(BufWriter(File))`,
`write_entry_stream`, 64 KiB chunks) over a mixed corpus: 192 MiB incompressible
"media" + 48 MiB compressible text (80/20 by bytes, a realistic media folder).
## Results (4 cores, this container)
| mode | wall s | cpu s | MB/s | out MiB | speedup |
|-----------------------|-------:|------:|-------:|--------:|--------:|
| all-Deflate (BEFORE) | 5.786 | 5.88 | 41.5 | 198.8 | 1.00× |
| mime-aware (AFTER) | 1.341 | 1.38 | 178.9 | 198.7 | **4.31×** |
| all-Stored (bound) | 0.150 | 0.19 | 1601.5 | 240.0 | 38.6× |
- **4.31× faster wall clock and 4.3× less CPU** on the mixed corpus, with the
archive **0.05 % smaller** (media never deflated anyway; text keeps Deflate).
- The remaining 1.38 s CPU in mime-aware is the text deflate + CRC32 — the
irreducible part. Pure-media folders approach the all-Stored bound (the
archive becomes blob-read-bound instead of CPU-bound).
- Side effect on the runtime: the writing task no longer occupies ~a full core
per media download — on a 4-core box that's ~25 % of total CPU handed back
to other requests for the duration of every archive.