perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -0,0 +1,250 @@
|
||||
//! NC chroot / default-drive resolution benchmark — 2 queries/request vs moka.
|
||||
//!
|
||||
//! The NextCloud basic-auth middleware wraps EVERY protected NC route and,
|
||||
//! even with app-password verification fully cached, used to resolve the
|
||||
//! chroot from scratch per request:
|
||||
//!
|
||||
//! 1. `find_default_for_user` — drives JOIN folders (drive_pg_repository)
|
||||
//! 2. `get_folder(root_id)` — folders by PK
|
||||
//!
|
||||
//! The native `/webdav` surface repeats query 1 per request (Mode-B scope
|
||||
//! resolution), WOPI repeats it per call. The change memoises (1) inside
|
||||
//! `DrivePgRepository` and (2) in the middleware's `NC_CHROOT_CACHE`
|
||||
//! (both 30 s TTL). This bench isolates exactly that: the per-request DB
|
||||
//! cost of the chroot resolution — the two production query shapes vs a
|
||||
//! moka hit — under sync-storm concurrency against the real pool.
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_chroot_cache
|
||||
//! Tunables (env): BENCH_POOL (20), BENCH_SECONDS (4), BENCH_CONCURRENCIES ("8,64").
|
||||
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use sqlx::{PgPool, Row};
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Seeded {
|
||||
user_id: Uuid,
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool) -> Seeded {
|
||||
// user → (drive + root folder + root_folder_id stamp) in one tx —
|
||||
// trg_no_orphan_root_folder is INITIALLY DEFERRED and checks at commit.
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let user_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, role)
|
||||
VALUES ('bench_chroot', 'bench_chroot@bench.invalid', 'user')
|
||||
RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed user");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, default_for_user) VALUES ('personal', $1) RETURNING id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('Personal', '/Personal', 'Personal', $1) RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp root");
|
||||
tx.commit().await.expect("commit");
|
||||
Seeded { user_id }
|
||||
}
|
||||
|
||||
async fn cleanup(pool: &PgPool, user_id: Uuid) {
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id = $1")
|
||||
.bind(user_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// The exact production BEFORE: both chroot queries, sequentially (the
|
||||
/// middleware awaits the drive row to learn root_folder_id first).
|
||||
async fn one_op_before(pool: &PgPool, user_id: Uuid, queries: &AtomicUsize) {
|
||||
let row = sqlx::query(
|
||||
r#"
|
||||
SELECT d.id, d.kind, d.default_for_user, d.root_folder_id,
|
||||
d.quota_bytes, d.used_bytes, d.policies,
|
||||
d.created_at, d.updated_at,
|
||||
f.name AS root_folder_name
|
||||
FROM storage.drives d
|
||||
JOIN storage.folders f ON f.id = d.root_folder_id
|
||||
WHERE d.default_for_user = $1
|
||||
"#,
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("drive query");
|
||||
let root_id: Uuid = row.get("root_folder_id");
|
||||
|
||||
let _folder = sqlx::query(
|
||||
"SELECT id, name, parent_id, path, created_at, updated_at
|
||||
FROM storage.folders WHERE id = $1",
|
||||
)
|
||||
.bind(root_id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("folder query");
|
||||
queries.fetch_add(2, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
#[allow(dead_code)]
|
||||
struct ChrootValue {
|
||||
root_id: Uuid,
|
||||
name: String,
|
||||
path: String,
|
||||
}
|
||||
|
||||
struct Stats {
|
||||
rps: f64,
|
||||
p50: f64,
|
||||
p95: f64,
|
||||
p99: f64,
|
||||
}
|
||||
|
||||
fn summarize(mut lats: Vec<f64>, secs: u64) -> Stats {
|
||||
lats.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
let n = lats.len();
|
||||
let pct = |p: f64| {
|
||||
if n == 0 {
|
||||
0.0
|
||||
} else {
|
||||
lats[((n as f64 * p) as usize).min(n - 1)]
|
||||
}
|
||||
};
|
||||
Stats {
|
||||
rps: n as f64 / secs as f64,
|
||||
p50: pct(0.50),
|
||||
p95: pct(0.95),
|
||||
p99: pct(0.99),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL — the dev Postgres URL");
|
||||
|
||||
let pool_size: u32 = env_or("BENCH_POOL", 20);
|
||||
let secs: u64 = env_or("BENCH_SECONDS", 4);
|
||||
let concurrencies: Vec<usize> = env::var("BENCH_CONCURRENCIES")
|
||||
.ok()
|
||||
.map(|s| s.split(',').filter_map(|x| x.trim().parse().ok()).collect())
|
||||
.unwrap_or_else(|| vec![8, 64]);
|
||||
|
||||
let pool = Arc::new(
|
||||
PgPoolOptions::new()
|
||||
.max_connections(pool_size)
|
||||
.min_connections(pool_size)
|
||||
.acquire_timeout(Duration::from_secs(10))
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect Postgres"),
|
||||
);
|
||||
|
||||
let seeded = seed(&pool).await;
|
||||
let user_id = seeded.user_id;
|
||||
|
||||
// AFTER: what the middleware pays on a warm cache — a moka lookup.
|
||||
let cache: moka::sync::Cache<Uuid, ChrootValue> = moka::sync::Cache::builder()
|
||||
.max_capacity(100_000)
|
||||
.time_to_live(Duration::from_secs(30))
|
||||
.build();
|
||||
cache.insert(
|
||||
user_id,
|
||||
ChrootValue {
|
||||
root_id: Uuid::new_v4(),
|
||||
name: "Personal".into(),
|
||||
path: "/Personal".into(),
|
||||
},
|
||||
);
|
||||
|
||||
println!("\n#############################################################");
|
||||
println!("# NC chroot resolution: BEFORE (2 queries/req) vs AFTER (moka)");
|
||||
println!("# pool={pool_size} window={secs}s/run");
|
||||
println!("#############################################################\n");
|
||||
println!(
|
||||
"| {:>5} | {:<6} | {:>10} | {:>9} | {:>9} | {:>9} | {:>9} |",
|
||||
"conc", "mode", "req/s", "p50 µs", "p95 µs", "p99 µs", "queries"
|
||||
);
|
||||
|
||||
for &conc in &concurrencies {
|
||||
for mode in ["BEFORE", "AFTER"] {
|
||||
let queries = Arc::new(AtomicUsize::new(0));
|
||||
let deadline = Instant::now() + Duration::from_secs(secs);
|
||||
let mut handles = Vec::new();
|
||||
for _ in 0..conc {
|
||||
let pool = pool.clone();
|
||||
let cache = cache.clone();
|
||||
let queries = queries.clone();
|
||||
let mode = mode.to_string();
|
||||
handles.push(tokio::spawn(async move {
|
||||
let mut lats = Vec::new();
|
||||
while Instant::now() < deadline {
|
||||
let t = Instant::now();
|
||||
if mode == "BEFORE" {
|
||||
one_op_before(&pool, user_id, &queries).await;
|
||||
} else {
|
||||
let v = cache.get(&user_id).expect("warm cache");
|
||||
std::hint::black_box(v);
|
||||
}
|
||||
lats.push(t.elapsed().as_secs_f64() * 1_000_000.0);
|
||||
if mode == "AFTER" {
|
||||
// moka hit is ~100 ns; yield so the loop doesn't
|
||||
// monopolise workers and skew the run count.
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
}
|
||||
lats
|
||||
}));
|
||||
}
|
||||
let mut all = Vec::new();
|
||||
for h in handles {
|
||||
all.extend(h.await.unwrap());
|
||||
}
|
||||
let s = summarize(all, secs);
|
||||
println!(
|
||||
"| {:>5} | {:<6} | {:>10.0} | {:>9.2} | {:>9.2} | {:>9.2} | {:>9} |",
|
||||
conc,
|
||||
mode,
|
||||
s.rps,
|
||||
s.p50,
|
||||
s.p95,
|
||||
s.p99,
|
||||
queries.load(Ordering::Relaxed)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
cleanup(&pool, user_id).await;
|
||||
println!("\n(BEFORE = the two production chroot queries; AFTER = warm moka hit.");
|
||||
println!(" Every NC request pays this before its handler runs.)");
|
||||
}
|
||||
@@ -0,0 +1,286 @@
|
||||
//! WebDAV dead-properties fetch benchmark — per-child N+1 vs batched ANY($1).
|
||||
//!
|
||||
//! The streaming PROPFIND walker (`webdav_handler.rs`) fetches dead properties
|
||||
//! ONE CHILD AT A TIME, sequentially, for every Depth:1 listing page:
|
||||
//!
|
||||
//! for file in &batch { file_deads.push(store.get_all(File(id)).await) }
|
||||
//!
|
||||
//! and `DeadPropertyStore::get_all` filters with
|
||||
//! `folder_id IS NOT DISTINCT FROM $1 AND file_id IS NOT DISTINCT FROM $2`,
|
||||
//! which PostgreSQL cannot serve from a B-tree index (IS NOT DISTINCT FROM is
|
||||
//! not an indexable operator) — so each of the N sequential round-trips also
|
||||
//! degrades to a seq scan as the table grows.
|
||||
//!
|
||||
//! This bench isolates exactly the dead-prop portion of a Depth:1 PROPFIND of
|
||||
//! a folder with N children, comparing the three query shapes:
|
||||
//!
|
||||
//! OLD — N sequential `IS NOT DISTINCT FROM` queries (production today)
|
||||
//! EQ — N sequential plain `file_id = $1` queries (indexable, still N+1)
|
||||
//! BATCH — ⌈N/500⌉ `file_id = ANY($1)` queries (one per PROPFIND page)
|
||||
//!
|
||||
//! Two table sizes are measured: the seeded-children-only table and one with
|
||||
//! extra noise rows (dead props on other resources), which is where the
|
||||
//! seq-scan cost of OLD shows up.
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_dead_props
|
||||
//! Tunables (env): BENCH_CHILDREN (2000), BENCH_PAGE (500 = PROPFIND_BATCH_SIZE),
|
||||
//! BENCH_NOISE_ROWS (20000), BENCH_REPS (5).
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use sqlx::{PgPool, Row};
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Seeded {
|
||||
drive_id: Uuid,
|
||||
file_ids: Vec<Uuid>,
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, children: usize, noise: usize) -> Seeded {
|
||||
// Drive (kind 'shared' needs no user FK) → root folder → N files → props.
|
||||
// The root folder + drive.root_folder_id must land in ONE transaction:
|
||||
// trg_no_orphan_root_folder is INITIALLY DEFERRED and checks at commit.
|
||||
let mut tx = pool.begin().await.expect("begin seed tx");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes) VALUES ('shared', NULL) RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed drive");
|
||||
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_dead_props', '/bench_dead_props', 'bench_dead_props', $1)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("seed folder");
|
||||
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp root_folder_id");
|
||||
tx.commit().await.expect("commit seed tx");
|
||||
|
||||
// Children of the PROPFIND'd folder, one dead prop each.
|
||||
let file_ids: Vec<Uuid> = sqlx::query_scalar(
|
||||
"INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||
SELECT 'f' || i, $1, 'benchdead000000000000000000000000000000000000000000000000000000',
|
||||
1024, 'image/jpeg', $2
|
||||
FROM generate_series(1, $3) AS i
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(children as i32)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("seed files");
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.webdav_dead_properties (file_id, namespace, local_name, value)
|
||||
SELECT id, 'urn:bench', 'displayname', 'bench value'
|
||||
FROM storage.files WHERE folder_id = $1",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("seed dead props");
|
||||
|
||||
// Noise: dead props attached to OTHER files (a second folder) so the
|
||||
// table has realistic volume — this is what OLD's seq scans pay for.
|
||||
if noise > 0 {
|
||||
// Child of the main folder — root folders need the deferred
|
||||
// four-write dance, children don't.
|
||||
let noise_folder: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, parent_id, path, lpath, drive_id)
|
||||
VALUES ('noise', $2, '/bench_dead_props/noise', 'bench_dead_props.noise', $1)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.bind(folder_id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("seed noise folder");
|
||||
sqlx::query(
|
||||
"WITH f AS (
|
||||
INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||
SELECT 'n' || i, $1, 'benchdead000000000000000000000000000000000000000000000000000000',
|
||||
1024, 'image/jpeg', $2
|
||||
FROM generate_series(1, $3) AS i
|
||||
RETURNING id
|
||||
)
|
||||
INSERT INTO storage.webdav_dead_properties (file_id, namespace, local_name, value)
|
||||
SELECT id, 'urn:bench', 'noise', 'x' FROM f",
|
||||
)
|
||||
.bind(noise_folder)
|
||||
.bind(drive_id)
|
||||
.bind(noise as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("seed noise props");
|
||||
}
|
||||
|
||||
sqlx::query("ANALYZE storage.webdav_dead_properties")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
|
||||
Seeded { drive_id, file_ids }
|
||||
}
|
||||
|
||||
async fn cleanup(pool: &PgPool, drive_id: Uuid) {
|
||||
// drives → folders/files → dead props all cascade.
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(drive_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// OLD: production `get_all` shape — sequential, IS NOT DISTINCT FROM.
|
||||
async fn run_old(pool: &PgPool, ids: &[Uuid]) -> usize {
|
||||
let mut rows_seen = 0;
|
||||
for id in ids {
|
||||
let rows = sqlx::query(
|
||||
"SELECT namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2",
|
||||
)
|
||||
.bind(Option::<Uuid>::None)
|
||||
.bind(Some(*id))
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("old get_all");
|
||||
rows_seen += rows.len();
|
||||
}
|
||||
rows_seen
|
||||
}
|
||||
|
||||
/// EQ: still N sequential round-trips, but with an indexable `=` predicate.
|
||||
async fn run_eq(pool: &PgPool, ids: &[Uuid]) -> usize {
|
||||
let mut rows_seen = 0;
|
||||
for id in ids {
|
||||
let rows = sqlx::query(
|
||||
"SELECT namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE file_id = $1",
|
||||
)
|
||||
.bind(*id)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("eq get_all");
|
||||
rows_seen += rows.len();
|
||||
}
|
||||
rows_seen
|
||||
}
|
||||
|
||||
/// BATCH: one `= ANY($1)` query per PROPFIND page of 500 children.
|
||||
async fn run_batch(pool: &PgPool, ids: &[Uuid], page: usize) -> usize {
|
||||
let mut rows_seen = 0;
|
||||
for chunk in ids.chunks(page) {
|
||||
let rows = sqlx::query(
|
||||
"SELECT file_id, namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE file_id = ANY($1)",
|
||||
)
|
||||
.bind(chunk)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("batch get_all");
|
||||
// Decode file_id like the real batched store method will (map key).
|
||||
for row in &rows {
|
||||
let _: Uuid = row.get("file_id");
|
||||
}
|
||||
rows_seen += rows.len();
|
||||
}
|
||||
rows_seen
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL — the dev Postgres URL");
|
||||
|
||||
let children: usize = env_or("BENCH_CHILDREN", 2000);
|
||||
let page: usize = env_or("BENCH_PAGE", 500);
|
||||
let noise: usize = env_or("BENCH_NOISE_ROWS", 20_000);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.min_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect Postgres");
|
||||
|
||||
for &with_noise in &[false, true] {
|
||||
let n = if with_noise { noise } else { 0 };
|
||||
let seeded = seed(&pool, children, n).await;
|
||||
let total_rows: i64 =
|
||||
sqlx::query_scalar("SELECT count(*) FROM storage.webdav_dead_properties")
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap_or(0);
|
||||
|
||||
println!("\n== folder with {children} children, dead-props table = {total_rows} rows ==");
|
||||
println!(
|
||||
"{:<28} {:>10} {:>12} {:>9}",
|
||||
"mode", "queries", "total ms", "vs OLD"
|
||||
);
|
||||
|
||||
let mut base = None;
|
||||
for (label, queries) in [
|
||||
("OLD seq, IS NOT DISTINCT", children),
|
||||
("EQ seq, file_id = $1", children),
|
||||
("BATCH file_id = ANY, /page", children.div_ceil(page)),
|
||||
] {
|
||||
let mut times = Vec::with_capacity(reps);
|
||||
let mut rows = 0;
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
rows = match label.split_whitespace().next().unwrap() {
|
||||
"OLD" => run_old(&pool, &seeded.file_ids).await,
|
||||
"EQ" => run_eq(&pool, &seeded.file_ids).await,
|
||||
_ => run_batch(&pool, &seeded.file_ids, page).await,
|
||||
};
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
assert_eq!(rows, children, "each child has exactly 1 dead prop");
|
||||
let ms = median(times);
|
||||
let speedup = base
|
||||
.map(|b: f64| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!("{label:<28} {queries:>10} {ms:>12.2} {speedup:>9}");
|
||||
}
|
||||
|
||||
cleanup(&pool, seeded.drive_id).await;
|
||||
}
|
||||
|
||||
println!("\n(total ms = the dead-prop portion of one Depth:1 PROPFIND of the folder,");
|
||||
println!(" i.e. what the walker adds on top of the file/folder listing queries)");
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
//! People-tab benchmark — full faces scan (embeddings included) vs grouped COUNT.
|
||||
//!
|
||||
//! `PeopleService::list_people` used to call `faces_for_user`, dragging every
|
||||
//! face row — each with a 2,048-byte embedding BYTEA — across the wire and
|
||||
//! decoding it into a fresh `Vec<f32>`, only to (a) count faces per person and
|
||||
//! (b) resolve ~a-handful of cover faces to file ids. The change replaces it
|
||||
//! with `person_face_stats` (grouped COUNT) + `file_ids_for_faces` (one
|
||||
//! `= ANY` over just the cover ids).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_people_list
|
||||
//! Tunables: BENCH_FACES (10000), BENCH_PERSONS (20), BENCH_REPS (5)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Seeded {
|
||||
user_id: Uuid,
|
||||
drive_id: Uuid,
|
||||
cover_ids: Vec<Uuid>,
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, faces: usize, persons: usize) -> Seeded {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let user_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, role)
|
||||
VALUES ('bench_people', 'bench_people@bench.invalid', 'user') RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("user");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, default_for_user) VALUES ('personal', $1) RETURNING id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_people', '/bench_people', 'bench_people', $1) RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp root");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
// Photo files the faces point at.
|
||||
let file_ids: Vec<Uuid> = sqlx::query_scalar(
|
||||
"INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||
SELECT 'p' || i, $1, 'benchpeople0000000000000000000000000000000000000000000000000000',
|
||||
1024, 'image/jpeg', $2
|
||||
FROM generate_series(1, $3) AS i
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(faces as i32)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
|
||||
// Persons + faces (2 KiB embedding each, like the real 512×f32).
|
||||
let mut person_ids = Vec::with_capacity(persons);
|
||||
for i in 0..persons {
|
||||
let pid: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO faces.persons (user_id, display_name) VALUES ($1, $2) RETURNING id",
|
||||
)
|
||||
.bind(user_id)
|
||||
.bind(format!("Person {i}"))
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("person");
|
||||
person_ids.push(pid);
|
||||
}
|
||||
|
||||
let embedding = vec![0u8; 2048];
|
||||
let mut cover_ids = Vec::with_capacity(persons);
|
||||
for (i, file_id) in file_ids.iter().enumerate() {
|
||||
let pid = person_ids[i % persons];
|
||||
let face_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO faces.faces
|
||||
(file_id, user_id, person_id, bbox, det_score, quality, embedding, blob_hash)
|
||||
VALUES ($1, $2, $3, ARRAY[0.1,0.1,0.2,0.2]::real[], 0.99, 0.9, $4,
|
||||
'benchpeople0000000000000000000000000000000000000000000000000000')
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(file_id)
|
||||
.bind(user_id)
|
||||
.bind(pid)
|
||||
.bind(&embedding)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("face");
|
||||
if i < persons {
|
||||
cover_ids.push(face_id);
|
||||
}
|
||||
}
|
||||
sqlx::query("ANALYZE faces.faces").execute(pool).await.ok();
|
||||
|
||||
Seeded {
|
||||
user_id,
|
||||
drive_id,
|
||||
cover_ids,
|
||||
}
|
||||
}
|
||||
|
||||
async fn cleanup(pool: &PgPool, s: &Seeded) {
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(s.drive_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id = $1")
|
||||
.bind(s.user_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let faces: usize = env_or("BENCH_FACES", 10_000);
|
||||
let persons: usize = env_or("BENCH_PERSONS", 20);
|
||||
let reps: usize = env_or("BENCH_REPS", 5);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {faces} faces / {persons} persons (one-time)…");
|
||||
let seeded = seed(&pool, faces, persons).await;
|
||||
|
||||
println!(
|
||||
"\n# GET /api/people data fetch: BEFORE (full face rows) vs AFTER (COUNT + cover ANY)"
|
||||
);
|
||||
println!("{:<28} {:>12} {:>14}", "mode", "total ms", "bytes moved");
|
||||
|
||||
let mut base = None;
|
||||
for mode in ["BEFORE full-rows", "AFTER count+covers"] {
|
||||
let mut times = Vec::with_capacity(reps);
|
||||
let mut bytes = 0usize;
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
if mode.starts_with("BEFORE") {
|
||||
// faces_for_user shape: every column incl. embedding.
|
||||
let rows: Vec<(Uuid, Uuid, Option<Uuid>, Vec<u8>)> = sqlx::query_as(
|
||||
"SELECT id, file_id, person_id, embedding FROM faces.faces WHERE user_id = $1",
|
||||
)
|
||||
.bind(seeded.user_id)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.expect("full rows");
|
||||
bytes = rows.iter().map(|r| r.3.len() + 48).sum();
|
||||
assert_eq!(rows.len(), faces);
|
||||
} else {
|
||||
let stats: Vec<(Uuid, i64)> = sqlx::query_as(
|
||||
"SELECT person_id, COUNT(*) FROM faces.faces
|
||||
WHERE user_id = $1 AND person_id IS NOT NULL GROUP BY person_id",
|
||||
)
|
||||
.bind(seeded.user_id)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.expect("stats");
|
||||
let covers: Vec<(Uuid, Uuid)> = sqlx::query_as(
|
||||
"SELECT id, file_id FROM faces.faces WHERE user_id = $1 AND id = ANY($2)",
|
||||
)
|
||||
.bind(seeded.user_id)
|
||||
.bind(&seeded.cover_ids)
|
||||
.fetch_all(&pool)
|
||||
.await
|
||||
.expect("covers");
|
||||
bytes = (stats.len() + covers.len()) * 32;
|
||||
assert_eq!(stats.len(), persons);
|
||||
}
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
let ms = median(times);
|
||||
let speedup = base
|
||||
.map(|b: f64| format!("({:.1}x)", b / ms))
|
||||
.unwrap_or_default();
|
||||
println!("{mode:<28} {ms:>12.2} {bytes:>14} {speedup}");
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
}
|
||||
|
||||
cleanup(&pool, &seeded).await;
|
||||
}
|
||||
@@ -0,0 +1,233 @@
|
||||
//! PROPFIND folder-listing pagination benchmark — LIMIT/OFFSET vs keyset.
|
||||
//!
|
||||
//! The streaming PROPFIND walker pages a folder's children 500 at a time in
|
||||
//! name order (`list_files_batch`). The old shape was `ORDER BY name LIMIT
|
||||
//! 500 OFFSET k` with no supporting index — every page bitmap-scanned all N
|
||||
//! children and top-sorted them, so a full folder walk was O(N²/500) row
|
||||
//! visits. The change adds `idx_files_folder_name (folder_id, name) WHERE
|
||||
//! NOT is_trashed` and switches the cursor to keyset (`name > $last`), making
|
||||
//! each page one O(page) index-range read.
|
||||
//!
|
||||
//! Modes (full walk of the folder, all pages):
|
||||
//! OFFSET/no-idx — the true BEFORE (index dropped for the run)
|
||||
//! OFFSET/idx — index alone, old query shape
|
||||
//! KEYSET/idx — the AFTER
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_propfind_paging
|
||||
//! Tunables: BENCH_FILES (20000), BENCH_PAGE (500), BENCH_REPS (3)
|
||||
|
||||
use std::env;
|
||||
use std::time::Instant;
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, files: usize) -> (Uuid, Uuid) {
|
||||
let mut tx = pool.begin().await.expect("begin");
|
||||
let drive_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.drives (kind, quota_bytes) VALUES ('shared', NULL) RETURNING id",
|
||||
)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("drive");
|
||||
let folder_id: Uuid = sqlx::query_scalar(
|
||||
"INSERT INTO storage.folders (name, path, lpath, drive_id)
|
||||
VALUES ('bench_paging', '/bench_paging', 'bench_paging', $1) RETURNING id",
|
||||
)
|
||||
.bind(drive_id)
|
||||
.fetch_one(&mut *tx)
|
||||
.await
|
||||
.expect("folder");
|
||||
sqlx::query("UPDATE storage.drives SET root_folder_id = $1 WHERE id = $2")
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.expect("stamp");
|
||||
tx.commit().await.expect("commit");
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO storage.files (name, folder_id, blob_hash, size, mime_type, drive_id)
|
||||
SELECT 'file_' || LPAD(i::text, 8, '0') || '.jpg', $1,
|
||||
'benchpaging00000000000000000000000000000000000000000000000000000',
|
||||
1024, 'image/jpeg', $2
|
||||
FROM generate_series(1, $3) AS i",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(drive_id)
|
||||
.bind(files as i32)
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("files");
|
||||
sqlx::query("ANALYZE storage.files")
|
||||
.execute(pool)
|
||||
.await
|
||||
.ok();
|
||||
(drive_id, folder_id)
|
||||
}
|
||||
|
||||
const COLS: &str = "fi.id::text, fi.name, fi.folder_id::text, fo.path, fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint, fi.blob_hash";
|
||||
|
||||
type Row = (
|
||||
String,
|
||||
String,
|
||||
Option<String>,
|
||||
Option<String>,
|
||||
i64,
|
||||
String,
|
||||
i64,
|
||||
i64,
|
||||
String,
|
||||
);
|
||||
|
||||
/// Full folder walk with the old LIMIT/OFFSET shape. Returns rows seen.
|
||||
async fn walk_offset(pool: &PgPool, folder: Uuid, page: i64) -> usize {
|
||||
let mut offset = 0i64;
|
||||
let mut seen = 0usize;
|
||||
loop {
|
||||
let rows: Vec<Row> = sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id = $1 AND NOT fi.is_trashed
|
||||
ORDER BY fi.name LIMIT $2 OFFSET $3"
|
||||
))
|
||||
.bind(folder)
|
||||
.bind(page)
|
||||
.bind(offset)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
.expect("offset page");
|
||||
let n = rows.len();
|
||||
seen += n;
|
||||
if (n as i64) < page {
|
||||
break;
|
||||
}
|
||||
offset += n as i64;
|
||||
}
|
||||
seen
|
||||
}
|
||||
|
||||
/// Full folder walk with the new keyset shape.
|
||||
async fn walk_keyset(pool: &PgPool, folder: Uuid, page: i64) -> usize {
|
||||
let mut after: Option<String> = None;
|
||||
let mut seen = 0usize;
|
||||
loop {
|
||||
let rows: Vec<Row> = if let Some(a) = &after {
|
||||
sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id = $1 AND NOT fi.is_trashed AND fi.name > $3
|
||||
ORDER BY fi.name LIMIT $2"
|
||||
))
|
||||
.bind(folder)
|
||||
.bind(page)
|
||||
.bind(a)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
} else {
|
||||
sqlx::query_as(&format!(
|
||||
"SELECT {COLS}
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
WHERE fi.folder_id = $1 AND NOT fi.is_trashed
|
||||
ORDER BY fi.name LIMIT $2"
|
||||
))
|
||||
.bind(folder)
|
||||
.bind(page)
|
||||
.fetch_all(pool)
|
||||
.await
|
||||
}
|
||||
.expect("keyset page");
|
||||
let n = rows.len();
|
||||
seen += n;
|
||||
if (n as i64) < page {
|
||||
break;
|
||||
}
|
||||
after = rows.last().map(|r| r.1.clone());
|
||||
}
|
||||
seen
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let files: usize = env_or("BENCH_FILES", 20_000);
|
||||
let page: i64 = env_or("BENCH_PAGE", 500);
|
||||
let reps: usize = env_or("BENCH_REPS", 3);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect");
|
||||
println!("seeding {files} files (one-time)…");
|
||||
let (drive_id, folder_id) = seed(&pool, files).await;
|
||||
|
||||
println!("\n# full PROPFIND walk of a {files}-file folder, {page}/page");
|
||||
println!("{:<18} {:>12} {:>9}", "mode", "total ms", "vs OLD");
|
||||
|
||||
let mut base = None;
|
||||
for mode in ["OFFSET/no-idx", "OFFSET/idx", "KEYSET/idx"] {
|
||||
match mode {
|
||||
"OFFSET/no-idx" => {
|
||||
sqlx::query("DROP INDEX IF EXISTS storage.idx_files_folder_name")
|
||||
.execute(&pool)
|
||||
.await
|
||||
.ok();
|
||||
}
|
||||
"OFFSET/idx" => {
|
||||
sqlx::query(
|
||||
"CREATE INDEX IF NOT EXISTS idx_files_folder_name
|
||||
ON storage.files (folder_id, name) WHERE NOT is_trashed",
|
||||
)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.expect("create index");
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
let mut times = Vec::with_capacity(reps);
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let seen = if mode.starts_with("OFFSET") {
|
||||
walk_offset(&pool, folder_id, page).await
|
||||
} else {
|
||||
walk_keyset(&pool, folder_id, page).await
|
||||
};
|
||||
assert_eq!(seen, files);
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
let ms = median(times);
|
||||
let speedup = base
|
||||
.map(|b: f64| format!("{:.1}x", b / ms))
|
||||
.unwrap_or_else(|| "1.0x".into());
|
||||
if base.is_none() {
|
||||
base = Some(ms);
|
||||
}
|
||||
println!("{mode:<18} {ms:>12.1} {speedup:>9}");
|
||||
}
|
||||
|
||||
let _ = sqlx::query("DELETE FROM storage.drives WHERE id = $1")
|
||||
.bind(drive_id)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
//! Quota-path benchmark — full `auth.users` row vs narrow 2-column read.
|
||||
//!
|
||||
//! `check_storage_quota` (every upload) and `get_user_storage_info` (every
|
||||
//! quota-reporting PROPFIND) used to call `get_user_by_id`, whose SELECT
|
||||
//! drags the whole user row — including `image`, an avatar data URI of up
|
||||
//! to 512 KiB — across the wire to read two i64s. The change reads only
|
||||
//! `(storage_used_bytes, storage_quota_bytes)`
|
||||
//! (`UserPgRepository::get_storage_usage`). Companion change measured here
|
||||
//! as "SKIP": PROPFINDs whose prop list never names a quota prop now skip
|
||||
//! the resolution entirely (`PropFindRequest::wants_quota`).
|
||||
//!
|
||||
//! Run (needs Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_quota_path
|
||||
//! Tunables: BENCH_SECONDS (4), BENCH_CONCURRENCIES ("8,64"), BENCH_IMAGE_KB (512)
|
||||
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use sqlx::PgPool;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
async fn seed(pool: &PgPool, image_kb: usize) -> Uuid {
|
||||
// Realistic worst-ish case: an avatar data URI at the documented cap.
|
||||
let image = format!("data:image/png;base64,{}", "A".repeat(image_kb * 1024 - 22));
|
||||
sqlx::query_scalar(
|
||||
"INSERT INTO auth.users (username, email, role, image)
|
||||
VALUES ('bench_quota', 'bench_quota@bench.invalid', 'user', $1)
|
||||
RETURNING id",
|
||||
)
|
||||
.bind(&image)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("seed user")
|
||||
}
|
||||
|
||||
async fn cleanup(pool: &PgPool, user_id: Uuid) {
|
||||
let _ = sqlx::query("DELETE FROM auth.users WHERE id = $1")
|
||||
.bind(user_id)
|
||||
.execute(pool)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// BEFORE: the full-row SELECT `get_user_by_id` runs (same column list).
|
||||
async fn one_op_full(pool: &PgPool, id: Uuid) {
|
||||
let _row = sqlx::query(
|
||||
r#"
|
||||
SELECT
|
||||
id, username, email, password_hash, role::text as role_text,
|
||||
storage_quota_bytes, storage_used_bytes,
|
||||
created_at, updated_at, last_login_at, active,
|
||||
oidc_provider, oidc_subject, image, is_external,
|
||||
given_name, family_name, email_verified_at, preferred_locale, notify_on_share,
|
||||
ui_preferences
|
||||
FROM auth.users
|
||||
WHERE id = $1
|
||||
"#,
|
||||
)
|
||||
.bind(id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("full row");
|
||||
}
|
||||
|
||||
/// AFTER: the narrow `get_storage_usage` SELECT.
|
||||
async fn one_op_narrow(pool: &PgPool, id: Uuid) {
|
||||
let _row: (i64, i64) = sqlx::query_as(
|
||||
"SELECT storage_used_bytes, storage_quota_bytes FROM auth.users WHERE id = $1",
|
||||
)
|
||||
.bind(id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("narrow row");
|
||||
}
|
||||
|
||||
struct Stats {
|
||||
rps: f64,
|
||||
p50: f64,
|
||||
p99: f64,
|
||||
}
|
||||
|
||||
fn summarize(mut lats: Vec<f64>, secs: u64) -> Stats {
|
||||
lats.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
let n = lats.len();
|
||||
let pct = |p: f64| {
|
||||
if n == 0 {
|
||||
0.0
|
||||
} else {
|
||||
lats[((n as f64 * p) as usize).min(n - 1)]
|
||||
}
|
||||
};
|
||||
Stats {
|
||||
rps: n as f64 / secs as f64,
|
||||
p50: pct(0.50),
|
||||
p99: pct(0.99),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL").expect("set DATABASE_URL");
|
||||
let secs: u64 = env_or("BENCH_SECONDS", 4);
|
||||
let image_kb: usize = env_or("BENCH_IMAGE_KB", 512);
|
||||
let concurrencies: Vec<usize> = env::var("BENCH_CONCURRENCIES")
|
||||
.ok()
|
||||
.map(|s| s.split(',').filter_map(|x| x.trim().parse().ok()).collect())
|
||||
.unwrap_or_else(|| vec![8, 64]);
|
||||
|
||||
let pool = Arc::new(
|
||||
PgPoolOptions::new()
|
||||
.max_connections(20)
|
||||
.min_connections(20)
|
||||
.acquire_timeout(Duration::from_secs(10))
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect"),
|
||||
);
|
||||
let user_id = seed(&pool, image_kb).await;
|
||||
|
||||
println!("\n# quota lookup: full user row (incl. {image_kb} KiB avatar) vs 2-column read");
|
||||
println!(
|
||||
"| {:>5} | {:<7} | {:>10} | {:>9} | {:>9} |",
|
||||
"conc", "mode", "ops/s", "p50 µs", "p99 µs"
|
||||
);
|
||||
for &conc in &concurrencies {
|
||||
for mode in ["FULL", "NARROW"] {
|
||||
let deadline = Instant::now() + Duration::from_secs(secs);
|
||||
let mut handles = Vec::new();
|
||||
for _ in 0..conc {
|
||||
let pool = pool.clone();
|
||||
let mode = mode.to_string();
|
||||
handles.push(tokio::spawn(async move {
|
||||
let mut lats = Vec::new();
|
||||
while Instant::now() < deadline {
|
||||
let t = Instant::now();
|
||||
if mode == "FULL" {
|
||||
one_op_full(&pool, user_id).await;
|
||||
} else {
|
||||
one_op_narrow(&pool, user_id).await;
|
||||
}
|
||||
lats.push(t.elapsed().as_secs_f64() * 1e6);
|
||||
}
|
||||
lats
|
||||
}));
|
||||
}
|
||||
let mut all = Vec::new();
|
||||
for h in handles {
|
||||
all.extend(h.await.unwrap());
|
||||
}
|
||||
let s = summarize(all, secs);
|
||||
println!(
|
||||
"| {:>5} | {:<7} | {:>10.0} | {:>9.1} | {:>9.1} |",
|
||||
conc, mode, s.rps, s.p50, s.p99
|
||||
);
|
||||
}
|
||||
}
|
||||
println!("\n(SKIP: PROPFINDs not naming quota props now issue NEITHER query — 0 round-trips.)");
|
||||
|
||||
cleanup(&pool, user_id).await;
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
//! Static-asset compression benchmark — on-the-fly Brotli per request vs
|
||||
//! serving a precompressed sibling.
|
||||
//!
|
||||
//! The SPA router compressed every compressible static response on the fly
|
||||
//! (tower-http `CompressionLayer`, backed by `async-compression`'s Brotli at
|
||||
//! `Level::Default`) — the same immutable `/_app/immutable` bundle re-encoded
|
||||
//! on EVERY request. The change teaches `ServeDir` to serve build-time
|
||||
//! `.br`/`.gz` siblings (`precompressed_br()/precompressed_gzip()` +
|
||||
//! `frontend/scripts/precompress.mjs`), so a request costs a file read.
|
||||
//!
|
||||
//! This isolates exactly that per-request delta on a JS-bundle-like payload:
|
||||
//! BEFORE — Brotli-encode the asset with async-compression Level::Default
|
||||
//! (what the layer does per request)
|
||||
//! AFTER — read the precompressed sibling from disk (what ServeDir does)
|
||||
//!
|
||||
//! Run (no Postgres needed):
|
||||
//! cargo run --release --features bench --example bench_static_precompress
|
||||
//! Tunables: BENCH_ASSET_KB (700), BENCH_REPS (30)
|
||||
|
||||
use std::env;
|
||||
use std::io::Write as _;
|
||||
use std::time::Instant;
|
||||
|
||||
use tokio::io::AsyncReadExt;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// JS-like corpus: repetitive identifiers + literals, compresses like a real
|
||||
/// minified bundle (roughly 3-5×).
|
||||
fn synth_js(len: usize, seed: &mut u64) -> Vec<u8> {
|
||||
const FRAGS: &[&str] = &[
|
||||
"function(e,t,n){var r=this;",
|
||||
"return Object.assign({},",
|
||||
"const a=document.querySelector(",
|
||||
"export default{data(){return{",
|
||||
"await fetch(url,{method:'POST',headers:",
|
||||
".map(function(x){return x.id});",
|
||||
"if(void 0!==e&&null!==t){",
|
||||
"console.error('unhandled',err);",
|
||||
];
|
||||
let mut out = Vec::with_capacity(len);
|
||||
while out.len() < len {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
out.extend_from_slice(FRAGS[(*seed as usize) % FRAGS.len()].as_bytes());
|
||||
// sprinkle some varying identifiers so it's not pathological
|
||||
let _ = write!(out, "v{}", *seed % 1000);
|
||||
}
|
||||
out.truncate(len);
|
||||
out
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let asset_kb: usize = env_or("BENCH_ASSET_KB", 700);
|
||||
let reps: usize = env_or("BENCH_REPS", 30);
|
||||
let mut seed = 0xC0FFEEu64;
|
||||
let asset = synth_js(asset_kb * 1024, &mut seed);
|
||||
|
||||
// Precompress once (build-time cost, paid once per deploy).
|
||||
let dir = tempfile::tempdir().expect("tempdir");
|
||||
let br_path = dir.path().join("bundle.js.br");
|
||||
let t = Instant::now();
|
||||
let precompressed = {
|
||||
use async_compression::tokio::bufread::BrotliEncoder;
|
||||
let mut enc = BrotliEncoder::new(std::io::Cursor::new(asset.clone()));
|
||||
let mut out = Vec::new();
|
||||
enc.read_to_end(&mut out).await.expect("precompress");
|
||||
out
|
||||
};
|
||||
let build_ms = t.elapsed().as_secs_f64() * 1000.0;
|
||||
std::fs::write(&br_path, &precompressed).expect("write .br");
|
||||
|
||||
println!(
|
||||
"asset: {} KiB JS-like → {} KiB brotli ({}% smaller); one-time build cost {:.1} ms\n",
|
||||
asset.len() / 1024,
|
||||
precompressed.len() / 1024,
|
||||
100 - precompressed.len() * 100 / asset.len(),
|
||||
build_ms
|
||||
);
|
||||
|
||||
// BEFORE: per-request Brotli at the layer's default level.
|
||||
let mut enc_times = Vec::with_capacity(reps);
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
use async_compression::tokio::bufread::BrotliEncoder;
|
||||
let mut enc = BrotliEncoder::new(std::io::Cursor::new(asset.clone()));
|
||||
let mut out = Vec::new();
|
||||
enc.read_to_end(&mut out).await.expect("encode");
|
||||
std::hint::black_box(&out);
|
||||
enc_times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
|
||||
// AFTER: per-request read of the precompressed sibling.
|
||||
let mut read_times = Vec::with_capacity(reps);
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
let mut f = tokio::fs::File::open(&br_path).await.expect("open");
|
||||
let mut out = Vec::new();
|
||||
f.read_to_end(&mut out).await.expect("read");
|
||||
std::hint::black_box(&out);
|
||||
read_times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
|
||||
// ── Dynamic-response level sweep ─────────────────────────────────────
|
||||
// The global API CompressionLayer (main.rs) compresses JSON responses
|
||||
// per request. async-compression's Level::Default for Brotli is
|
||||
// QUALITY 11 (brotli-8.0.2 encode.rs:323 via compression-codecs) — a
|
||||
// deploy-grade setting on a per-request path. Sweep levels on a
|
||||
// JSON-like 64 KiB body to pick the runtime quality.
|
||||
let json_body = synth_js(64 * 1024, &mut seed); // JSON compresses like JS
|
||||
println!("\n# per-request Brotli level on a 64 KiB JSON-like API response");
|
||||
println!("{:<22} {:>10} {:>12}", "level", "ms/resp", "out KiB");
|
||||
for (label, level) in [
|
||||
("Default (= q11!)", async_compression::Level::Default),
|
||||
("Precise(4)", async_compression::Level::Precise(4)),
|
||||
("Fastest", async_compression::Level::Fastest),
|
||||
] {
|
||||
let mut times = Vec::with_capacity(reps);
|
||||
let mut out_len = 0;
|
||||
for _ in 0..reps {
|
||||
let t = Instant::now();
|
||||
use async_compression::tokio::bufread::BrotliEncoder;
|
||||
let mut enc =
|
||||
BrotliEncoder::with_quality(std::io::Cursor::new(json_body.clone()), level);
|
||||
let mut out = Vec::new();
|
||||
enc.read_to_end(&mut out).await.expect("encode");
|
||||
out_len = out.len();
|
||||
std::hint::black_box(&out);
|
||||
times.push(t.elapsed().as_secs_f64() * 1000.0);
|
||||
}
|
||||
println!(
|
||||
"{:<22} {:>10.2} {:>12.1}",
|
||||
label,
|
||||
median(times),
|
||||
out_len as f64 / 1024.0
|
||||
);
|
||||
}
|
||||
|
||||
let enc = median(enc_times);
|
||||
let read = median(read_times);
|
||||
println!(
|
||||
"{:<34} {:>10} {:>9}",
|
||||
"mode (per request)", "ms", "vs BEFORE"
|
||||
);
|
||||
println!(
|
||||
"{:<34} {:>10.2} {:>9}",
|
||||
"BEFORE on-the-fly Brotli", enc, "1.0x"
|
||||
);
|
||||
println!(
|
||||
"{:<34} {:>10.3} {:>8.0}x",
|
||||
"AFTER precompressed read",
|
||||
read,
|
||||
enc / read
|
||||
);
|
||||
println!("\n(BEFORE also holds ~1 tokio task busy for the duration on every request;");
|
||||
println!(" AFTER additionally ships the deploy-time q11 encoding, usually smaller than");
|
||||
println!(" the runtime default level.)");
|
||||
}
|
||||
@@ -0,0 +1,262 @@
|
||||
//! ZIP entry-compression benchmark — `Deflate`-always vs MIME-aware `Stored`.
|
||||
//!
|
||||
//! Isolates the ONE variable the ZIP-export change touches: the per-entry
|
||||
//! `Compression` mode chosen by `ZipService::write_prefetched_file` /
|
||||
//! `BatchOperations::add_file_entry_streamed`. It rebuilds the *exact*
|
||||
//! production writer stack —
|
||||
//!
|
||||
//! `ZipFileWriter::with_tokio(BufWriter(File))` + `write_entry_stream`
|
||||
//! fed in ~64 KiB chunks (the blob-stream chunk size)
|
||||
//!
|
||||
//! — and writes the same corpus once per mode, measuring wall time, process
|
||||
//! CPU time (utime+stime from `/proc/self/stat`), and final archive size.
|
||||
//!
|
||||
//! Corpora:
|
||||
//! • `media` — incompressible bytes (models JPEG/HEIC/MP4/WebP, the
|
||||
//! dominant "download folder" payload). Deflate here is pure CPU burn.
|
||||
//! • `text` — compressible text (models docs/source). Deflate genuinely
|
||||
//! shrinks these; the MIME-aware change keeps deflating them.
|
||||
//! • `mixed` — 80 % media / 20 % text by bytes: `all-Deflate` row is the
|
||||
//! production behaviour BEFORE the change; `mime-aware` row (Stored for
|
||||
//! media, Deflate for text) is AFTER.
|
||||
//!
|
||||
//! Run (no Postgres needed):
|
||||
//! cargo run --release --features bench --example bench_zip_media
|
||||
//! Tunables (env):
|
||||
//! BENCH_MEDIA_FILES (48) BENCH_MEDIA_MB (4) per-file size
|
||||
//! BENCH_TEXT_FILES (24) BENCH_TEXT_MB (2)
|
||||
//! BENCH_REPS (3) median reported
|
||||
|
||||
use std::env;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use async_zip::base::write::ZipFileWriter;
|
||||
use async_zip::{Compression, ZipEntryBuilder};
|
||||
use futures::io::AsyncWriteExt as FuturesWriteExt;
|
||||
use tokio::io::BufWriter;
|
||||
|
||||
const CHUNK: usize = 64 * 1024; // blob-stream chunk size on the real path
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
/// Process CPU seconds (user + system) from /proc/self/stat — covers all
|
||||
/// threads, so it catches deflate work wherever tokio schedules it.
|
||||
fn cpu_seconds() -> f64 {
|
||||
let stat = std::fs::read_to_string("/proc/self/stat").expect("read /proc/self/stat");
|
||||
// utime and stime are fields 14 and 15 (1-based), after the comm field
|
||||
// which may contain spaces — skip past the closing paren first.
|
||||
let after = &stat[stat.rfind(')').unwrap() + 2..];
|
||||
let fields: Vec<&str> = after.split_whitespace().collect();
|
||||
let utime: u64 = fields[11].parse().unwrap(); // field 14 overall
|
||||
let stime: u64 = fields[12].parse().unwrap(); // field 15 overall
|
||||
(utime + stime) as f64 / 100.0 // USER_HZ = 100 on Linux
|
||||
}
|
||||
|
||||
/// Deterministic xorshift64* stream — incompressible "media" bytes.
|
||||
fn fill_random(buf: &mut [u8], seed: &mut u64) {
|
||||
for chunk in buf.chunks_mut(8) {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
let bytes = seed.wrapping_mul(0x2545F4914F6CDD1D).to_le_bytes();
|
||||
let n = chunk.len();
|
||||
chunk.copy_from_slice(&bytes[..n]);
|
||||
}
|
||||
}
|
||||
|
||||
/// Compressible pseudo-text (~3-4× deflate ratio, like real docs/source).
|
||||
fn fill_text(buf: &mut [u8], seed: &mut u64) {
|
||||
const WORDS: &[&str] = &[
|
||||
"the",
|
||||
"quick",
|
||||
"brown",
|
||||
"fox",
|
||||
"jumps",
|
||||
"over",
|
||||
"lazy",
|
||||
"dog",
|
||||
"folder",
|
||||
"file",
|
||||
"storage",
|
||||
"performance",
|
||||
"benchmark",
|
||||
"archive",
|
||||
"download",
|
||||
"stream",
|
||||
];
|
||||
let mut pos = 0;
|
||||
while pos < buf.len() {
|
||||
*seed ^= *seed << 13;
|
||||
*seed ^= *seed >> 7;
|
||||
*seed ^= *seed << 17;
|
||||
let w = WORDS[(*seed as usize) % WORDS.len()].as_bytes();
|
||||
let n = w.len().min(buf.len() - pos);
|
||||
buf[pos..pos + n].copy_from_slice(&w[..n]);
|
||||
pos += n;
|
||||
if pos < buf.len() {
|
||||
buf[pos] = b' ';
|
||||
pos += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct CorpusFile {
|
||||
name: String,
|
||||
data: Vec<u8>,
|
||||
is_media: bool,
|
||||
}
|
||||
|
||||
struct RunResult {
|
||||
wall: Duration,
|
||||
cpu: f64,
|
||||
bytes_out: u64,
|
||||
}
|
||||
|
||||
/// Write the corpus through the exact production writer stack, choosing the
|
||||
/// compression mode per entry with `pick`.
|
||||
async fn write_zip(files: &[CorpusFile], pick: impl Fn(&CorpusFile) -> Compression) -> RunResult {
|
||||
let temp = tempfile::NamedTempFile::new().expect("temp file");
|
||||
let tokio_file = tokio::fs::File::create(temp.path()).await.expect("create");
|
||||
let buf_writer = BufWriter::with_capacity(256 * 1024, tokio_file);
|
||||
let mut zip = ZipFileWriter::with_tokio(buf_writer);
|
||||
|
||||
let cpu0 = cpu_seconds();
|
||||
let t0 = Instant::now();
|
||||
for f in files {
|
||||
let entry = ZipEntryBuilder::new(f.name.clone().into(), pick(f));
|
||||
let mut w = zip.write_entry_stream(entry).await.expect("entry start");
|
||||
for chunk in f.data.chunks(CHUNK) {
|
||||
w.write_all(chunk).await.expect("chunk write");
|
||||
}
|
||||
w.close().await.expect("entry close");
|
||||
}
|
||||
let mut compat = zip.close().await.expect("zip close");
|
||||
compat.close().await.expect("flush");
|
||||
let wall = t0.elapsed();
|
||||
let cpu = cpu_seconds() - cpu0;
|
||||
|
||||
let bytes_out = std::fs::metadata(temp.path()).map(|m| m.len()).unwrap_or(0);
|
||||
RunResult {
|
||||
wall,
|
||||
cpu,
|
||||
bytes_out,
|
||||
}
|
||||
}
|
||||
|
||||
fn median(mut xs: Vec<f64>) -> f64 {
|
||||
xs.sort_by(|a, b| a.partial_cmp(b).unwrap());
|
||||
xs[xs.len() / 2]
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
let media_files: usize = env_or("BENCH_MEDIA_FILES", 48);
|
||||
let media_mb: usize = env_or("BENCH_MEDIA_MB", 4);
|
||||
let text_files: usize = env_or("BENCH_TEXT_FILES", 24);
|
||||
let text_mb: usize = env_or("BENCH_TEXT_MB", 2);
|
||||
let reps: usize = env_or("BENCH_REPS", 3);
|
||||
|
||||
let mut seed = 0x9E3779B97F4A7C15u64;
|
||||
let mut corpus: Vec<CorpusFile> = Vec::new();
|
||||
for i in 0..media_files {
|
||||
let mut data = vec![0u8; media_mb * 1024 * 1024];
|
||||
fill_random(&mut data, &mut seed);
|
||||
corpus.push(CorpusFile {
|
||||
name: format!("photos/IMG_{i:04}.jpg"),
|
||||
data,
|
||||
is_media: true,
|
||||
});
|
||||
}
|
||||
for i in 0..text_files {
|
||||
let mut data = vec![0u8; text_mb * 1024 * 1024];
|
||||
fill_text(&mut data, &mut seed);
|
||||
corpus.push(CorpusFile {
|
||||
name: format!("docs/notes_{i:04}.txt"),
|
||||
data,
|
||||
is_media: false,
|
||||
});
|
||||
}
|
||||
let media_bytes: usize = corpus
|
||||
.iter()
|
||||
.filter(|f| f.is_media)
|
||||
.map(|f| f.data.len())
|
||||
.sum();
|
||||
let text_bytes: usize = corpus
|
||||
.iter()
|
||||
.filter(|f| !f.is_media)
|
||||
.map(|f| f.data.len())
|
||||
.sum();
|
||||
let total_mb = (media_bytes + text_bytes) as f64 / 1048576.0;
|
||||
println!(
|
||||
"corpus: {} media files ({} MiB, incompressible) + {} text files ({} MiB, compressible), {} reps\n",
|
||||
media_files,
|
||||
media_bytes / 1048576,
|
||||
text_files,
|
||||
text_bytes / 1048576,
|
||||
reps
|
||||
);
|
||||
|
||||
// (label, per-entry compression picker)
|
||||
type Picker = Box<dyn Fn(&CorpusFile) -> Compression>;
|
||||
let modes: Vec<(&str, Picker)> = vec![
|
||||
(
|
||||
"all-Deflate (BEFORE)",
|
||||
Box::new(|_: &CorpusFile| Compression::Deflate),
|
||||
),
|
||||
(
|
||||
"mime-aware (AFTER) ",
|
||||
Box::new(|f: &CorpusFile| {
|
||||
if f.is_media {
|
||||
Compression::Stored
|
||||
} else {
|
||||
Compression::Deflate
|
||||
}
|
||||
}),
|
||||
),
|
||||
(
|
||||
"all-Stored (bound) ",
|
||||
Box::new(|_: &CorpusFile| Compression::Stored),
|
||||
),
|
||||
];
|
||||
|
||||
println!(
|
||||
"{:<22} {:>9} {:>9} {:>10} {:>11} {:>9}",
|
||||
"mode", "wall s", "cpu s", "MB/s", "out MiB", "ratio"
|
||||
);
|
||||
let mut baseline_wall = None;
|
||||
for (label, pick) in &modes {
|
||||
let mut walls = Vec::new();
|
||||
let mut cpus = Vec::new();
|
||||
let mut out = 0u64;
|
||||
for _ in 0..reps {
|
||||
let r = write_zip(&corpus, pick).await;
|
||||
walls.push(r.wall.as_secs_f64());
|
||||
cpus.push(r.cpu);
|
||||
out = r.bytes_out;
|
||||
}
|
||||
let wall = median(walls);
|
||||
let cpu = median(cpus);
|
||||
let speedup = baseline_wall
|
||||
.map(|b: f64| format!("{:.2}x", b / wall))
|
||||
.unwrap_or_else(|| "1.00x".into());
|
||||
if baseline_wall.is_none() {
|
||||
baseline_wall = Some(wall);
|
||||
}
|
||||
println!(
|
||||
"{:<22} {:>9.3} {:>9.2} {:>10.1} {:>11.1} {:>9}",
|
||||
label,
|
||||
wall,
|
||||
cpu,
|
||||
total_mb / wall,
|
||||
out as f64 / 1048576.0,
|
||||
speedup
|
||||
);
|
||||
}
|
||||
println!("\n(archive `out MiB` for mime-aware stays ~= all-Deflate: media doesn't deflate,");
|
||||
println!(" text keeps Deflate — the win is CPU/wall, not size loss)");
|
||||
}
|
||||
Reference in New Issue
Block a user