perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -34,6 +34,7 @@
|
||||
//! it was not handled by the path-based store either, so this is a
|
||||
//! parity decision, not a regression.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use sqlx::{PgPool, Row};
|
||||
@@ -126,17 +127,23 @@ impl DeadPropertyStore {
|
||||
}
|
||||
|
||||
/// Delete a specific dead property. No-op if not present.
|
||||
///
|
||||
/// Filters on the concrete id column (`folder_id = $1` / `file_id = $1`)
|
||||
/// rather than the old `IS NOT DISTINCT FROM` pair — PostgreSQL cannot
|
||||
/// serve `IS NOT DISTINCT FROM` from a B-tree index, so every lookup
|
||||
/// degraded to a sequential scan as the table grew. The `=` shape is
|
||||
/// served by the partial unique indexes from migration 20260830000001.
|
||||
/// (Same rationale for `get_all` / `get` / the batched readers below —
|
||||
/// measured in `benches/DEAD-PROPS.md`.)
|
||||
pub async fn remove(&self, r: ResourceRef, name: &QualifiedName) -> Result<(), DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
sqlx::query(&format!(
|
||||
"DELETE FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2
|
||||
AND namespace = $3
|
||||
AND local_name = $4",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1
|
||||
AND namespace = $2
|
||||
AND local_name = $3",
|
||||
))
|
||||
.bind(id)
|
||||
.bind(&name.namespace)
|
||||
.bind(&name.name)
|
||||
.execute(&*self.pool)
|
||||
@@ -150,28 +157,64 @@ impl DeadPropertyStore {
|
||||
&self,
|
||||
r: ResourceRef,
|
||||
) -> Result<Vec<(QualifiedName, Option<String>)>, DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
let rows = sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
let rows = sqlx::query(&format!(
|
||||
"SELECT namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1",
|
||||
))
|
||||
.bind(id)
|
||||
.fetch_all(&*self.pool)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("DeadPropertyStore", format!("get_all: {e}")))?;
|
||||
|
||||
Ok(rows
|
||||
.into_iter()
|
||||
.map(|r| {
|
||||
let namespace: String = r.get("namespace");
|
||||
let local_name: String = r.get("local_name");
|
||||
let value: Option<String> = r.get("value");
|
||||
(QualifiedName::new(namespace, local_name), value)
|
||||
})
|
||||
.collect())
|
||||
Ok(rows.into_iter().map(row_to_prop).collect())
|
||||
}
|
||||
|
||||
/// Batched variant of [`get_all`] for every file in a PROPFIND page:
|
||||
/// ONE `file_id = ANY($1)` round-trip instead of N sequential queries.
|
||||
/// Files with no dead properties are simply absent from the map.
|
||||
pub async fn get_all_for_files(
|
||||
&self,
|
||||
file_ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
self.get_all_batched("file_id", file_ids).await
|
||||
}
|
||||
|
||||
/// Batched variant of [`get_all`] for every subfolder in a PROPFIND page.
|
||||
pub async fn get_all_for_folders(
|
||||
&self,
|
||||
folder_ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
self.get_all_batched("folder_id", folder_ids).await
|
||||
}
|
||||
|
||||
async fn get_all_batched(
|
||||
&self,
|
||||
column: &str,
|
||||
ids: &[Uuid],
|
||||
) -> Result<HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>, DomainError> {
|
||||
if ids.is_empty() {
|
||||
return Ok(HashMap::new());
|
||||
}
|
||||
let rows = sqlx::query(&format!(
|
||||
"SELECT {column} AS resource_id, namespace, local_name, value
|
||||
FROM storage.webdav_dead_properties
|
||||
WHERE {column} = ANY($1)",
|
||||
))
|
||||
.bind(ids)
|
||||
.fetch_all(&*self.pool)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("DeadPropertyStore", format!("get_all_batched: {e}"))
|
||||
})?;
|
||||
|
||||
let mut map: HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> = HashMap::new();
|
||||
for row in rows {
|
||||
let resource_id: Uuid = row.get("resource_id");
|
||||
map.entry(resource_id).or_default().push(row_to_prop(row));
|
||||
}
|
||||
Ok(map)
|
||||
}
|
||||
|
||||
/// Return a specific dead property, or `None` if not stored.
|
||||
@@ -181,16 +224,14 @@ impl DeadPropertyStore {
|
||||
r: ResourceRef,
|
||||
name: &QualifiedName,
|
||||
) -> Result<Option<Option<String>>, DomainError> {
|
||||
let (folder_id, file_id) = split_ref(r);
|
||||
let row = sqlx::query(
|
||||
let (column, id) = split_ref(r);
|
||||
let row = sqlx::query(&format!(
|
||||
"SELECT value FROM storage.webdav_dead_properties
|
||||
WHERE folder_id IS NOT DISTINCT FROM $1
|
||||
AND file_id IS NOT DISTINCT FROM $2
|
||||
AND namespace = $3
|
||||
AND local_name = $4",
|
||||
)
|
||||
.bind(folder_id)
|
||||
.bind(file_id)
|
||||
WHERE {column} = $1
|
||||
AND namespace = $2
|
||||
AND local_name = $3",
|
||||
))
|
||||
.bind(id)
|
||||
.bind(&name.namespace)
|
||||
.bind(&name.name)
|
||||
.fetch_optional(&*self.pool)
|
||||
@@ -201,16 +242,23 @@ impl DeadPropertyStore {
|
||||
}
|
||||
}
|
||||
|
||||
/// Splits a `ResourceRef` into `(folder_id, file_id)` Option pairs for
|
||||
/// binding into SQL. The unused slot is `None` so `IS NOT DISTINCT FROM`
|
||||
/// matches the NULL stored in the unused column.
|
||||
fn split_ref(r: ResourceRef) -> (Option<Uuid>, Option<Uuid>) {
|
||||
/// Maps a `ResourceRef` onto the column that stores it plus the id to bind.
|
||||
/// The column name is one of two compile-time literals — never user input —
|
||||
/// so interpolating it into the SQL text is safe.
|
||||
fn split_ref(r: ResourceRef) -> (&'static str, Uuid) {
|
||||
match r {
|
||||
ResourceRef::Folder(id) => (Some(id), None),
|
||||
ResourceRef::File(id) => (None, Some(id)),
|
||||
ResourceRef::Folder(id) => ("folder_id", id),
|
||||
ResourceRef::File(id) => ("file_id", id),
|
||||
}
|
||||
}
|
||||
|
||||
fn row_to_prop(r: sqlx::postgres::PgRow) -> (QualifiedName, Option<String>) {
|
||||
let namespace: String = r.get("namespace");
|
||||
let local_name: String = r.get("local_name");
|
||||
let value: Option<String> = r.get("value");
|
||||
(QualifiedName::new(namespace, local_name), value)
|
||||
}
|
||||
|
||||
pub fn create_dead_property_store(pool: Arc<PgPool>) -> Arc<DeadPropertyStore> {
|
||||
Arc::new(DeadPropertyStore::new(pool))
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user