perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes
Every change is benchmark-verified (harness + before/after numbers in benches/, measured on this branch; reproduction commands in each doc): DAV / sync-client hot paths - PROPFIND dead-properties: one = ANY($1) query per 500-child page instead of one sequential query per child, and indexable `=` predicates instead of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and both NC REPORT handlers. [benches/DEAD-PROPS.md] - Folder paging: keyset cursor (name > $last) + new partial index (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page. Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration 20260917000000. [benches/PROPFIND-PAGING.md] - NC chroot / default-drive resolution: moka caches (30 s TTL, explicit invalidation on drive mutations) for find_default_for_user and the markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us). [benches/CHROOT-CACHE.md] - Quota: PROPFINDs whose prop list never names a quota prop skip the 2-query resolution entirely (wants_quota()); the remaining lookups read 2 columns instead of the full auth.users row with its <=512 KiB avatar (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every upload quota check. [benches/QUOTA-PATH.md] CPU on the request path - ZIP exports (folder download, share ZIP, batch download): entries whose MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md] - Compression layers: tower-http's default maps to Brotli QUALITY 11 (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4): 99x less CPU for ~15% more bytes. SPA assets are now precompressed at build time (scripts/precompress.mjs, 77% smaller) and served via ServeDir::precompressed_br/gzip: 2016x less per-request work, and clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md] Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md] - Content-search ReBAC re-verification: new AuthorizationEngine::check_files_read_batch (default = old loop; PgAclEngine override batches drive resolution + reuses role cache). 200 sequential point SELECTs per search -> 1-2 queries. - Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent recording (2 writes/file) for subtree entries already authorized at the root - mirrors the native folder-download path. ~6,000 statements removed from a 2,000-file archive. - CDC chunk manifests: immutable by content address, now moka-cached (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete) - removes one manifest query (p50 0.44-4.4 ms) from every stream, range and full blob read. - People tab: grouped COUNT + batched cover lookup instead of dragging every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB -> 3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE. [benches/PEOPLE-LIST.md] - Photos timeline cursor: raw timestamptz comparison instead of EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an index boundary again, deep scroll stops re-scanning skipped rows. - Public share landing: one atomic UPDATE ... access_count + 1 (was SELECT + full-row write-back: racy, lost updates, clobbered concurrent owner edits) - 3 round-trips -> 2 per visit. - move_to_trash: dead full-entity SELECT feeding a documented no-op removed from both branches; dead fields dropped from TrashService. - NFC normalization: is_nfc_quick fast path skips the decompose/recompose state machine for the ~100% already-NFC case (every row loaded from PG). Frontend - Large folders paint after page one (~200 items) via fetchFolderListing's new onPage hook instead of waiting for every sequential page. - Tested-and-reverted (kept for the record): cached Intl.Collator for name sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched. New bench harnesses under examples/ (bench feature): zip_media, dead_props, chroot_cache, quota_path, people_list, propfind_paging, static_precompress. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
@@ -30,7 +30,8 @@ use crate::domain::services::authorization::{Permission, Resource, Subject};
|
||||
use crate::infrastructure::services::path_resolver_service::ResolvedResource;
|
||||
use crate::infrastructure::services::webdav_dead_property_store::ResourceRef;
|
||||
use crate::interfaces::api::handlers::webdav_handler::{
|
||||
PROPFIND_BATCH_SIZE, file_dead_props, folder_dead_props, streamed_file_dead_props,
|
||||
PROPFIND_BATCH_SIZE, dead_props_for, file_dead_props, files_dead_props_map, folder_dead_props,
|
||||
folders_dead_props_map,
|
||||
};
|
||||
use crate::interfaces::errors::AppError;
|
||||
use crate::interfaces::range_requests::{not_modified_response, range_response};
|
||||
@@ -297,9 +298,10 @@ async fn handle_propfind(
|
||||
.map_err(|e| AppError::bad_request(format!("Failed to read body: {}", e)))?;
|
||||
|
||||
// Parse (and thereby validate) the PROPFIND body. The NC response
|
||||
// always emits the full property set, so the parsed request is not
|
||||
// consulted further — but malformed XML must still fail with 400.
|
||||
let _propfind = if body_bytes.is_empty() {
|
||||
// always emits the full property set; the parsed request is consulted
|
||||
// only to skip the quota DB round-trips when the client's explicit
|
||||
// prop list never names a quota prop. Malformed XML still fails 400.
|
||||
let propfind = if body_bytes.is_empty() {
|
||||
PropFindRequest {
|
||||
prop_find_type: crate::application::adapters::webdav_adapter::PropFindType::AllProp,
|
||||
}
|
||||
@@ -341,7 +343,13 @@ async fn handle_propfind(
|
||||
// function's username arg. Refining the owner-id usages
|
||||
// back to the canonical username is deferred to the
|
||||
// NcSession commit.
|
||||
let quota = state.resolve_webdav_quota(user.id, chroot.drive_id).await;
|
||||
// Explicit prop lists that never name a quota prop skip the
|
||||
// 2-query quota resolution (benches/QUOTA-PATH.md).
|
||||
let quota = if propfind.wants_quota() {
|
||||
state.resolve_webdav_quota(user.id, chroot.drive_id).await
|
||||
} else {
|
||||
None
|
||||
};
|
||||
Ok(build_nc_streaming_propfind(
|
||||
state.clone(),
|
||||
folder,
|
||||
@@ -1519,11 +1527,17 @@ fn build_nc_streaming_propfind(
|
||||
|
||||
// ── Children (only if Depth != 0) ────────────────────────────
|
||||
if depth != "0" {
|
||||
// Files in pages.
|
||||
let mut offset: i64 = 0;
|
||||
// Files in pages (keyset cursor — O(page) per page instead of
|
||||
// the quadratic LIMIT/OFFSET walk).
|
||||
let mut after_name: Option<String> = None;
|
||||
loop {
|
||||
let batch = file_service
|
||||
.list_files_batch_with_perms(Some(&folder.id), user_id, offset, PROPFIND_BATCH_SIZE)
|
||||
.list_files_batch_with_perms(
|
||||
Some(&folder.id),
|
||||
user_id,
|
||||
after_name.as_deref(),
|
||||
PROPFIND_BATCH_SIZE,
|
||||
)
|
||||
.await
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))?;
|
||||
if batch.is_empty() {
|
||||
@@ -1541,15 +1555,15 @@ fn build_nc_streaming_propfind(
|
||||
};
|
||||
let file_uuids: Vec<String> = batch.iter().map(|f| f.id.clone()).collect();
|
||||
let (file_id_map, _) = batch_resolve_ids(file_id_svc, &file_uuids, &[]).await;
|
||||
let mut file_deads = Vec::with_capacity(batch_len);
|
||||
for file in &batch {
|
||||
file_deads.push(streamed_file_dead_props(&state.webdav_dead_props, file).await);
|
||||
}
|
||||
// One batched dead-props query per page, not one per child
|
||||
// (benches/DEAD-PROPS.md).
|
||||
let file_deads = files_dead_props_map(&state.webdav_dead_props, &batch).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(batch_len * 1024);
|
||||
{
|
||||
let mut xml = Writer::new(&mut chunk);
|
||||
for (file, dead) in batch.iter().zip(file_deads.iter()) {
|
||||
for file in batch.iter() {
|
||||
let dead = dead_props_for(&file.id, &file_deads);
|
||||
let child_sub = if subpath.is_empty() {
|
||||
file.name.clone()
|
||||
} else {
|
||||
@@ -1567,7 +1581,7 @@ fn build_nc_streaming_propfind(
|
||||
if (batch_len as i64) < PROPFIND_BATCH_SIZE {
|
||||
break;
|
||||
}
|
||||
offset += batch_len as i64;
|
||||
after_name = batch.last().map(|f| f.name.clone());
|
||||
}
|
||||
|
||||
// Subfolders in pages — also collections, same trailing-slash rule.
|
||||
@@ -1594,15 +1608,15 @@ fn build_nc_streaming_propfind(
|
||||
};
|
||||
let folder_uuids: Vec<String> = result.items.iter().map(|sf| sf.id.clone()).collect();
|
||||
let (_, sub_id_map) = batch_resolve_ids(file_id_svc, &[], &folder_uuids).await;
|
||||
let mut sub_deads = Vec::with_capacity(result.items.len());
|
||||
for sf in &result.items {
|
||||
sub_deads.push(folder_dead_props(&state.webdav_dead_props, sf).await);
|
||||
}
|
||||
// Batched — see benches/DEAD-PROPS.md.
|
||||
let sub_deads =
|
||||
folders_dead_props_map(&state.webdav_dead_props, &result.items).await;
|
||||
|
||||
let mut chunk = Vec::with_capacity(result.items.len() * 1024);
|
||||
{
|
||||
let mut xml = Writer::new(&mut chunk);
|
||||
for (sf, dead) in result.items.iter().zip(sub_deads.iter()) {
|
||||
for sf in result.items.iter() {
|
||||
let dead = dead_props_for(&sf.id, &sub_deads);
|
||||
let child_sub = if subpath.is_empty() {
|
||||
sf.name.clone()
|
||||
} else {
|
||||
|
||||
Reference in New Issue
Block a user