cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes

Browse Source 
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
2026-07-16 14:20:20 +00:00
parent b69c18b934
commit aba89c4f5d
52 changed files with 3262 additions and 444 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/interfaces/api/handlers/webdav_handler.rs
+78 -34
View File
@@ -38,6 +38,7 @@ use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::{AuthUser, CurrentUser};
use crate::interfaces::range_requests::{not_modified_response, range_response};
use percent_encoding::{AsciiSet, NON_ALPHANUMERIC, percent_decode_str, utf8_percent_encode};
use std::collections::HashMap;
use std::sync::Arc;
/// Characters that MUST NOT be percent-encoded inside a URI path segment.
@@ -557,7 +558,13 @@ async fn handle_propfind(
created_by: None,
updated_by: None,
};
let quota = state.resolve_webdav_quota(user.id, Uuid::nil()).await;
// Skip the 2-query quota resolution when the request's prop list
// never mentions quota (benches/QUOTA-PATH.md).
let quota = if propfind_request.wants_quota() {
state.resolve_webdav_quota(user.id, Uuid::nil()).await
} else {
None
};
return build_streaming_propfind_response(
root_folder,
None, // folder_id = None → root children (drive-root folders)
@@ -597,7 +604,11 @@ async fn handle_propfind(
)
.await?;
let folder_id = folder.id.clone();
let quota = state.resolve_webdav_quota(user.id, drive_id).await;
let quota = if propfind_request.wants_quota() {
state.resolve_webdav_quota(user.id, drive_id).await
} else {
None
};
return build_streaming_propfind_response(
folder,
Some(folder_id),
@@ -663,7 +674,11 @@ async fn handle_propfind(
)
.await?;
let folder_id = folder.id.clone();
let quota = state.resolve_webdav_quota(user.id, drive_id).await;
let quota = if propfind_request.wants_quota() {
state.resolve_webdav_quota(user.id, drive_id).await
} else {
None
};
return build_streaming_propfind_response(
folder,
Some(folder_id),
@@ -788,19 +803,18 @@ async fn build_streaming_propfind_response(
break;
}
// Materialise dead-props for the whole page before
// we start writing — keeps the borrow checker happy
// (the writer borrows the FolderDto and the dead-props
// vec for the duration of write_folder_entry_*).
let mut subfolder_deads = Vec::with_capacity(result.items.len());
for subfolder in &result.items {
subfolder_deads.push(folder_dead_props(&dead_props_store, subfolder).await);
}
// ONE batched dead-props query per page instead of a
// sequential per-child round-trip — the N+1 shape cost
// 1-4.5 s of pure DB chatter on a 2000-child folder
// (measured in benches/DEAD-PROPS.md).
let subfolder_deads =
folders_dead_props_map(&dead_props_store, &result.items).await;
let mut chunk = Vec::with_capacity(result.items.len() * 800);
{
let mut w = Writer::new(&mut chunk);
for (subfolder, child_dead) in result.items.iter().zip(subfolder_deads.iter()) {
for subfolder in result.items.iter() {
let child_dead = dead_props_for(&subfolder.id, &subfolder_deads);
let href = format!("{}{}/", base_href, encode_path_segment(&subfolder.name));
WebDavAdapter::write_folder_entry_with_dead_props(&mut w, subfolder, &propfind_request, &href, child_dead, quota)
.map_err(|e| std::io::Error::other(e.to_string()))?;
@@ -815,11 +829,17 @@ async fn build_streaming_propfind_response(
page += 1;
}
// Stream files in pages (user-scoped)
let mut offset: i64 = 0;
// Stream files in pages (user-scoped, keyset cursor — O(page)
// per page instead of the quadratic LIMIT/OFFSET walk).
let mut after_name: Option<String> = None;
loop {
let batch: Vec<FileDto> = file_retrieval_service
.list_files_batch_with_perms(fid_ref, user_id, offset, PROPFIND_BATCH_SIZE)
.list_files_batch_with_perms(
fid_ref,
user_id,
after_name.as_deref(),
PROPFIND_BATCH_SIZE,
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
@@ -828,15 +848,14 @@ async fn build_streaming_propfind_response(
}
let batch_len = batch.len();
let mut file_deads = Vec::with_capacity(batch_len);
for file in &batch {
file_deads.push(streamed_file_dead_props(&dead_props_store, file).await);
}
// Batched: one = ANY($1) query per 500-file page.
let file_deads = files_dead_props_map(&dead_props_store, &batch).await;
let mut chunk = Vec::with_capacity(batch_len * 800);
{
let mut w = Writer::new(&mut chunk);
for (file, child_dead) in batch.iter().zip(file_deads.iter()) {
for file in batch.iter() {
let child_dead = dead_props_for(&file.id, &file_deads);
let href = format!("{}{}", base_href, encode_path_segment(&file.name));
WebDavAdapter::write_file_entry_with_dead_props(&mut w, file, &propfind_request, &href, child_dead)
.map_err(|e| std::io::Error::other(e.to_string()))?;
@@ -847,7 +866,7 @@ async fn build_streaming_propfind_response(
if (batch_len as i64) < PROPFIND_BATCH_SIZE {
break;
}
offset += batch_len as i64;
after_name = batch.last().map(|f| f.name.clone());
}
}
@@ -1381,20 +1400,45 @@ pub(crate) async fn folder_dead_props(
.unwrap_or_default()
}
/// File-leaf variant for the streaming walker (takes a `&DeadPropertyStore`
/// rather than the full `&Arc<AppState>` so it can be called from inside
/// the async-stream future without cloning state).
pub(crate) async fn streamed_file_dead_props(
/// Batched dead-props fetch for a whole PROPFIND page of files: ONE
/// `file_id = ANY($1)` round-trip instead of one query per child (the old
/// per-child `streamed_file_dead_props` loop cost seconds on large folders —
/// benches/DEAD-PROPS.md). Same leniency as the single-resource helpers:
/// any failure → empty map, so the PROPFIND still emits live properties.
pub(crate) async fn files_dead_props_map(
store: &DeadPropertyStore,
file: &FileDto,
) -> Vec<(QualifiedName, Option<String>)> {
let Ok(file_id) = Uuid::parse_str(&file.id) else {
return Vec::new();
};
store
.get_all(ResourceRef::File(file_id))
.await
.unwrap_or_default()
files: &[FileDto],
) -> HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> {
let ids: Vec<Uuid> = files
.iter()
.filter_map(|f| Uuid::parse_str(&f.id).ok())
.collect();
store.get_all_for_files(&ids).await.unwrap_or_default()
}
/// Folder-page variant of [`files_dead_props_map`].
pub(crate) async fn folders_dead_props_map(
store: &DeadPropertyStore,
folders: &[FolderDto],
) -> HashMap<Uuid, Vec<(QualifiedName, Option<String>)>> {
let ids: Vec<Uuid> = folders
.iter()
.filter_map(|f| Uuid::parse_str(&f.id).ok())
.collect();
store.get_all_for_folders(&ids).await.unwrap_or_default()
}
/// Looks up one resource's dead props in a batched map (resources with no
/// dead properties are absent from the map → empty slice).
pub(crate) fn dead_props_for<'a>(
id: &str,
map: &'a HashMap<Uuid, Vec<(QualifiedName, Option<String>)>>,
) -> &'a [(QualifiedName, Option<String>)] {
Uuid::parse_str(id)
.ok()
.and_then(|u| map.get(&u))
.map(|v| v.as_slice())
.unwrap_or(&[])
}
/// A single condition inside a `List` of the WebDAV `If:` header