perf: eliminate N+1 hot-path queries, cache immutable lookups, stop re-compressing compressed bytes

Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):

DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
  of one sequential query per child, and indexable `=` predicates instead
  of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
  DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
  both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
  (folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
  Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
  20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
  invalidation on drive mutations) for find_default_for_user and the
  markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
  NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
  [benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
  2-query resolution entirely (wants_quota()); the remaining lookups read
  2 columns instead of the full auth.users row with its <=512 KiB avatar
  (11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
  upload quota check. [benches/QUOTA-PATH.md]

CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
  MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
  of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
  for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
  unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
  (verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
  response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
  99x less CPU for ~15% more bytes. SPA assets are now precompressed at
  build time (scripts/precompress.mjs, 77% smaller) and served via
  ServeDir::precompressed_br/gzip: 2016x less per-request work, and
  clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]

Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
  AuthorizationEngine::check_files_read_batch (default = old loop;
  PgAclEngine override batches drive resolution + reuses role cache).
  200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
  recording (2 writes/file) for subtree entries already authorized at the
  root - mirrors the native folder-download path. ~6,000 statements
  removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
  (weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
  - removes one manifest query (p50 0.44-4.4 ms) from every stream,
  range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
  every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
  3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
  [benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
  EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
  index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
  SELECT + full-row write-back: racy, lost updates, clobbered concurrent
  owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
  removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
  state machine for the ~100% already-NFC case (every row loaded from PG).

Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
  new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
  sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
  fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.

New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
This commit is contained in:
Claude
2026-07-16 14:20:20 +00:00
parent b69c18b934
commit aba89c4f5d
52 changed files with 3262 additions and 444 deletions
+38 -11
View File
@@ -734,7 +734,13 @@ impl BatchOperationService {
{
Ok(file_dto) => {
match self
.add_file_entry_streamed(&mut zip, file_id, &file_dto.name, user_id)
.add_file_entry_streamed(
&mut zip,
file_id,
&file_dto.name,
&file_dto.mime_type,
Some(user_id),
)
.await
{
Ok(_) => items_added += 1,
@@ -758,7 +764,7 @@ impl BatchOperationService {
{
Ok(root_folder) => {
match self
.add_folder_subtree_to_zip(&mut zip, folder_id, &root_folder, user_id)
.add_folder_subtree_to_zip(&mut zip, folder_id, &root_folder)
.await
{
Ok(_) => items_added += 1,
@@ -803,24 +809,44 @@ impl BatchOperationService {
}
/// Streams a single file into an async ZIP entry (~64 KB peak RAM per file).
///
/// Already-compressed content (per its MIME type) is `Stored` — deflating
/// JPEG/MP4/… burns ~a CPU core per download for ~0 % size gain.
///
/// `caller_id = Some(uid)` enforces the per-file Read check and records
/// the access in Recents (explicitly-selected top-level files).
/// `None` = the file was enumerated from a folder subtree whose ROOT the
/// caller already passed `get_folder_with_perms` for — per-file
/// re-authorization and per-file Recent spam (2 writes/file via the
/// recent hook) are skipped, mirroring `ZipService::create_folder_zip`
/// on the native folder-download path (benches/ZIP-BATCH-AUTHZ.md).
async fn add_file_entry_streamed(
&self,
zip: &mut ZipFileWriter<tokio_util::compat::Compat<BufWriter<tokio::fs::File>>>,
file_id: &str,
entry_name: &str,
caller_id: Uuid,
mime_type: &str,
caller_id: Option<Uuid>,
) -> Result<(), BatchOperationError> {
let entry = ZipEntryBuilder::new(entry_name.to_string().into(), Compression::Deflate);
let compression = crate::common::mime_detect::zip_entry_compression(mime_type);
let entry = ZipEntryBuilder::new(entry_name.to_string().into(), compression);
let mut writer = zip
.write_entry_stream(entry)
.await
.map_err(|e| BatchOperationError::Internal(format!("zip entry start: {}", e)))?;
let stream = self
.file_retrieval
.get_file_stream_with_perms(file_id, caller_id)
.await
.map_err(BatchOperationError::Domain)?;
let stream = match caller_id {
Some(uid) => self
.file_retrieval
.get_file_stream_with_perms(file_id, uid)
.await
.map_err(BatchOperationError::Domain)?,
None => self
.file_retrieval
.get_file_stream(file_id)
.await
.map_err(BatchOperationError::Domain)?,
};
let mut stream = std::pin::Pin::from(stream);
while let Some(chunk) = stream.next().await {
@@ -849,7 +875,6 @@ impl BatchOperationService {
zip: &mut ZipFileWriter<tokio_util::compat::Compat<BufWriter<tokio::fs::File>>>,
folder_id: &str,
root_folder: &FolderDto,
caller_id: Uuid,
) -> Result<(), BatchOperationError> {
// Bulk-fetch folder tree (small — one entry per folder)
let all_folders = self
@@ -903,8 +928,10 @@ impl BatchOperationService {
if let Some(files) = files_by_folder.get(&folder.id) {
for file in files {
let file_path = format!("{}{}", zip_dir, file.name);
// Subtree pre-authorized at the root folder — see
// `add_file_entry_streamed` docs for why `None`.
if let Err(e) = self
.add_file_entry_streamed(zip, &file.id, &file_path, caller_id)
.add_file_entry_streamed(zip, &file.id, &file_path, &file.mime_type, None)
.await
{
info!("Could not add file {} to ZIP: {}", file.name, e);