cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning

Browse Source 
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change
gated by a before/after benchmark — an AFTER that did not beat its BEFORE
was to be rolled back; none needed it. Equivalence gates assert identical
row sequences / byte-identical output on every behavior-preserving rewrite):

DB hot paths (local PG16, EXPLAIN-verified):
- Web-UI listing (list_resources_paged): cursor pushed INSIDE the
  folders/files UNION-ALL branches as sargable row-value comparisons with
  per-branch ORDER/LIMIT + two partial expression indexes
  (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page
  (19.5x); other sort modes at parity or better. New migration
  20260918000000. [benches/LISTING-KEYSET.md section in ROUND3]
- Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N
  on the timeline index, joins moved above the top-N. 50k-photo library:
  97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early"
  comment was refuted by EXPLAIN.
- PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off
  idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET
  (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x).

Concurrency:
- Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV
  connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB);
  now 1 (300 ms). Failed verifications remain uncached.
- CachedBlobBackend per-hash single-flight + unique tmp names: 16
  concurrent cold readers = 16 full remote downloads racing truncating
  writes on ONE deterministic .tmp (corruptible cache); now 1 download
  (16x less egress, 2.8x wall on a shared link) and torn files can never
  be renamed into the cache.

I/O and allocations:
- Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls);
  chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls).
- S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer
  pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying
  every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk).
- Entity->DTO mapping: Arc<str> interning of closed-set display fields +
  common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of
  clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster).
- CardDAV REPORT: deleted dead per-contact vCard pre-generation and the
  O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms,
  9.8x); byte-identical XML asserted.
- Search-results cache: byte weigher + 32 MiB budget
  (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let
  ~300 MiB of enriched rows sit in RSS; read latency parity.
- Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph
  nodes, three SDK stacks gone from every build). tokio "process" is now
  an explicit feature (was enabled transitively by aws-config).

Frontend:
- Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and
  4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate
  asserts output identity across locales and a 3x floor.

Validation: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints
smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor
walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login);
frontend npm run check clean, new vitest gates green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
Claude
2026-07-17 11:10:27 +00:00
parent 7d95a19907
commit cd4c62042a
43 changed files with 5290 additions and 439 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/interfaces/api/handlers/carddav_handler.rs
+2 -15
View File
@@ -385,7 +385,6 @@ async fn handle_propfind(
CardDavAdapter::generate_contacts_response(
&mut response_body,
std::slice::from_ref(&contact),
&[(contact.uid.clone(), contact_to_vcard(&contact))],
&report,
base_href,
)
@@ -449,22 +448,10 @@ async fn handle_report(
.map_err(AppError::from)?,
};
// Generate vCards
let vcards: Vec<(String, String)> = contacts
.iter()
.map(|c| (c.uid.clone(), contact_to_vcard(c)))
.collect();
let base_href = &format!("/carddav/{}/", address_book_id);
let mut response_body = Vec::new();
CardDavAdapter::generate_contacts_response(
&mut response_body,
&contacts,
&vcards,
&report,
base_href,
)
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
CardDavAdapter::generate_contacts_response(&mut response_body, &contacts, &report, base_href)
.map_err(|e| AppError::internal_error(format!("Failed to generate XML: {}", e)))?;
Ok(Response::builder()
.status(StatusCode::MULTI_STATUS)
src/interfaces/api/handlers/webdav_handler.rs
+18 -18
View File
@@ -781,25 +781,25 @@ async fn build_streaming_propfind_response(
// ── Children (only if Depth == 1) ────────────────────────
if depth == "1" {
let pagination = crate::application::dtos::pagination::PaginationRequestDto {
page: 0,
page_size: PROPFIND_BATCH_SIZE as usize,
};
let fid_ref = folder_id.as_deref();
// Stream sub-folders in pages (user-scoped)
let mut page = 0usize;
// Stream sub-folders in pages (user-scoped, keyset cursor —
// O(page) per page off idx_folders_unique_name instead of the
// quadratic COUNT(*) OVER() + LIMIT/OFFSET walk; 4.5x on a
// 5k-dir parent, benches/FOLDER-KEYSET.md).
let mut after_folder: Option<String> = None;
loop {
let pag = crate::application::dtos::pagination::PaginationRequestDto {
page,
page_size: pagination.page_size,
};
let result = folder_service
.list_folders_paginated_with_perms(fid_ref, user_id, &pag)
let batch = folder_service
.list_folders_batch_with_perms(
fid_ref,
user_id,
after_folder.as_deref(),
PROPFIND_BATCH_SIZE as usize,
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
if result.items.is_empty() {
if batch.is_empty() {
break;
}
@@ -808,25 +808,25 @@ async fn build_streaming_propfind_response(
// 1-4.5 s of pure DB chatter on a 2000-child folder
// (measured in benches/DEAD-PROPS.md).
let subfolder_deads =
folders_dead_props_map(&dead_props_store, &result.items).await;
folders_dead_props_map(&dead_props_store, &batch).await;
let mut chunk = Vec::with_capacity(result.items.len() * 800);
let mut chunk = Vec::with_capacity(batch.len() * 800);
{
let mut w = Writer::new(&mut chunk);
for subfolder in result.items.iter() {
for subfolder in batch.iter() {
let child_dead = dead_props_for(&subfolder.id, &subfolder_deads);
let href = format!("{}{}/", base_href, encode_path_segment(&subfolder.name));
WebDavAdapter::write_folder_entry_with_dead_props(&mut w, subfolder, &propfind_request, &href, child_dead, quota)
.map_err(|e| std::io::Error::other(e.to_string()))?;
}
}
let has_more = result.pagination.has_next;
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
after_folder = batch.last().map(|f| f.name.clone());
yield Bytes::from(chunk);
if !has_more {
break;
}
page += 1;
}
// Stream files in pages (user-scoped, keyset cursor — O(page)
src/interfaces/nextcloud/webdav_handler.rs
+20 -17
View File
@@ -16,7 +16,6 @@ use uuid::Uuid;
use crate::application::adapters::webdav_adapter::{
PropFindRequest, PropPatchOp, QualifiedName, WebDavAdapter, is_protected_property,
};
use crate::application::dtos::pagination::PaginationRequestDto;
use crate::application::ports::authorization_ports::AuthorizationEngine;
use crate::application::ports::favorites_ports::FavoritesUseCase;
use crate::application::ports::file_ports::{
@@ -1584,38 +1583,42 @@ fn build_nc_streaming_propfind(
after_name = batch.last().map(|f| f.name.clone());
}
// Subfolders in pages — also collections, same trailing-slash rule.
let mut page = 0usize;
// Subfolders in pages — also collections, same trailing-slash
// rule. Keyset cursor: O(page) per page off
// idx_folders_unique_name instead of the quadratic
// COUNT(*) OVER() + LIMIT/OFFSET walk (benches/FOLDER-KEYSET.md).
let mut after_folder: Option<String> = None;
loop {
let pag = PaginationRequestDto {
page,
page_size: PROPFIND_BATCH_SIZE as usize,
};
let result = folder_service
.list_folders_paginated_with_perms(Some(&folder.id), user_id, &pag)
let batch = folder_service
.list_folders_batch_with_perms(
Some(&folder.id),
user_id,
after_folder.as_deref(),
PROPFIND_BATCH_SIZE as usize,
)
.await
.map_err(|e| std::io::Error::other(e.to_string()))?;
if result.items.is_empty() {
if batch.is_empty() {
break;
}
let favs = if let Some(fav) = fav_svc {
let items: Vec<(&str, &str)> =
result.items.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
batch.iter().map(|sf| (sf.id.as_str(), "folder")).collect();
fav.batch_check_favorites(user_id, &items).await.unwrap_or_default()
} else {
HashSet::new()
};
let folder_uuids: Vec<String> = result.items.iter().map(|sf| sf.id.clone()).collect();
let folder_uuids: Vec<String> = batch.iter().map(|sf| sf.id.clone()).collect();
let (_, sub_id_map) = batch_resolve_ids(file_id_svc, &[], &folder_uuids).await;
// Batched — see benches/DEAD-PROPS.md.
let sub_deads =
folders_dead_props_map(&state.webdav_dead_props, &result.items).await;
folders_dead_props_map(&state.webdav_dead_props, &batch).await;
let mut chunk = Vec::with_capacity(result.items.len() * 1024);
let mut chunk = Vec::with_capacity(batch.len() * 1024);
{
let mut xml = Writer::new(&mut chunk);
for sf in result.items.iter() {
for sf in batch.iter() {
let dead = dead_props_for(&sf.id, &sub_deads);
let child_sub = if subpath.is_empty() {
sf.name.clone()
@@ -1629,13 +1632,13 @@ fn build_nc_streaming_propfind(
.map_err(std::io::Error::other)?;
}
}
let has_more = result.pagination.has_next;
let has_more = (batch.len() as i64) == PROPFIND_BATCH_SIZE;
after_folder = batch.last().map(|sf| sf.name.clone());
yield Bytes::from(chunk);
if !has_more {
break;
}
page += 1;
}
}
src/interfaces/upload_ingest.rs
+13 -2
View File
@@ -273,11 +273,16 @@ pub fn multipart_field_stream(
pub fn stream_from_files(
paths: Vec<PathBuf>,
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
// 512 KiB per poll: each ReaderStream poll on a tokio::fs::File is one
// blocking-pool dispatch + one read(2) of the buffer size. The old
// 64 KiB buffer paid 8x the dispatches/syscalls of every other blob
// read path (STREAM_CHUNK_SIZE = 256 KiB) for the single read pass
// over every completed chunked upload (benches/UPLOAD-SPOOL.md).
stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
.and_then(|path| async move {
tokio::fs::File::open(path)
.await
.map(|file| ReaderStream::with_capacity(file, 64 * 1024))
.map(|file| ReaderStream::with_capacity(file, 512 * 1024))
})
.try_flatten()
}
@@ -319,9 +324,15 @@ pub async fn stream_body_to_path(
max_bytes: usize,
checksum_alg: Option<ChecksumAlg>,
) -> Result<StreamedToPath, AppError> {
let mut file = tokio::fs::File::create(path)
// BufWriter coalesces the per-HTTP-frame writes (~16-64 KiB each) into
// 512 KiB write(2)s — a bare tokio File dispatches one blocking-pool op
// per frame (benches/UPLOAD-SPOOL.md). Same capacity as the dedup
// handler's spool loop. On the error paths below the partial file is
// removed, so silently dropping unflushed buffer contents is fine.
let file = tokio::fs::File::create(path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
let mut file = tokio::io::BufWriter::with_capacity(512 * 1024, file);
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);