perf: round 4 — one-pass row paths, drive-selector cache, CalDAV single-parse, streamed Azure, batched hydration
Nine benchmark-gated changes (benches/ROUND4.md; every one ships with a BEFORE/AFTER bench + equivalence gate, rollback rule as ROUND2/3): - Row→entity path build: one-pass StoragePath::from_folder_and_name / from_joined + normalize_storage_name_owned + alloc-free Display — 743→417 ns/file-row (1.78x), −5 allocs/row on every listing surface. - WebDAV drive-selector: per-user readable_cache (single-flight, 30 s TTL, explicit invalidation incl. membership + group changes) replaces the grants join per request — 441 µs → 0.8 µs (~550x), 0 queries warm. - CalDAV from_ical/update_ical_data: 8 full IcalParser runs per VEVENT → 1 (7.1x per PUT, 4.4x on 50-event imports); alloc-free split_vevents, chunk scan without the whole-body uppercase copy (1.4x), borrowed-key UID grouping (1.3x), REPORT props no longer cloned. - PROPFIND emit: partition Vecs dropped (single-pass 404 list) + stack rendered RFC 3339/2822 dates, sizes, quoted etags (common::fmt, chrono-byte-identical, sweep-tested) on both DAV surfaces — 1.22x per page, 17.9→12.0 allocs/row. - Grant-listing hydration: calendars/address books/playlists batch hydrate via = ANY($1) — 15 serial queries → 1 (~13x per sync poll). - user-flags cache: get→insert → try_get_with single-flight (32→1 queries per cold herd). - Azure downloads: whole-blob Vec buffering → streamed SDK pages — TTFB 349→4 ms (87x), peak heap 480→1.9 MiB (254x) on 256 MiB blobs; new OXICLOUD_AZURE_ENDPOINT_URL override (Azurite/bench hook). - Face indexing: unbounded per-image tokio::spawn → core-count semaphore, permit before blob read — peak heap 1175→176 MiB (6.7x). Checks: cargo fmt, clippy --all-features --all-targets -D warnings, cargo test --workspace (523 passed) + --features test_utils. hurl API suite and dockerized integration DB not runnable in this environment — left to CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017aJu9ghvuT8WqC31ZEGTBA
This commit is contained in:
@@ -28,11 +28,35 @@ fn is_image(content_type: &str) -> bool {
|
||||
content_type.starts_with("image/")
|
||||
}
|
||||
|
||||
/// Concurrent index-task budget. Env override
|
||||
/// `OXICLOUD_FACES_INDEX_CONCURRENCY`, else the effective core count —
|
||||
/// each task is a full-image read + decode + ONNX inference, so more
|
||||
/// permits than cores only adds RAM pressure, not throughput.
|
||||
fn max_concurrent_index() -> usize {
|
||||
std::env::var("OXICLOUD_FACES_INDEX_CONCURRENCY")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.filter(|&n: &usize| n > 0)
|
||||
.unwrap_or_else(|| {
|
||||
std::thread::available_parallelism()
|
||||
.map(|n| n.get())
|
||||
.unwrap_or(2)
|
||||
})
|
||||
}
|
||||
|
||||
pub struct FaceIndexingService {
|
||||
pool: Arc<PgPool>,
|
||||
repo: Arc<FacePgRepository>,
|
||||
analyzer: Arc<dyn FaceAnalyzerPort>,
|
||||
blob_root: PathBuf,
|
||||
/// Bounds concurrent indexing tasks. The lifecycle hooks spawn one
|
||||
/// task per uploaded/copied image with no ceiling, so a bulk upload
|
||||
/// used to fan out N simultaneous full-image reads + decodes +
|
||||
/// inferences — peak RSS N × image size plus CPU thrash. Same
|
||||
/// invariant as `ThumbnailService::decode_semaphore`: the permit is
|
||||
/// acquired BEFORE the blob read, so peak memory is
|
||||
/// `permits × image size` regardless of upload concurrency.
|
||||
index_semaphore: Arc<tokio::sync::Semaphore>,
|
||||
}
|
||||
|
||||
impl FaceIndexingService {
|
||||
@@ -43,6 +67,7 @@ impl FaceIndexingService {
|
||||
repo,
|
||||
analyzer,
|
||||
blob_root,
|
||||
index_semaphore: Arc::new(tokio::sync::Semaphore::new(max_concurrent_index())),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -60,7 +85,15 @@ impl FaceIndexingService {
|
||||
let repo = self.repo.clone();
|
||||
let analyzer = self.analyzer.clone();
|
||||
let blob_path = self.blob_path(&blob_hash);
|
||||
let semaphore = self.index_semaphore.clone();
|
||||
tokio::spawn(async move {
|
||||
// Queue behind the concurrency budget BEFORE touching the
|
||||
// blob — excess tasks wait holding only this tiny future,
|
||||
// not a decoded image.
|
||||
let _permit = semaphore
|
||||
.acquire_owned()
|
||||
.await
|
||||
.expect("face index semaphore never closes");
|
||||
if delete_first {
|
||||
let _ = repo.delete_faces_for_file(file_id).await;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user