cd4c62042a
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
372 lines
15 KiB
TOML
372 lines
15 KiB
TOML
[package]
|
|
name = "oxicloud"
|
|
version = "0.8.2"
|
|
edition = "2024"
|
|
default-run = "oxicloud"
|
|
|
|
|
|
[dependencies]
|
|
mimalloc = { version = "0.1.52", default-features = false }
|
|
axum = { version = "0.8.9", features = ["multipart", "http1", "http2", "tokio", "macros"] }
|
|
# "process" was previously enabled implicitly through aws-config's feature
|
|
# unification; ffmpeg_video_frame_service needs it, so declare it ourselves.
|
|
tokio = { version = "1.52.3", features = ["rt-multi-thread", "macros", "io-util", "net", "time", "sync", "fs", "process"] }
|
|
tokio-util = { version = "0.7.18", features = ["io", "codec", "compat"] }
|
|
tokio-stream = { version = "0.1.18", features = ["fs", "sync"] }
|
|
bytes = "1.11.1"
|
|
tempfile = "3.27.0"
|
|
tower = "0.5.3"
|
|
tower-http = { version = "0.6.11", features = ["fs", "compression-gzip", "compression-br", "trace", "cors", "add-extension", "request-id", "set-header", "limit"] }
|
|
flate2 = "1.1.9"
|
|
tracing = "0.1.44"
|
|
tracing-subscriber = { version = "0.3.23", features = ["env-filter"] }
|
|
chrono = { version = "0.4.45", features = ["serde"] }
|
|
# RFC 5545 iCalendar parser + emitter.
|
|
#
|
|
# Adopted 2026-07-14 to replace the hand-rolled property-scan in
|
|
# `src/domain/entities/calendar_event.rs::extract_ical_property`,
|
|
# which used a naive `format!("\n{}:", name)` substring search and
|
|
# refused ANY property carrying parameters (`DTSTART;VALUE=DATE:...`,
|
|
# `RECURRENCE-ID;VALUE=DATE:...`, `ATTENDEE;CN=…;PARTSTAT=…:mailto:…`).
|
|
# That broke all-day events and made the domain unaware of exception
|
|
# instances (see AtalayaLabs/OxiCloud#528).
|
|
#
|
|
# The crate is the widely-used Rust parser (~1500 SLOC, MIT/Apache),
|
|
# actively maintained by @Peltoche as `ical-rs` on GitHub. It handles
|
|
# line-folding, escaped characters, parameter maps, and every standard
|
|
# component. If a spec conformance gap is found, we contribute upstream.
|
|
ical = "0.11"
|
|
http-body = "1.0.1"
|
|
serde = { version = "1.0.228", features = ["derive"] }
|
|
serde_json = "1.0.150"
|
|
futures = "0.3.32"
|
|
async-stream = "0.3.6"
|
|
async-trait = "0.1.89"
|
|
mime_guess = "2.0.5"
|
|
uuid = { version = "1.23.3", features = ["v4", "v7", "serde"] }
|
|
thiserror = "2.0.18"
|
|
|
|
mockall = { version = "0.14.0", optional = true }
|
|
sqlx = { version = "0.8.6", features = ["postgres", "runtime-tokio", "tls-rustls", "chrono", "uuid", "json", "migrate"] }
|
|
jsonwebtoken = { version = "10.4.0", features = ["rust_crypto"] }
|
|
argon2 = "0.5.3"
|
|
rand_core = { version = "0.6", features = ["std", "getrandom"] }
|
|
quick-xml = "0.41.0"
|
|
dotenvy = "0.15.7"
|
|
moka = { version = "0.12.15", features = ["future", "sync"] }
|
|
http-range-header = "0.4"
|
|
image = { version = "0.25.10", default-features = false, features = ["jpeg", "png", "gif", "webp"] }
|
|
# Shrink-on-load JPEG decode (DCT 1/2·1/4·1/8 scaling) for thumbnails — pure
|
|
# Rust, no C toolchain. The `image` crate's zune-jpeg backend can't scale during
|
|
# decode; this can, cutting decode time/RAM ~order-of-magnitude on large photos.
|
|
jpeg-decoder = "0.3"
|
|
# SIMD (AVX2/SSE4.1/NEON) image resizing for thumbnails — far faster than the
|
|
# `image` crate's scalar resampler, and it speeds every format (incl. PNG/WebP).
|
|
fast_image_resize = "5"
|
|
# Lossy WebP encoding for thumbnails. The `image` crate's webp feature is
|
|
# lossless-only; this binds libwebp (vendored + built via `cc`, no system dep)
|
|
# for quality-controlled lossy output (~25-30% smaller than JPEG at equal SSIM).
|
|
webp = "0.3"
|
|
id3 = "1.17"
|
|
mp3-duration = "0.1"
|
|
kamadak-exif = "0.6.1"
|
|
md-5 = "0.11.0"
|
|
sha2 = "0.11.0"
|
|
unicode-normalization = "0.1.25"
|
|
blake3 = { version = "1.8.5", features = ["rayon", "mmap"] }
|
|
hex = "0.4.3"
|
|
http-body-util = "0.1.3"
|
|
percent-encoding = "2.3.2"
|
|
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-webpki-roots","rustls-tls-native-roots"] }
|
|
base64 = "0.22.1"
|
|
fs2 = "0.4"
|
|
rayon = "1.12.0"
|
|
infer = "0.19"
|
|
async-compression = { version = "0.4.42", features = ["tokio", "gzip"] }
|
|
async_zip = { version = "0.0.18", features = ["tokio", "deflate"] }
|
|
dashmap = "6.2.1"
|
|
socket2 = { version = "0.6.4", features = ["all"] }
|
|
urlencoding = "2.1.3"
|
|
utoipa = { version = "5.5.0", features = ["axum_extras", "uuid", "chrono"] }
|
|
# NOTE: aws-config and aws-smithy-types were removed as direct deps in the
|
|
# round-3 perf pass — S3BlobBackend builds its client purely from
|
|
# aws_sdk_s3::config with static credentials; nothing referenced either
|
|
# crate, and aws-config alone pulled aws-sdk-sso/ssooidc/sts (~90 crates)
|
|
# into every build (benches/ROUND3.md).
|
|
aws-sdk-s3 = "1.136.0"
|
|
azure_core = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
|
azure_storage = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
|
azure_storage_blobs = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
|
|
aes-gcm = "0.10.3"
|
|
lru = "0.16.4"
|
|
fastcdc = { version = "4.0.0", features = ["tokio"] }
|
|
lettre = { version = "0.11.18", default-features = false, features = ["smtp-transport", "tokio1-rustls-tls", "rustls-native-certs", "builder"] }
|
|
idna = "1.1"
|
|
smol_str = { version = "0.3.6", features = ["serde"] }
|
|
accept-language = "3.1.0"
|
|
askama = "0.16.0"
|
|
tantivy = "0.26.1"
|
|
zip = { version = "8.6.0", default-features = false, features = ["deflate"] }
|
|
pdf-extract = "0.12.0"
|
|
nom-exif = "3.6.1"
|
|
extism = { version = "1.30.0", optional = true }
|
|
toml = { version = "1.1.2", optional = true }
|
|
file-rotate = { version = "0.7.6", optional = true }
|
|
ort = { version = "2.0.0-rc.12", default-features = false, features = ["load-dynamic", "ndarray", "tracing", "api-24"], optional = true }
|
|
ndarray = { version = "0.17.2", optional = true }
|
|
|
|
[features]
|
|
default = []
|
|
test_utils = ["mockall"]
|
|
integration_tests = []
|
|
# WASM plugin runtime (Extism). Opt-in: bundles wasmtime, a large engine most
|
|
# deployments won't use. Activation also requires OXICLOUD_ENABLE_PLUGINS=true.
|
|
plugins = ["dep:extism", "dep:toml", "dep:file-rotate"]
|
|
# Empty marker feature that gates the `load-seed` binary so it isn't built
|
|
# by default (and skipped in prod Docker builds). Kept separate from
|
|
# test_utils so enabling it doesn't change the oxicloud dependency graph —
|
|
# this lets one `cargo build` produce both `oxicloud` and `load-seed`
|
|
# without recompiling oxicloud with mockall in scope.
|
|
load_seed_bin = []
|
|
# Real ONNX-backed face analyzer (detector + embedder) for the People feature.
|
|
# Opt-in: pulls `ort` (ONNX Runtime, load-dynamic — dlopen's libonnxruntime at
|
|
# runtime) + `ndarray`, a heavy stack most deployments won't use. Activation also
|
|
# requires OXICLOUD_ENABLE_FACES=true *and* operator-provided ONNX models; without
|
|
# this feature the People pipeline falls back to the inert NoopFaceAnalyzer.
|
|
faces-onnx = ["dep:ort", "dep:ndarray"]
|
|
# Performance benchmark harness (Phase 0). Exposes `bench_support` + thin public
|
|
# wrappers over the private thumbnail render functions so `benches/` and
|
|
# `examples/` can measure them. Off by default — adds nothing to prod builds.
|
|
# Run with: `cargo bench --features bench` / `cargo run --release --features bench --example bench_thumbnails_mem`.
|
|
bench = []
|
|
|
|
[dev-dependencies]
|
|
criterion = "0.5"
|
|
|
|
[lints.rust]
|
|
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(integration_tests)'] }
|
|
|
|
[[bin]]
|
|
name = "generate-openapi"
|
|
path = "src/bin/generate-openapi.rs"
|
|
|
|
[[bin]]
|
|
name = "migrate-nfc-filenames"
|
|
path = "src/bin/migrate-nfc-filenames.rs"
|
|
|
|
[[bin]]
|
|
name = "load-seed"
|
|
path = "src/bin/load-seed.rs"
|
|
# Test fixture seeder. Gated behind the empty `load_seed_bin` feature so
|
|
# `cargo build --release` (and the prod Dockerfile) skip it. tests/load/run.sh
|
|
# and load-nightly.yml build it explicitly with --features load_seed_bin.
|
|
required-features = ["load_seed_bin"]
|
|
|
|
# Phase 0 perf harness — Task 0.2 (criterion latency + output-size bench).
|
|
[[bench]]
|
|
name = "thumbnails"
|
|
path = "benches/thumbnails.rs"
|
|
harness = false
|
|
required-features = ["bench"]
|
|
|
|
# Phase 0 perf harness — Task 0.3 (peak-RAM + saturated-throughput baseline).
|
|
[[example]]
|
|
name = "bench_thumbnails_mem"
|
|
path = "examples/bench_thumbnails_mem.rs"
|
|
required-features = ["bench"]
|
|
|
|
# ZIP entry-compression benchmark — Deflate-always vs MIME-aware Stored for
|
|
# already-compressed media on the folder/batch ZIP download path. No Postgres.
|
|
[[example]]
|
|
name = "bench_zip_media"
|
|
path = "examples/bench_zip_media.rs"
|
|
required-features = ["bench"]
|
|
|
|
# WebDAV dead-properties fetch benchmark — PROPFIND's per-child N+1 (with a
|
|
# non-indexable IS NOT DISTINCT FROM predicate) vs batched = ANY($1) per page
|
|
# (needs the dev Postgres up).
|
|
[[example]]
|
|
name = "bench_dead_props"
|
|
path = "examples/bench_dead_props.rs"
|
|
required-features = ["bench"]
|
|
|
|
# NC chroot / default-drive resolution benchmark — the middleware's 2 uncached
|
|
# queries per request vs the moka caches (needs the dev Postgres up).
|
|
[[example]]
|
|
name = "bench_chroot_cache"
|
|
path = "examples/bench_chroot_cache.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Quota-path benchmark — full auth.users row (incl. 512 KiB avatar) vs the
|
|
# narrow 2-column read, on every upload check / quota PROPFIND (needs Postgres).
|
|
[[example]]
|
|
name = "bench_quota_path"
|
|
path = "examples/bench_quota_path.rs"
|
|
required-features = ["bench"]
|
|
|
|
# People-tab benchmark — full faces scan (2 KiB embedding per row) vs grouped
|
|
# COUNT + batched cover lookup (needs Postgres).
|
|
[[example]]
|
|
name = "bench_people_list"
|
|
path = "examples/bench_people_list.rs"
|
|
required-features = ["bench"]
|
|
|
|
# PROPFIND folder-paging benchmark — LIMIT/OFFSET full-folder rescan per page
|
|
# vs keyset + (folder_id, name) index (needs Postgres).
|
|
[[example]]
|
|
name = "bench_propfind_paging"
|
|
path = "examples/bench_propfind_paging.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Static-asset compression benchmark — per-request Brotli vs precompressed
|
|
# sibling read. No Postgres.
|
|
[[example]]
|
|
name = "bench_static_precompress"
|
|
path = "examples/bench_static_precompress.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Round-2 battery: range-from-cache, NC chunk gate, delta prefetch, ingest
|
|
# overlap (real store_from_stream; run with OXICLOUD_INGEST_OVERLAP=0/1),
|
|
# ZIP streaming TTFB. Sections 1 and 4 need Postgres.
|
|
[[example]]
|
|
name = "bench_round2"
|
|
path = "examples/bench_round2.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Video thumbnail benchmark — Option B (server-side ffmpeg frame → WebP). Needs
|
|
# `ffmpeg` on PATH (libx264/libx265/libvpx-vp9 to synthesize the test corpus).
|
|
[[example]]
|
|
name = "bench_video_thumbnails"
|
|
path = "examples/bench_video_thumbnails.rs"
|
|
required-features = ["bench"]
|
|
|
|
# DB connection-pool tail-latency benchmark (needs the dev Postgres up).
|
|
[[example]]
|
|
name = "bench_db_pool"
|
|
path = "examples/bench_db_pool.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Blob-manifest read round-trip benchmark: OLD 2 queries vs NEW 1 (needs dev Postgres).
|
|
[[example]]
|
|
name = "bench_blob_manifest"
|
|
path = "examples/bench_blob_manifest.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Blob download read-ahead benchmark — sweeps the local backend's chunk prefetch
|
|
# depth (read_prefetch / buffered(N)) under disk-bound vs network-bound consumers
|
|
# and warm vs cold page cache. No Postgres needed.
|
|
[[example]]
|
|
name = "bench_blob_prefetch"
|
|
path = "examples/bench_blob_prefetch.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Tokio runtime tuning benchmark — worker over-subscription (throughput + p99
|
|
# under CPU contention) and blocking-pool RSS blast radius, default vs tuned.
|
|
# No Postgres needed; run under `taskset -c 0,1` to model a 2-core quota.
|
|
[[example]]
|
|
name = "bench_tokio_runtime"
|
|
path = "examples/bench_tokio_runtime.rs"
|
|
required-features = ["bench"]
|
|
|
|
# CPU pool concurrency benchmark — thumbnail decode throughput + p99 as the
|
|
# decode-permit count is swept, showing the effect of sizing the image pools to
|
|
# the CFS quota (effective_parallelism) vs the host core count. No Postgres;
|
|
# run under `taskset -c 0,1` to model a 2-core quota.
|
|
[[example]]
|
|
name = "bench_pool_concurrency"
|
|
path = "examples/bench_pool_concurrency.rs"
|
|
required-features = ["bench"]
|
|
|
|
# ACL owner-cache benchmark — owner query vs moka hit (needs the dev Postgres up).
|
|
[[example]]
|
|
name = "bench_owner_cache"
|
|
path = "examples/bench_owner_cache.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Round-3 battery ─────────────────────────────────────────────────────────────
|
|
|
|
# Web-UI folder listing — whole-folder rescan + top-N sort per page vs keyset
|
|
# pushdown into the UNION-ALL branches + (folder_id, LOWER(name), id) indexes
|
|
# (needs the dev Postgres up).
|
|
[[example]]
|
|
name = "bench_listing_keyset"
|
|
path = "examples/bench_listing_keyset.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Photos timeline — full-library scan + top-N above the grants join vs
|
|
# per-drive LATERAL top-N on the media-timeline index (needs Postgres).
|
|
[[example]]
|
|
name = "bench_photos_timeline"
|
|
path = "examples/bench_photos_timeline.rs"
|
|
required-features = ["bench"]
|
|
|
|
# PROPFIND subfolder paging — LIMIT/OFFSET + COUNT(*) OVER() per page vs
|
|
# keyset batch, mirroring the files-side PROPFIND-PAGING fix (needs Postgres).
|
|
[[example]]
|
|
name = "bench_folder_keyset"
|
|
path = "examples/bench_folder_keyset.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Basic-auth thundering herd — K concurrent cache misses each paying Argon2id
|
|
# vs single-flight try_get_with (needs Postgres).
|
|
[[example]]
|
|
name = "bench_auth_herd"
|
|
path = "examples/bench_auth_herd.rs"
|
|
required-features = ["bench"]
|
|
|
|
# CachedBlobBackend — miss stampede (N duplicate remote fetches racing on one
|
|
# .tmp) vs per-hash single-flight; warm-hit index throughput. No Postgres.
|
|
[[example]]
|
|
name = "bench_blob_cache"
|
|
path = "examples/bench_blob_cache.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Upload spool/assembly I/O — ReaderStream capacity sweep on part-file reads
|
|
# and BufWriter vs bare-File frame writes on the chunk spool path. No Postgres.
|
|
[[example]]
|
|
name = "bench_upload_spool"
|
|
path = "examples/bench_upload_spool.rs"
|
|
required-features = ["bench"]
|
|
|
|
# S3 chunk PUT — HEAD-before-PUT vs unconditional PUT against a local axum
|
|
# stub with injected latency; Azure Bytes-vs-to_vec copy micro. No Postgres.
|
|
[[example]]
|
|
name = "bench_s3_put"
|
|
path = "examples/bench_s3_put.rs"
|
|
required-features = ["bench"]
|
|
|
|
# File/Folder -> DTO mapping allocations — Arc<str> interning of closed-set
|
|
# display fields, 1-alloc etag/size formatting. No Postgres.
|
|
[[example]]
|
|
name = "bench_dto_map"
|
|
path = "examples/bench_dto_map.rs"
|
|
required-features = ["bench"]
|
|
|
|
# CardDAV REPORT — dead per-contact vCard pre-generation + O(N^2) uid scan vs
|
|
# single on-demand generation. No Postgres.
|
|
[[example]]
|
|
name = "bench_carddav_report"
|
|
path = "examples/bench_carddav_report.rs"
|
|
required-features = ["bench"]
|
|
|
|
# Search-results cache RSS — entry-count capacity vs byte weigher. No Postgres.
|
|
[[example]]
|
|
name = "bench_search_cache_mem"
|
|
path = "examples/bench_search_cache_mem.rs"
|
|
required-features = ["bench"]
|
|
|
|
[profile.release]
|
|
lto = "thin"
|
|
codegen-units = 1
|
|
opt-level = 3
|
|
strip = true
|
|
|
|
[profile.dev]
|
|
opt-level = 1
|
|
debug = true
|
|
|
|
[profile.bench]
|
|
lto = "fat"
|
|
codegen-units = 1
|
|
opt-level = 3
|