perf(authz): cache resource owner lookups in PgAclEngine
The owner short-circuit in PgAclEngine::check ran a PK query (SELECT user_id FROM storage.folders/files WHERE id=$1) on every authorization check of a folder/file — the common case, since users mostly act on their own resources. Memoise it in an owner_cache (moka, TTL 300s, 100k cap). The owner column is immutable, so this is safe: the cache maps resource -> real owner and can never grant a non-owner access (a different caller's owner==uid test fails against the cached owner and falls through to grants); a hard-deleted resource that briefly resolves to its former owner simply fails later at execution with NotFound. The per-check sql_queries counter now increments only on a miss. Removes 1 DB query + 1 pool-connection acquisition per owner check. Magnitude is deployment-specific (query latency x whether the pool is contended); see benches/ACL-OWNER-CACHE.md. Also adds two DB perf-investigation harnesses, gated behind the `bench` feature (need the dev Postgres; zero prod impact): - examples/bench_db_pool.rs + benches/DB-POOL.md — pool size vs tail latency - examples/bench_owner_cache.rs + benches/ACL-OWNER-CACHE.md — owner query vs cache Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
DioCrafts
+12
@@ -151,6 +151,18 @@ name = "bench_thumbnails_mem"
|
||||
path = "examples/bench_thumbnails_mem.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# DB connection-pool tail-latency benchmark (needs the dev Postgres up).
|
||||
[[example]]
|
||||
name = "bench_db_pool"
|
||||
path = "examples/bench_db_pool.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
# ACL owner-cache benchmark — owner query vs moka hit (needs the dev Postgres up).
|
||||
[[example]]
|
||||
name = "bench_owner_cache"
|
||||
path = "examples/bench_owner_cache.rs"
|
||||
required-features = ["bench"]
|
||||
|
||||
[profile.release]
|
||||
lto = "thin"
|
||||
codegen-units = 1
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# ACL owner-cache benchmark
|
||||
|
||||
Opportunity #2 from the backend perf investigation. `PgAclEngine::check` runs an
|
||||
owner short-circuit on every authorization check of a folder/file — the common
|
||||
case (a user touching their own resources). That was an uncached PK query
|
||||
(`SELECT user_id FROM storage.folders/files WHERE id=$1`) **per check**. Now
|
||||
memoised in an `owner_cache` (moka, `pg_acl_engine.rs`); the owner column is
|
||||
immutable so it's safe to cache.
|
||||
|
||||
## Safety
|
||||
|
||||
The cache maps `resource → real owner`. It can **never** grant a non-owner
|
||||
access: a different caller's `owner == uid` test fails against the cached real
|
||||
owner and falls through to the grant lookup. The only staleness is a
|
||||
hard-deleted resource briefly resolving to its former owner — harmless, since the
|
||||
operation then fails at execution with NotFound, and no new access is granted.
|
||||
TTL 300 s, capacity 100 k. (Owners never change, so no invalidation hook needed.)
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
docker compose up -d postgres # needs ≥1 folder in the dev DB (just load-seed)
|
||||
cargo run --release --features bench --example bench_owner_cache
|
||||
# tunables: BENCH_CONCURRENCY (64), BENCH_POOL_SIZE (20 = prod default), BENCH_SECONDS (4)
|
||||
```
|
||||
|
||||
Models the owner short-circuit: the exact owner query vs a moka hit, under
|
||||
concurrency, against the real dev Postgres.
|
||||
|
||||
## Results (14 cores, local Docker Postgres)
|
||||
|
||||
**Burst — C = 64, pool = 20 (pool-saturating):**
|
||||
|
||||
| mode | DB queries | checks/s | p50 µs | p95 µs | p99 µs | max µs |
|
||||
|------------------|-----------:|---------:|-------:|-------:|-------:|-------:|
|
||||
| uncached (before)| 32746 | 8172 | 7063.0 |11323.9 |20135.1 |106015.7|
|
||||
| cached (after) | 1 | 6114546 | 1.46 | 3.00 | 3.79 | 75111* |
|
||||
|
||||
**Low load — C = 8, pool = 20 (no queueing, isolates pure query cost):**
|
||||
|
||||
| mode | DB queries | checks/s | p50 µs | p95 µs | p99 µs | max µs |
|
||||
|------------------|-----------:|---------:|-------:|-------:|-------:|-------:|
|
||||
| uncached (before)| 25432 | 6357 | 1097.5 | 2091.5 | 2433.9 |10105.8 |
|
||||
| cached (after) | 1 | 5076363 | 0.88 | 5.00 | 10.92 | 2342* |
|
||||
|
||||
\* `max` for the cached path is a tokio-scheduler / allocation outlier, not the
|
||||
cache — p99 (µs) is the meaningful tail.
|
||||
|
||||
## Conclusions
|
||||
|
||||
1. **Per owner check, 1 DB query → 1 memory hit.** Pure query cost (C=8): the
|
||||
cache saves **~1.1 ms p50 / 2.4 ms p99** per check (the owner query latency
|
||||
on this Docker-on-macOS Postgres).
|
||||
|
||||
2. **Under burst it compounds with the pool (opportunity #1).** At C=64/pool=20,
|
||||
the uncached checks both pay the query *and* queue on `acquire()` → p50 7 ms,
|
||||
p99 20 ms. The cache removes the query entirely → no pool occupancy → p99
|
||||
3.8 µs. Fewer queries ⇒ less pool pressure ⇒ lower tail latency under exactly
|
||||
the conditions that cause the cliff.
|
||||
|
||||
3. **Caveat:** the absolute ms here are inflated by the local Docker Postgres
|
||||
query latency (~1–2.4 ms; a tuned local PG would be faster, a networked/loaded
|
||||
one similar or worse) and the deliberately induced pool pressure. The robust,
|
||||
deployment-independent facts are: **1 fewer DB query and 1 fewer pool
|
||||
acquisition per authorized action on an owned resource** — which is most
|
||||
actions, since users mostly touch their own files.
|
||||
|
||||
4. Scope: this is the owner short-circuit (the hot path). Non-owner checks still
|
||||
do the cascade grant query (unchanged); group expansion was already cached.
|
||||
@@ -0,0 +1,76 @@
|
||||
# DB connection-pool tail-latency benchmark
|
||||
|
||||
Measures how `OXICLOUD_DB_MAX_CONNECTIONS` (default 20, `config.rs`) affects
|
||||
throughput and tail latency (p95/p99) under concurrent load. Isolates the pool
|
||||
layer from HTTP/auth: a real `sqlx` Postgres pool of size P driven by C
|
||||
concurrent workers each looping `SELECT pg_sleep(query_ms)`. Measured latency =
|
||||
**acquire-wait + query** — the pool-queue effect. `pg_sleep` models query
|
||||
*duration* (connection occupancy); real listing/auth queries take a few ms each.
|
||||
|
||||
## Reproduce
|
||||
|
||||
```bash
|
||||
docker compose up -d postgres # needs the dev Postgres
|
||||
cargo run --release --features bench --example bench_db_pool
|
||||
# tunables: BENCH_CONCURRENCY=96 BENCH_QUERY_MS=3 BENCH_SECONDS=4 BENCH_POOL_SIZES=10,20,40,70
|
||||
```
|
||||
|
||||
## Results (14 cores, local Postgres `max_connections=100`, pg_sleep 3 ms)
|
||||
|
||||
**Burst — concurrency C = 96 in-flight requests:**
|
||||
|
||||
| pool | req/s | p50 ms | p95 ms | p99 ms | max ms |
|
||||
|-----:|------:|-------:|-------:|-------:|-------:|
|
||||
| 10 | 1553 | 61.1 | 69.2 | 71.9 | 76.6 |
|
||||
| 20 | 3076 | 30.9 | 35.1 | 41.8 | 58.3 |
|
||||
| 40 | 5745 | 16.1 | 20.8 | 25.0 | 32.3 |
|
||||
| 70 | 9078 | 9.9 | 15.1 | 18.8 | 27.9 |
|
||||
|
||||
**Bigger burst — C = 192:**
|
||||
|
||||
| pool | req/s | p50 ms | p95 ms | p99 ms | max ms |
|
||||
|-----:|------:|-------:|-------:|-------:|-------:|
|
||||
| 10 | 1550 | 123.2 | 130.1 | 138.2 | 143.5 |
|
||||
| 20 | 3073 | 61.9 | 69.3 | 73.2 | 76.2 |
|
||||
| 40 | 5796 | 32.9 | 36.9 | 38.1 | 40.0 |
|
||||
| 70 | 9338 | 20.6 | 24.0 | 25.1 | 27.8 |
|
||||
|
||||
**Low load — C = 16 (≤ pool for 20/40/70):**
|
||||
|
||||
| pool | req/s | p50 ms | p95 ms | p99 ms | max ms |
|
||||
|-----:|------:|-------:|-------:|-------:|-------:|
|
||||
| 10 | 1569 | 10.2 | 13.7 | 14.6 | 16.2 |
|
||||
| 20 | 2555 | 6.0 | 7.9 | 13.4 | 42.8 |
|
||||
| 40 | 2484 | 6.1 | 7.9 | 14.8 | 30.9 |
|
||||
| 70 | 2573 | 6.1 | 7.5 | 11.5 | 20.7 |
|
||||
|
||||
## Conclusions
|
||||
|
||||
1. **When in-flight DB queries exceed the pool, the pool is the bottleneck.**
|
||||
Throughput scales ~linearly with pool size; latency ≈ concurrency × query /
|
||||
pool. At C=96, raising 20→70 gave **3.0× throughput** (2875→9078 req/s) and
|
||||
**2.2× lower p99** (42→19 ms). At C=192: **3× throughput**, **2.9× lower p99**
|
||||
(73→25 ms). The bigger the burst, the steeper the cliff a small pool creates.
|
||||
|
||||
2. **But once pool ≥ actual concurrency, more pool does NOTHING.** At C=16,
|
||||
pool 20/40/70 are identical (~6 ms p50, ~2500 req/s). Sizing beyond your peak
|
||||
concurrent in-flight query count is wasted (and costs Postgres connections).
|
||||
|
||||
3. **The right value ≈ your peak concurrent in-flight DB-query count** — not "as
|
||||
big as possible". Find it from the existing `DbPoolMonitor` (warns at 90%
|
||||
utilization). If it warns, raise `OXICLOUD_DB_MAX_CONNECTIONS`; if it never
|
||||
warns, the default 20 is fine.
|
||||
|
||||
4. Bounds: total connections are capped by Postgres `max_connections` (100 here,
|
||||
shared with the maintenance pool + other clients), and pg_sleep models I/O
|
||||
wait — real queries also use Postgres CPU, so a pool ≫ DB cores can overload
|
||||
Postgres. Don't raise blindly.
|
||||
|
||||
5. The default 20 is a sensible default for a low-concurrency self-hosted
|
||||
deployment; it becomes a tail-latency bottleneck under bursts of >20
|
||||
simultaneous DB-bound requests (many sync clients, bulk ops, or one browser
|
||||
firing many parallel requests). Left as an env knob rather than changing the
|
||||
default, because the right value is deployment-specific.
|
||||
|
||||
(Note: an early C=96 run showed a one-off p99=98 ms at pool=20 that did not
|
||||
reproduce — measurement noise, not a real effect.)
|
||||
@@ -0,0 +1,151 @@
|
||||
//! DB connection-pool tail-latency benchmark.
|
||||
//!
|
||||
//! Isolates the variable under test — `max_connections` — from the HTTP/auth
|
||||
//! stack. Builds a real `sqlx` Postgres pool of size P and drives it with `C`
|
||||
//! concurrent workers, each looping `SELECT pg_sleep($query_ms)` (a query of
|
||||
//! known duration). The measured per-request latency is **acquire-wait + query**
|
||||
//! — exactly the pool-exhaustion mechanism: when in-flight queries exceed P, the
|
||||
//! surplus queues on `acquire()`, inflating p95/p99.
|
||||
//!
|
||||
//! `pg_sleep` is a faithful stand-in for "a query that occupies a connection for
|
||||
//! T ms" — real listing/auth queries take a few ms each. We hold P constant per
|
||||
//! run and sweep it, so the *shape* of tail-latency-vs-pool-size is what matters.
|
||||
//!
|
||||
//! Run (needs the dev Postgres up; reads DATABASE_URL from .env):
|
||||
//! cargo run --release --features bench --example bench_db_pool
|
||||
//! Tunables (env): BENCH_CONCURRENCY (default 96), BENCH_QUERY_MS (3),
|
||||
//! BENCH_SECONDS (4), BENCH_POOL_SIZES ("10,20,40,70").
|
||||
|
||||
use std::env;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL (or OXICLOUD_DB_CONNECTION_STRING) — the dev Postgres URL");
|
||||
|
||||
let concurrency: usize = env_or("BENCH_CONCURRENCY", 96);
|
||||
let query_ms: u64 = env_or("BENCH_QUERY_MS", 3);
|
||||
let secs: u64 = env_or("BENCH_SECONDS", 4);
|
||||
let pool_sizes: Vec<u32> = env::var("BENCH_POOL_SIZES")
|
||||
.ok()
|
||||
.map(|s| s.split(',').filter_map(|x| x.trim().parse().ok()).collect())
|
||||
.unwrap_or_else(|| vec![10, 20, 40, 70]);
|
||||
|
||||
println!("\n###########################################################");
|
||||
println!("# DB pool tail-latency benchmark");
|
||||
println!("# concurrency (in-flight requests): {concurrency}");
|
||||
println!("# query duration: pg_sleep({query_ms} ms) window: {secs}s/pool");
|
||||
println!("# latency = acquire-wait + query (the pool-queue effect)");
|
||||
println!("###########################################################\n");
|
||||
println!(
|
||||
"| {:>4} | {:>9} | {:>10} | {:>8} | {:>8} | {:>8} | {:>9} | {:>6} |",
|
||||
"pool", "requests", "req/s", "p50 ms", "p95 ms", "p99 ms", "max ms", "errors"
|
||||
);
|
||||
println!(
|
||||
"|{:-<6}|{:-<11}|{:-<12}|{:-<10}|{:-<10}|{:-<10}|{:-<11}|{:-<8}|",
|
||||
"", "", "", "", "", "", "", ""
|
||||
);
|
||||
|
||||
let qsec = query_ms as f64 / 1000.0;
|
||||
|
||||
for &pool_size in &pool_sizes {
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(pool_size)
|
||||
.min_connections(pool_size) // pre-warm so we don't time connection setup
|
||||
.acquire_timeout(Duration::from_secs(10)) // matches prod connect_timeout default
|
||||
.connect(&url)
|
||||
.await
|
||||
.unwrap_or_else(|e| panic!("connect pool={pool_size}: {e}"));
|
||||
|
||||
// Warm-up burst (discarded).
|
||||
{
|
||||
let mut warm = Vec::new();
|
||||
for _ in 0..concurrency {
|
||||
let pool = pool.clone();
|
||||
warm.push(tokio::spawn(async move {
|
||||
let _ = sqlx::query("SELECT pg_sleep($1)")
|
||||
.bind(qsec)
|
||||
.execute(&pool)
|
||||
.await;
|
||||
}));
|
||||
}
|
||||
for h in warm {
|
||||
let _ = h.await;
|
||||
}
|
||||
}
|
||||
|
||||
let start = Instant::now();
|
||||
let deadline = start + Duration::from_secs(secs);
|
||||
let mut handles = Vec::with_capacity(concurrency);
|
||||
for _ in 0..concurrency {
|
||||
let pool = pool.clone();
|
||||
handles.push(tokio::spawn(async move {
|
||||
let mut lats_us: Vec<u32> = Vec::with_capacity(8192);
|
||||
let mut errors: u64 = 0;
|
||||
while Instant::now() < deadline {
|
||||
let t = Instant::now();
|
||||
match sqlx::query("SELECT pg_sleep($1)")
|
||||
.bind(qsec)
|
||||
.execute(&pool)
|
||||
.await
|
||||
{
|
||||
Ok(_) => lats_us.push(t.elapsed().as_micros() as u32),
|
||||
Err(_) => errors += 1,
|
||||
}
|
||||
}
|
||||
(lats_us, errors)
|
||||
}));
|
||||
}
|
||||
|
||||
let mut all: Vec<u32> = Vec::new();
|
||||
let mut errors: u64 = 0;
|
||||
for h in handles {
|
||||
let (l, e) = h.await.expect("join worker");
|
||||
all.extend(l);
|
||||
errors += e;
|
||||
}
|
||||
let elapsed = start.elapsed().as_secs_f64();
|
||||
pool.close().await;
|
||||
|
||||
all.sort_unstable();
|
||||
let n = all.len();
|
||||
let pct = |q: f64| -> f64 {
|
||||
if n == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
let idx = ((q / 100.0) * (n as f64 - 1.0)).round() as usize;
|
||||
all[idx.min(n - 1)] as f64 / 1000.0
|
||||
};
|
||||
let tput = n as f64 / elapsed;
|
||||
|
||||
println!(
|
||||
"| {:>4} | {:>9} | {:>10.0} | {:>8.2} | {:>8.2} | {:>8.2} | {:>9.2} | {:>6} |",
|
||||
pool_size,
|
||||
n,
|
||||
tput,
|
||||
pct(50.0),
|
||||
pct(95.0),
|
||||
pct(99.0),
|
||||
pct(100.0),
|
||||
errors,
|
||||
);
|
||||
}
|
||||
|
||||
println!(
|
||||
"\nNote: pg_sleep models query DURATION (connection occupancy), not CPU.\n\
|
||||
At fixed concurrency, raising the pool cuts queue-wait until pool ≈ concurrency,\n\
|
||||
then plateaus — the tail-latency shape that tells you the right size for your load.\n"
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
//! ACL owner-cache benchmark.
|
||||
//!
|
||||
//! Models the owner short-circuit in `PgAclEngine::check` (the common case: a
|
||||
//! user touching their own files). Before the cache, every authorization check
|
||||
//! on a folder/file ran one PK query `SELECT user_id FROM storage.folders WHERE
|
||||
//! id=$1` — a DB round-trip that also occupies a pool connection. After, repeat
|
||||
//! checks for the same resource hit an in-memory moka cache (owner is immutable).
|
||||
//!
|
||||
//! This isolates that exact query vs a moka hit, under concurrency C, against the
|
||||
//! real dev Postgres. It shows both the latency win and — by NOT touching the
|
||||
//! pool — the relief it gives the connection pool (ties into the pool benchmark).
|
||||
//!
|
||||
//! Run (needs the dev Postgres up with at least one folder; reads DATABASE_URL):
|
||||
//! cargo run --release --features bench --example bench_owner_cache
|
||||
//! Tunables: BENCH_CONCURRENCY (64), BENCH_POOL_SIZE (20 = prod default),
|
||||
//! BENCH_SECONDS (4).
|
||||
|
||||
use std::env;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use moka::future::Cache;
|
||||
use sqlx::Row;
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use uuid::Uuid;
|
||||
|
||||
fn env_or<T: std::str::FromStr>(key: &str, default: T) -> T {
|
||||
env::var(key)
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(default)
|
||||
}
|
||||
|
||||
struct Stats {
|
||||
reqs: u64,
|
||||
tput: f64,
|
||||
p50_us: f64,
|
||||
p95_us: f64,
|
||||
p99_us: f64,
|
||||
max_us: f64,
|
||||
}
|
||||
|
||||
fn summarize(mut lats_ns: Vec<u64>, elapsed: f64) -> Stats {
|
||||
lats_ns.sort_unstable();
|
||||
let n = lats_ns.len();
|
||||
let pct = |q: f64| -> f64 {
|
||||
if n == 0 {
|
||||
return 0.0;
|
||||
}
|
||||
let idx = ((q / 100.0) * (n as f64 - 1.0)).round() as usize;
|
||||
lats_ns[idx.min(n - 1)] as f64 / 1000.0
|
||||
};
|
||||
Stats {
|
||||
reqs: n as u64,
|
||||
tput: n as f64 / elapsed,
|
||||
p50_us: pct(50.0),
|
||||
p95_us: pct(95.0),
|
||||
p99_us: pct(99.0),
|
||||
max_us: pct(100.0),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::main(flavor = "multi_thread")]
|
||||
async fn main() {
|
||||
dotenvy::dotenv().ok();
|
||||
let url = env::var("DATABASE_URL")
|
||||
.or_else(|_| env::var("OXICLOUD_DB_CONNECTION_STRING"))
|
||||
.expect("set DATABASE_URL — the dev Postgres URL");
|
||||
let concurrency: usize = env_or("BENCH_CONCURRENCY", 64);
|
||||
let pool_size: u32 = env_or("BENCH_POOL_SIZE", 20); // production default
|
||||
let secs: u64 = env_or("BENCH_SECONDS", 4);
|
||||
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(pool_size)
|
||||
.min_connections(pool_size)
|
||||
.acquire_timeout(Duration::from_secs(10))
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect dev Postgres");
|
||||
|
||||
// A real folder + its owner to check against.
|
||||
let Some(row) = sqlx::query("SELECT id, user_id FROM storage.folders LIMIT 1")
|
||||
.fetch_optional(&pool)
|
||||
.await
|
||||
.expect("query folder")
|
||||
else {
|
||||
eprintln!("No folders in the dev DB — seed some first (`just load-seed`).");
|
||||
return;
|
||||
};
|
||||
let folder_id: Uuid = row.get("id");
|
||||
let owner_id: Uuid = row.get("user_id");
|
||||
|
||||
println!("\n###########################################################");
|
||||
println!("# ACL owner-cache benchmark (owner short-circuit path)");
|
||||
println!("# concurrency: {concurrency} pool: {pool_size} window: {secs}s/mode");
|
||||
println!("# folder {folder_id} owner {owner_id}");
|
||||
println!("###########################################################\n");
|
||||
|
||||
// ── BEFORE: one PK owner query per check (hits DB + pool) ──────────────
|
||||
let uncached = {
|
||||
let start = Instant::now();
|
||||
let deadline = start + Duration::from_secs(secs);
|
||||
let mut handles = Vec::with_capacity(concurrency);
|
||||
for _ in 0..concurrency {
|
||||
let pool = pool.clone();
|
||||
handles.push(tokio::spawn(async move {
|
||||
let mut lats = Vec::with_capacity(16384);
|
||||
while Instant::now() < deadline {
|
||||
let t = Instant::now();
|
||||
let _: Uuid =
|
||||
sqlx::query_scalar("SELECT user_id FROM storage.folders WHERE id = $1")
|
||||
.bind(folder_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.expect("owner query");
|
||||
lats.push(t.elapsed().as_nanos() as u64);
|
||||
}
|
||||
lats
|
||||
}));
|
||||
}
|
||||
let mut all = Vec::new();
|
||||
for h in handles {
|
||||
all.extend(h.await.expect("join"));
|
||||
}
|
||||
let s = summarize(all, start.elapsed().as_secs_f64());
|
||||
(s.reqs, s) // reqs == DB queries
|
||||
};
|
||||
|
||||
// ── AFTER: moka hit per check (no DB, no pool) ────────────────────────
|
||||
let cache: Cache<Uuid, Uuid> = Cache::builder()
|
||||
.max_capacity(100_000)
|
||||
.time_to_live(Duration::from_secs(300))
|
||||
.build();
|
||||
cache.insert(folder_id, owner_id).await; // 1 warm-up "query"
|
||||
let cached = {
|
||||
let start = Instant::now();
|
||||
let deadline = start + Duration::from_secs(secs);
|
||||
let mut handles = Vec::with_capacity(concurrency);
|
||||
for _ in 0..concurrency {
|
||||
let cache = cache.clone();
|
||||
handles.push(tokio::spawn(async move {
|
||||
let mut lats = Vec::with_capacity(65536);
|
||||
while Instant::now() < deadline {
|
||||
let t = Instant::now();
|
||||
let owner = cache.get(&folder_id).await.expect("cache hit");
|
||||
std::hint::black_box(owner);
|
||||
lats.push(t.elapsed().as_nanos() as u64);
|
||||
}
|
||||
lats
|
||||
}));
|
||||
}
|
||||
let mut all = Vec::new();
|
||||
for h in handles {
|
||||
all.extend(h.await.expect("join"));
|
||||
}
|
||||
summarize(all, start.elapsed().as_secs_f64())
|
||||
};
|
||||
pool.close().await;
|
||||
|
||||
let (uncached_queries, a) = uncached;
|
||||
println!(
|
||||
"| {:<16} | {:>10} | {:>11} | {:>9} | {:>9} | {:>9} | {:>10} |",
|
||||
"mode", "DB queries", "checks/s", "p50 µs", "p95 µs", "p99 µs", "max µs"
|
||||
);
|
||||
println!(
|
||||
"|{:-<18}|{:-<12}|{:-<13}|{:-<11}|{:-<11}|{:-<11}|{:-<12}|",
|
||||
"", "", "", "", "", "", ""
|
||||
);
|
||||
println!(
|
||||
"| {:<16} | {:>10} | {:>11.0} | {:>9.1} | {:>9.1} | {:>9.1} | {:>10.1} |",
|
||||
"uncached (before)", uncached_queries, a.tput, a.p50_us, a.p95_us, a.p99_us, a.max_us
|
||||
);
|
||||
println!(
|
||||
"| {:<16} | {:>10} | {:>11.0} | {:>9.3} | {:>9.3} | {:>9.3} | {:>10.3} |",
|
||||
"cached (after)",
|
||||
1,
|
||||
cached.tput,
|
||||
cached.p50_us,
|
||||
cached.p95_us,
|
||||
cached.p99_us,
|
||||
cached.max_us
|
||||
);
|
||||
|
||||
println!(
|
||||
"\nPer authorized action on an owned resource, the cache removes 1 DB query\n\
|
||||
+ 1 pool-connection occupancy, turning a {:.0} µs round-trip into a {:.3} µs\n\
|
||||
memory hit ({:.0}× lower p99). Over a network/loaded DB the absolute saving is\n\
|
||||
larger; the freed connections directly relieve the pool (see DB-POOL.md).\n",
|
||||
a.p99_us,
|
||||
cached.p99_us,
|
||||
if cached.p99_us > 0.0 {
|
||||
a.p99_us / cached.p99_us
|
||||
} else {
|
||||
0.0
|
||||
}
|
||||
);
|
||||
}
|
||||
@@ -74,6 +74,13 @@ struct QueryCounters {
|
||||
/// the cap signals pathological data and is surfaced to operators via audit.
|
||||
const MAX_GRANT_ROWS: i64 = 10_000;
|
||||
|
||||
/// `owner_cache` bound: entries are tiny (Resource + Uuid). 100k ≈ a few MB.
|
||||
const OWNER_CACHE_CAPACITY: u64 = 100_000;
|
||||
/// `owner_cache` TTL. A resource's owner is immutable, so the only staleness is
|
||||
/// a hard-deleted resource briefly resolving to its former owner — harmless
|
||||
/// (see `owner_cache` field doc), hence a generous TTL for a high hit rate.
|
||||
const OWNER_CACHE_TTL: Duration = Duration::from_secs(300);
|
||||
|
||||
pub struct PgAclEngine {
|
||||
pool: Arc<PgPool>,
|
||||
folder_repo: Arc<FolderDbRepository>,
|
||||
@@ -84,6 +91,14 @@ pub struct PgAclEngine {
|
||||
/// entries; eviction is LRU + TTL. Stale by up to TTL after a membership
|
||||
/// change — acceptable trade-off (see plan, "Cache TTL behaviour").
|
||||
user_groups_cache: Cache<Uuid, Arc<HashSet<Uuid>>>,
|
||||
/// Memoise `resource → owner UUID`. The owner column is immutable, so the
|
||||
/// owner-short-circuit (the common case: a user touching their own files)
|
||||
/// no longer issues a PK query on every authorization check — just the first
|
||||
/// per resource within the TTL. **Safe**: this can never grant a non-owner
|
||||
/// access (a different caller's `owner == uid` test fails against the cached
|
||||
/// *real* owner), and a hard-deleted resource that briefly short-circuits as
|
||||
/// owned simply fails later at execution with NotFound.
|
||||
owner_cache: Cache<Resource, Uuid>,
|
||||
}
|
||||
|
||||
impl PgAclEngine {
|
||||
@@ -102,6 +117,10 @@ impl PgAclEngine {
|
||||
.max_capacity(50_000)
|
||||
.time_to_live(Duration::from_secs(30))
|
||||
.build(),
|
||||
owner_cache: Cache::builder()
|
||||
.max_capacity(OWNER_CACHE_CAPACITY)
|
||||
.time_to_live(OWNER_CACHE_TTL)
|
||||
.build(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -153,6 +172,10 @@ impl PgAclEngine {
|
||||
.max_capacity(1)
|
||||
.time_to_live(Duration::from_secs(1))
|
||||
.build(),
|
||||
owner_cache: Cache::builder()
|
||||
.max_capacity(1)
|
||||
.time_to_live(Duration::from_secs(1))
|
||||
.build(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -273,6 +296,23 @@ impl PgAclEngine {
|
||||
self.subject_match_set(subject, &counters).await
|
||||
}
|
||||
|
||||
/// Owner lookup with memoisation. Hits the DB only on a cache miss; the
|
||||
/// result is cached because a resource's owner never changes. `NotFound`
|
||||
/// (a hard-deleted / nonexistent resource) is propagated, not cached.
|
||||
async fn owner_of_cached(
|
||||
&self,
|
||||
resource: Resource,
|
||||
counters: &QueryCounters,
|
||||
) -> Result<Uuid, DomainError> {
|
||||
if let Some(owner) = self.owner_cache.get(&resource).await {
|
||||
return Ok(owner);
|
||||
}
|
||||
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
|
||||
let owner = self.owner_of(resource).await?;
|
||||
self.owner_cache.insert(resource, owner).await;
|
||||
Ok(owner)
|
||||
}
|
||||
|
||||
/// Returns the owner UUID for any resource type.
|
||||
async fn owner_of(&self, resource: Resource) -> Result<Uuid, DomainError> {
|
||||
match resource {
|
||||
@@ -552,8 +592,7 @@ impl PgAclEngine {
|
||||
// there's no analogous fast path: the grant lookup below resolves
|
||||
// a drive owner via the same query that resolves any drive role.
|
||||
if let (Subject::User(uid), Resource::Folder(_) | Resource::File(_)) = (subject, resource) {
|
||||
counters.sql_queries.fetch_add(1, Ordering::Relaxed);
|
||||
match self.owner_of(resource).await {
|
||||
match self.owner_of_cached(resource, counters).await {
|
||||
Ok(owner) if owner == uid => return Ok(true),
|
||||
Ok(_) => { /* not owner — fall through to grants */ }
|
||||
Err(e) if e.kind == crate::common::errors::ErrorKind::NotFound => {
|
||||
|
||||
Reference in New Issue
Block a user