feat(storage): add BlobReferenceSource port + registry
Step 1 of docs/plan/derived-blobs.md. Makes "who references this blob hash" an extension point instead of SQL hardcoded in two places (dedup_gc's reap predicate and blobs_consistency's refcount recompute, both naming storage.files and storage.chunk_manifests directly). Adding a blob-owning table without teaching those two risks silent orphaning: GC sees ref_count = 0 and reaps live content. Behaviour is unchanged — this commit only introduces the port and the two sources that reproduce today's SQL. Wiring follows. Two levels, not one. add_reference bumps chunk_manifests.ref_count first and only falls back to storage.blobs.ref_count, so a reference lands on whichever counter its hash names and the two must be recomputed separately. RefLevel is a parameter rather than a property of a source, because storage.files legitimately contributes at both: a manifest-less legacy row references a chunk, a CDC row references a Blob. The NOT EXISTS guard on the chunk-level files term is load-bearing — for a single-chunk file the whole-file hash equals its lone chunk's hash, so without it the row is counted at both levels. SQL fragments rather than a per-hash count. blobs_consistency recomputes with one query per page, the expected count inlined as correlated subqueries; asking each source for a count per hash would turn that into sources x rows round-trips. So sources emit a fragment the registry sums into the existing page query, and count_references exists only for the on-demand path where the candidate set is already filtered to ref_count = 0. Fragments use their own aliases (cnt_f, cnt_m) rather than the sweeps' outer-row aliases (b, m). A fragment reusing `m` would shadow the outer alias in the manifest sweep and silently correlate against itself; there is a test for it. The SQL builders are free functions so the shape can be asserted without constructing a pool — sqlx's connect_lazy still needs a Tokio context, and the fragments are pure string assembly anyway. 9 unit tests. fmt, clippy --all-features --all-targets, build clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
@@ -0,0 +1,278 @@
|
||||
//! `BlobReferenceSource` — the extension point that teaches ref-counting
|
||||
//! and the consistency jobs about a table holding blob references.
|
||||
//!
|
||||
//! Before this port, "who references this hash" was hardcoded SQL in two
|
||||
//! places (`dedup_gc`'s reap predicate and `blobs_consistency`'s refcount
|
||||
//! recompute), both naming `storage.files` and `storage.chunk_manifests`
|
||||
//! directly. Any new blob-owning table therefore risked silent orphaning:
|
||||
//! `dedup_gc` sees `ref_count = 0`, or a manifest with no `storage.files`
|
||||
//! row behind it, and reaps live content.
|
||||
//!
|
||||
//! See `docs/plan/derived-blobs.md` for the design and the coverage matrix.
|
||||
//!
|
||||
//! # Two levels, and why a source may span both
|
||||
//!
|
||||
//! [`DedupService::add_reference`] bumps `chunk_manifests.ref_count` first
|
||||
//! and only falls back to `storage.blobs.ref_count`. So a reference lands
|
||||
//! on whichever counter its hash names, and the two must be recomputed
|
||||
//! separately — mixing them double-counts, systematically:
|
||||
//!
|
||||
//! * A **Blob** (`chunk_manifests.file_hash`) is "the content of a file".
|
||||
//! * A **Chunk** (`storage.blobs.hash`) is a physical byte payload.
|
||||
//! * For a single-chunk Blob the two hashes are **equal**, because both are
|
||||
//! BLAKE3 over the same bytes. That aliasing is why today's chunk-level
|
||||
//! recompute carries a `NOT EXISTS` clause, and why every fragment here
|
||||
//! must be level-correct rather than merely plausible.
|
||||
//!
|
||||
//! A source is not confined to one level: [`RefLevel::Chunk`] and
|
||||
//! [`RefLevel::Manifest`] fragments are requested independently, and
|
||||
//! `storage.files` legitimately contributes to both — a manifest-less
|
||||
//! legacy row references a chunk, a CDC row references a Blob.
|
||||
//!
|
||||
//! # Why SQL fragments rather than a per-hash count
|
||||
//!
|
||||
//! `blobs_consistency` recomputes refcounts with **one query per page**,
|
||||
//! the expected count inlined as correlated subqueries. Asking each source
|
||||
//! for a count per hash would turn that into `sources × rows` round-trips —
|
||||
//! a catastrophic regression on a table with millions of rows. So sources
|
||||
//! contribute a *fragment* that the registry sums into the existing page
|
||||
//! query, and [`BlobReferenceSource::count_references`] exists only for the
|
||||
//! on-demand path (`dedup_gc` checking a single reap candidate, where the
|
||||
//! candidate set is already filtered to `ref_count = 0`).
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
/// Which counter a source's references land on.
|
||||
///
|
||||
/// Not a property of the source — see the module docs; the same source may
|
||||
/// contribute at both levels.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
pub enum RefLevel {
|
||||
/// References a physical chunk. Feeds `storage.blobs.ref_count`.
|
||||
Chunk,
|
||||
/// References a Blob via its manifest. Feeds
|
||||
/// `chunk_manifests.ref_count`.
|
||||
Manifest,
|
||||
}
|
||||
|
||||
impl RefLevel {
|
||||
/// Both levels, for callers that sweep each in turn.
|
||||
pub const ALL: [RefLevel; 2] = [RefLevel::Chunk, RefLevel::Manifest];
|
||||
|
||||
/// Stable name for logs and consistency-finding fields.
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
RefLevel::Chunk => "chunk",
|
||||
RefLevel::Manifest => "manifest",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One table that holds references to blob hashes.
|
||||
///
|
||||
/// Implementors are registered on [`BlobReferenceRegistry`] during DI.
|
||||
/// Adding a blob-owning table **without** registering it is the failure
|
||||
/// this port exists to prevent.
|
||||
#[async_trait]
|
||||
pub trait BlobReferenceSource: Send + Sync {
|
||||
/// Short stable identifier for logs and consistency-finding `source`
|
||||
/// fields — `"files"`, `"chunks"`, `"content_derived"`, …
|
||||
///
|
||||
/// Stable across releases: log aggregators key off it.
|
||||
fn source_name(&self) -> &'static str;
|
||||
|
||||
/// A correlated-subquery fragment counting this source's references
|
||||
/// **at `level`** to `outer_hash_expr`, or `None` when this source
|
||||
/// holds no references at that level.
|
||||
///
|
||||
/// `outer_hash_expr` is the SQL expression naming the hash of the row
|
||||
/// being recomputed — `"b.hash"` when sweeping `storage.blobs`,
|
||||
/// `"m.file_hash"` when sweeping `storage.chunk_manifests`. The
|
||||
/// fragment must be a parenthesised scalar subquery so the registry can
|
||||
/// join fragments with `+`.
|
||||
///
|
||||
/// **Identifiers only.** `outer_hash_expr` is supplied by the sweep, never
|
||||
/// by a request; no fragment may interpolate caller input.
|
||||
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String>;
|
||||
|
||||
/// Count of references this source holds on `blob_hash`, across both
|
||||
/// levels.
|
||||
///
|
||||
/// **On-demand path only** — `dedup_gc` checking a single reap
|
||||
/// candidate. The consistency sweeps must use [`Self::ref_count_sql`];
|
||||
/// calling this per row would turn one query per page into
|
||||
/// `sources × rows` round-trips.
|
||||
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError>;
|
||||
|
||||
/// Iterate the hashes this source references, paged by the
|
||||
/// implementation's natural cursor (typically a primary key).
|
||||
///
|
||||
/// Used by `backend_consistency` to walk the backend against the union
|
||||
/// of all sources. Returns the page plus the cursor to resume from,
|
||||
/// `None` when exhausted.
|
||||
async fn list_referenced_blobs(
|
||||
&self,
|
||||
cursor: Option<Vec<u8>>,
|
||||
limit: usize,
|
||||
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError>;
|
||||
|
||||
/// Notification that `dedup_gc` reaped this blob.
|
||||
///
|
||||
/// Sources maintaining a denormalised refcount can clean up here. Most
|
||||
/// leave the default noop — the mapping row is normally deleted by the
|
||||
/// owning service's `on_blob_deleted` hook instead.
|
||||
fn on_blob_reaped(&self, _blob_hash: &str) {}
|
||||
}
|
||||
|
||||
/// The set of registered [`BlobReferenceSource`]s.
|
||||
///
|
||||
/// Assembled once during DI and shared (`Arc`) by `dedup_gc` and the
|
||||
/// consistency jobs, so all three agree on what "referenced" means.
|
||||
#[derive(Default)]
|
||||
pub struct BlobReferenceRegistry {
|
||||
sources: Vec<Arc<dyn BlobReferenceSource>>,
|
||||
}
|
||||
|
||||
impl BlobReferenceRegistry {
|
||||
pub fn new() -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
/// Register a source. Order is irrelevant — fragments are summed and
|
||||
/// counts added.
|
||||
pub fn register(&mut self, source: Arc<dyn BlobReferenceSource>) {
|
||||
self.sources.push(source);
|
||||
}
|
||||
|
||||
pub fn sources(&self) -> &[Arc<dyn BlobReferenceSource>] {
|
||||
&self.sources
|
||||
}
|
||||
|
||||
/// The summed SQL expression counting every source's references at
|
||||
/// `level` to `outer_hash_expr`.
|
||||
///
|
||||
/// Returns `"0"` when no source contributes at this level, which keeps
|
||||
/// the caller's query valid without a special case.
|
||||
pub fn ref_count_expr(&self, level: RefLevel, outer_hash_expr: &str) -> String {
|
||||
let fragments: Vec<String> = self
|
||||
.sources
|
||||
.iter()
|
||||
.filter_map(|s| s.ref_count_sql(level, outer_hash_expr))
|
||||
.collect();
|
||||
|
||||
if fragments.is_empty() {
|
||||
"0".to_string()
|
||||
} else {
|
||||
fragments.join("\n + ")
|
||||
}
|
||||
}
|
||||
|
||||
/// Total references held on `hash` across every source.
|
||||
///
|
||||
/// On-demand path only — see [`BlobReferenceSource::count_references`].
|
||||
pub async fn total_references(&self, hash: &str) -> Result<u64, DomainError> {
|
||||
let mut total = 0u64;
|
||||
for source in &self.sources {
|
||||
total = total.saturating_add(source.count_references(hash).await?);
|
||||
}
|
||||
Ok(total)
|
||||
}
|
||||
|
||||
/// Fan out a reap notification to every source.
|
||||
pub fn notify_reaped(&self, blob_hash: &str) {
|
||||
for source in &self.sources {
|
||||
source.on_blob_reaped(blob_hash);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
struct Stub {
|
||||
name: &'static str,
|
||||
chunk: Option<&'static str>,
|
||||
manifest: Option<&'static str>,
|
||||
count: u64,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl BlobReferenceSource for Stub {
|
||||
fn source_name(&self) -> &'static str {
|
||||
self.name
|
||||
}
|
||||
|
||||
fn ref_count_sql(&self, level: RefLevel, outer: &str) -> Option<String> {
|
||||
let tmpl = match level {
|
||||
RefLevel::Chunk => self.chunk?,
|
||||
RefLevel::Manifest => self.manifest?,
|
||||
};
|
||||
Some(tmpl.replace("{outer}", outer))
|
||||
}
|
||||
|
||||
async fn count_references(&self, _blob_hash: &str) -> Result<u64, DomainError> {
|
||||
Ok(self.count)
|
||||
}
|
||||
|
||||
async fn list_referenced_blobs(
|
||||
&self,
|
||||
_cursor: Option<Vec<u8>>,
|
||||
_limit: usize,
|
||||
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
|
||||
Ok((Vec::new(), None))
|
||||
}
|
||||
}
|
||||
|
||||
fn registry() -> BlobReferenceRegistry {
|
||||
let mut r = BlobReferenceRegistry::new();
|
||||
r.register(Arc::new(Stub {
|
||||
name: "a",
|
||||
chunk: Some("(SELECT 1 WHERE {outer} = 'x')"),
|
||||
manifest: None,
|
||||
count: 2,
|
||||
}));
|
||||
r.register(Arc::new(Stub {
|
||||
name: "b",
|
||||
chunk: Some("(SELECT 2 WHERE {outer} = 'y')"),
|
||||
manifest: Some("(SELECT 3 WHERE {outer} = 'z')"),
|
||||
count: 5,
|
||||
}));
|
||||
r
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunk_level_sums_every_contributing_source() {
|
||||
let expr = registry().ref_count_expr(RefLevel::Chunk, "b.hash");
|
||||
assert!(expr.contains("b.hash = 'x'"), "{expr}");
|
||||
assert!(expr.contains("b.hash = 'y'"), "{expr}");
|
||||
assert!(expr.contains('+'), "fragments must be summed: {expr}");
|
||||
}
|
||||
|
||||
/// A source returning `None` for a level must contribute nothing there —
|
||||
/// this is what keeps manifest-only tables out of the chunk recompute,
|
||||
/// where they would double-count against the single-chunk hash alias.
|
||||
#[test]
|
||||
fn manifest_level_skips_non_contributing_sources() {
|
||||
let expr = registry().ref_count_expr(RefLevel::Manifest, "m.file_hash");
|
||||
assert!(expr.contains("m.file_hash = 'z'"), "{expr}");
|
||||
assert!(!expr.contains('+'), "only one source contributes: {expr}");
|
||||
}
|
||||
|
||||
/// An empty level must still yield a valid scalar expression, so callers
|
||||
/// need no special case before a registry is fully populated.
|
||||
#[test]
|
||||
fn empty_level_yields_zero_literal() {
|
||||
let r = BlobReferenceRegistry::new();
|
||||
assert_eq!(r.ref_count_expr(RefLevel::Chunk, "b.hash"), "0");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn total_references_adds_across_sources() {
|
||||
assert_eq!(registry().total_references("deadbeef").await.unwrap(), 7);
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
pub mod auth_ports;
|
||||
pub mod authorization_ports;
|
||||
pub mod blob_lifecycle;
|
||||
pub mod blob_reference_ports;
|
||||
pub mod blob_storage_ports;
|
||||
pub mod cache_ports;
|
||||
pub mod calendar_ports;
|
||||
|
||||
@@ -0,0 +1,328 @@
|
||||
//! The two implicit blob-reference sources, made explicit.
|
||||
//!
|
||||
//! Before this module, "who references this hash" lived as hardcoded SQL
|
||||
//! inside `blobs_consistency`'s refcount recompute and `dedup_gc`'s reap
|
||||
//! predicate. These two implementations reproduce that SQL **exactly** —
|
||||
//! the fragments below sum to today's `actual_ref_count` expression — so
|
||||
//! the registry can be wired in without changing any observed count.
|
||||
//!
|
||||
//! See `docs/plan/derived-blobs.md` and
|
||||
//! [`crate::application::ports::blob_reference_ports`].
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use sqlx::{PgPool, Row};
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::application::ports::blob_reference_ports::{BlobReferenceSource, RefLevel};
|
||||
use crate::domain::errors::DomainError;
|
||||
|
||||
/// Aliases used inside the emitted fragments.
|
||||
///
|
||||
/// Deliberately distinct from the aliases the sweeps use for their outer
|
||||
/// row (`b` for `storage.blobs`, `m` for `storage.chunk_manifests`): a
|
||||
/// fragment reusing `m` would shadow the outer alias in the manifest-level
|
||||
/// sweep and silently correlate against itself.
|
||||
const FILES_ALIAS: &str = "cnt_f";
|
||||
const MANIFEST_ALIAS: &str = "cnt_m";
|
||||
|
||||
/// Fragment for [`FilesReferenceSource`], as a free function so the SQL
|
||||
/// shape can be tested without constructing a pool — it is a property of
|
||||
/// the module, not of an instance.
|
||||
fn files_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||
let f = FILES_ALIAS;
|
||||
match level {
|
||||
// Legacy whole-file blobs only — CDC files are counted at the
|
||||
// manifest level, and counting them here too would double up on the
|
||||
// single-chunk hash alias.
|
||||
RefLevel::Chunk => Some(format!(
|
||||
"(SELECT COUNT(*) FROM storage.files {f}
|
||||
WHERE {f}.blob_hash = {outer_hash_expr}
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM storage.chunk_manifests {MANIFEST_ALIAS}
|
||||
WHERE {MANIFEST_ALIAS}.file_hash = {f}.blob_hash
|
||||
))"
|
||||
)),
|
||||
RefLevel::Manifest => Some(format!(
|
||||
"(SELECT COUNT(*) FROM storage.files {f}
|
||||
WHERE {f}.blob_hash = {outer_hash_expr})"
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Fragment for [`ChunksReferenceSource`]. See [`files_ref_sql`].
|
||||
fn chunks_ref_sql(level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||
match level {
|
||||
RefLevel::Chunk => {
|
||||
let m = MANIFEST_ALIAS;
|
||||
Some(format!(
|
||||
"(SELECT COUNT(*) FROM storage.chunk_manifests {m}
|
||||
WHERE {outer_hash_expr} = ANY({m}.chunk_hashes))"
|
||||
))
|
||||
}
|
||||
// A manifest is never referenced by another manifest.
|
||||
RefLevel::Manifest => None,
|
||||
}
|
||||
}
|
||||
|
||||
// ─── storage.files ───────────────────────────────────────────────────────
|
||||
|
||||
/// References held by `storage.files.blob_hash`.
|
||||
///
|
||||
/// Contributes at **both** levels, which is why `RefLevel` is a parameter
|
||||
/// rather than a property of the source:
|
||||
///
|
||||
/// * [`RefLevel::Manifest`] — a CDC file's `blob_hash` names a manifest.
|
||||
/// * [`RefLevel::Chunk`] — a pre-CDC legacy file, whose `blob_hash` names a
|
||||
/// whole-file blob with no manifest behind it. The `NOT EXISTS` guard is
|
||||
/// load-bearing: for a single-chunk file the whole-file hash *equals* its
|
||||
/// lone chunk's hash, so without it the row would be counted at both
|
||||
/// levels.
|
||||
pub struct FilesReferenceSource {
|
||||
pool: Arc<PgPool>,
|
||||
}
|
||||
|
||||
impl FilesReferenceSource {
|
||||
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||
Self { pool }
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl BlobReferenceSource for FilesReferenceSource {
|
||||
fn source_name(&self) -> &'static str {
|
||||
"files"
|
||||
}
|
||||
|
||||
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||
files_ref_sql(level, outer_hash_expr)
|
||||
}
|
||||
|
||||
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError> {
|
||||
// No level split here: the question is "how many file rows name this
|
||||
// exact hash", and a hash names either a manifest or a legacy blob,
|
||||
// never both at once from the caller's point of view.
|
||||
let n: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM storage.files WHERE blob_hash = $1")
|
||||
.bind(blob_hash)
|
||||
.fetch_one(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error("BlobRefSource", format!("files count: {e}"))
|
||||
})?;
|
||||
Ok(n.max(0) as u64)
|
||||
}
|
||||
|
||||
async fn list_referenced_blobs(
|
||||
&self,
|
||||
cursor: Option<Vec<u8>>,
|
||||
limit: usize,
|
||||
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
|
||||
// Paged by the file's own PK so the cursor is stable under concurrent
|
||||
// inserts; `blob_hash` is not unique and would skip or repeat rows.
|
||||
let after: Option<Uuid> = match cursor {
|
||||
Some(bytes) => Some(decode_uuid_cursor(&bytes)?),
|
||||
None => None,
|
||||
};
|
||||
|
||||
let rows = sqlx::query(
|
||||
"SELECT id, blob_hash FROM storage.files
|
||||
WHERE ($1::uuid IS NULL OR id > $1)
|
||||
ORDER BY id
|
||||
LIMIT $2",
|
||||
)
|
||||
.bind(after)
|
||||
.bind(limit as i64)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("files page: {e}")))?;
|
||||
|
||||
let next = rows
|
||||
.last()
|
||||
.map(|r| r.get::<Uuid, _>("id").as_bytes().to_vec())
|
||||
.filter(|_| rows.len() == limit);
|
||||
let hashes = rows
|
||||
.iter()
|
||||
.map(|r| r.get::<String, _>("blob_hash"))
|
||||
.collect();
|
||||
Ok((hashes, next))
|
||||
}
|
||||
}
|
||||
|
||||
// ─── storage.chunk_manifests ─────────────────────────────────────────────
|
||||
|
||||
/// References held by `storage.chunk_manifests.chunk_hashes[]`.
|
||||
///
|
||||
/// Chunk level only — a manifest never references another manifest, so
|
||||
/// [`RefLevel::Manifest`] yields `None` and this source contributes nothing
|
||||
/// to the manifest recompute.
|
||||
pub struct ChunksReferenceSource {
|
||||
pool: Arc<PgPool>,
|
||||
}
|
||||
|
||||
impl ChunksReferenceSource {
|
||||
pub fn new(pool: Arc<PgPool>) -> Self {
|
||||
Self { pool }
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl BlobReferenceSource for ChunksReferenceSource {
|
||||
fn source_name(&self) -> &'static str {
|
||||
"chunks"
|
||||
}
|
||||
|
||||
fn ref_count_sql(&self, level: RefLevel, outer_hash_expr: &str) -> Option<String> {
|
||||
chunks_ref_sql(level, outer_hash_expr)
|
||||
}
|
||||
|
||||
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError> {
|
||||
let n: i64 = sqlx::query_scalar(
|
||||
"SELECT COUNT(*) FROM storage.chunk_manifests WHERE $1 = ANY(chunk_hashes)",
|
||||
)
|
||||
.bind(blob_hash)
|
||||
.fetch_one(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks count: {e}")))?;
|
||||
Ok(n.max(0) as u64)
|
||||
}
|
||||
|
||||
async fn list_referenced_blobs(
|
||||
&self,
|
||||
cursor: Option<Vec<u8>>,
|
||||
limit: usize,
|
||||
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError> {
|
||||
// Paged by the manifest PK, not by the unnested chunk hash: a single
|
||||
// manifest expands to many hashes, so the page boundary has to fall
|
||||
// between manifests or the cursor cannot be resumed unambiguously.
|
||||
let after: Option<String> = match cursor {
|
||||
Some(bytes) => Some(String::from_utf8(bytes).map_err(|e| {
|
||||
DomainError::internal_error("BlobRefSource", format!("bad chunk cursor: {e}"))
|
||||
})?),
|
||||
None => None,
|
||||
};
|
||||
|
||||
let rows = sqlx::query(
|
||||
"SELECT file_hash, chunk_hashes FROM storage.chunk_manifests
|
||||
WHERE ($1::text IS NULL OR file_hash > $1)
|
||||
ORDER BY file_hash
|
||||
LIMIT $2",
|
||||
)
|
||||
.bind(after)
|
||||
.bind(limit as i64)
|
||||
.fetch_all(self.pool.as_ref())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobRefSource", format!("chunks page: {e}")))?;
|
||||
|
||||
let next = rows
|
||||
.last()
|
||||
.map(|r| r.get::<String, _>("file_hash").into_bytes())
|
||||
.filter(|_| rows.len() == limit);
|
||||
let hashes = rows
|
||||
.iter()
|
||||
.flat_map(|r| r.get::<Vec<String>, _>("chunk_hashes"))
|
||||
.collect();
|
||||
Ok((hashes, next))
|
||||
}
|
||||
}
|
||||
|
||||
fn decode_uuid_cursor(bytes: &[u8]) -> Result<Uuid, DomainError> {
|
||||
let raw: [u8; 16] = bytes.try_into().map_err(|_| {
|
||||
DomainError::internal_error(
|
||||
"BlobRefSource",
|
||||
format!("bad uuid cursor: expected 16 bytes, got {}", bytes.len()),
|
||||
)
|
||||
})?;
|
||||
Ok(Uuid::from_bytes(raw))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::application::ports::blob_reference_ports::BlobReferenceRegistry;
|
||||
|
||||
/// The registry sums whatever the sources emit; these helpers exercise the
|
||||
/// same code path without needing a pool, since `ref_count_sql` is pure.
|
||||
fn summed(level: RefLevel, outer: &str) -> String {
|
||||
let frags: Vec<String> = [files_ref_sql(level, outer), chunks_ref_sql(level, outer)]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.collect();
|
||||
if frags.is_empty() {
|
||||
"0".to_string()
|
||||
} else {
|
||||
frags.join("\n + ")
|
||||
}
|
||||
}
|
||||
|
||||
/// The chunk-level expression must reproduce the two terms
|
||||
/// `blobs_consistency` inlines today: legacy-only files (guarded by
|
||||
/// NOT EXISTS) plus manifests citing the chunk.
|
||||
#[test]
|
||||
fn chunk_level_reproduces_todays_two_terms() {
|
||||
let expr = summed(RefLevel::Chunk, "b.hash");
|
||||
assert!(expr.contains("storage.files"), "{expr}");
|
||||
assert!(
|
||||
expr.contains("NOT EXISTS"),
|
||||
"legacy term must keep the CDC guard: {expr}"
|
||||
);
|
||||
assert!(
|
||||
expr.contains("= ANY(cnt_m.chunk_hashes)"),
|
||||
"chunk term missing: {expr}"
|
||||
);
|
||||
assert!(expr.contains("b.hash"), "must correlate on the outer row");
|
||||
assert!(expr.contains('+'), "both terms must be summed: {expr}");
|
||||
}
|
||||
|
||||
/// Only `storage.files` references a manifest, so the manifest-level
|
||||
/// expression is the single files term with no `NOT EXISTS` guard — the
|
||||
/// guard exists to keep CDC rows *out* of the chunk level, and applying
|
||||
/// it here would count nothing at all.
|
||||
#[test]
|
||||
fn manifest_level_is_files_only_and_unguarded() {
|
||||
let expr = summed(RefLevel::Manifest, "m.file_hash");
|
||||
assert!(expr.contains("storage.files"), "{expr}");
|
||||
assert!(!expr.contains("NOT EXISTS"), "{expr}");
|
||||
assert!(
|
||||
!expr.contains("chunk_hashes"),
|
||||
"chunks must not contribute at manifest level: {expr}"
|
||||
);
|
||||
assert!(!expr.contains('+'), "only one source contributes: {expr}");
|
||||
assert!(expr.contains("m.file_hash"));
|
||||
}
|
||||
|
||||
/// Fragments must not use the aliases the sweeps use for their outer row
|
||||
/// (`b` for storage.blobs, `m` for chunk_manifests), or the manifest sweep
|
||||
/// would shadow its own alias and silently correlate against itself.
|
||||
#[test]
|
||||
fn fragments_avoid_outer_row_aliases() {
|
||||
for level in RefLevel::ALL {
|
||||
let expr = summed(level, "m.file_hash");
|
||||
for bad in [
|
||||
"storage.files f",
|
||||
"storage.files b",
|
||||
"chunk_manifests m ",
|
||||
"chunk_manifests b",
|
||||
] {
|
||||
assert!(!expr.contains(bad), "alias collision at {level:?}: {expr}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A source declining a level must drop out of the sum entirely, which is
|
||||
/// what keeps manifest-only tables out of the chunk recompute where the
|
||||
/// single-chunk hash alias would double-count them.
|
||||
#[test]
|
||||
fn chunks_source_declines_manifest_level() {
|
||||
assert!(chunks_ref_sql(RefLevel::Manifest, "m.file_hash").is_none());
|
||||
assert!(chunks_ref_sql(RefLevel::Chunk, "b.hash").is_some());
|
||||
}
|
||||
|
||||
/// Guards the registry contract the sweeps rely on: an empty level still
|
||||
/// yields a valid scalar expression.
|
||||
#[test]
|
||||
fn empty_registry_yields_zero_literal() {
|
||||
let r = BlobReferenceRegistry::new();
|
||||
assert_eq!(r.ref_count_expr(RefLevel::Manifest, "m.file_hash"), "0");
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
mod address_book_pg_repository;
|
||||
mod app_password_pg_repository;
|
||||
pub mod blob_reference_sources;
|
||||
mod calendar_event_pg_repository;
|
||||
mod calendar_pg_repository;
|
||||
mod contact_group_pg_repository;
|
||||
|
||||
Reference in New Issue
Block a user