cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge pull request #698 from EdouardVanbelle/doc/derrived-blob

Browse Source
This commit is contained in:
Dionisio Pozo
2026-09-01 06:30:21 +02:00
committed by GitHub
parent 30fdaa552b 430aa8c71c
commit 1800fa9a47
4 changed files with 420 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.cargo/audit.toml
+29
View File
@@ -82,6 +82,35 @@ ignore = [
# with its own Store (see infrastructure/services/plugins/runtime.rs).
"RUSTSEC-2026-0222",
# wasmtime 43.0.2 — "Filesystem sandbox escape when paths or symlinks
# contain trailing slashes" (RUSTSEC-2026-0269, 8.8 high). Same crate,
# same chain and same absent upgrade path as RUSTSEC-2026-0222 above:
# extism 1.30.0 is the latest published and pins wasmtime 43, while the
# advisory's fixed releases are >=24.0.13 <25, >=36.0.14 <37,
# >=46.0.3 <47, >=47.0.4 — none in the 43.x line, so there is no
# version satisfying extism's requirement that carries the fix.
#
# NOT REACHABLE, and for a stronger reason than the build-feature
# gating: this is a WASI filesystem sandbox escape, and OxiCloud's
# plugin runtime gives plugins no filesystem to escape from.
# `plugins/runtime.rs::compile` builds every plugin with
# `.with_wasi(false)` and declares no `allowed_paths`, so there are no
# preopened directories — the escape needs one to traverse out of.
# `.disallow_all_hosts()` removes outbound network on the same path.
#
# The build-level gating from the entry above still applies on top:
# `plugins` is opt-in and absent from `default`, so the CI release
# binary does not link wasmtime; runtime activation additionally needs
# OXICLOUD_ENABLE_PLUGINS=true; and plugin binaries are admin-supplied,
# not attacker input.
#
# Un-ignore trigger: EITHER extism releases a version on wasmtime
# >=46.0.3 (check with `cargo tree -i wasmtime --features plugins`),
# OR `plugins/runtime.rs` gains `allowed_paths` / `with_wasi(true)` —
# at which point this stops being unreachable and blocks release
# rather than being ignored.
"RUSTSEC-2026-0269",
# astral-tokio-tar 0.5.6 — tar extraction advisories, transitive via
# testcontainers → testcontainers-modules, a DEV-dependency used only by
# the `--cfg integration_tests` harness to spin up throwaway Postgres
docs/architecture/derived-and-attached-blobs.md
+284
View File
@@ -0,0 +1,284 @@
# Derived and attached blobs
Two tables hang small artifacts off the blob store: thumbnails,
transcodes, uploaded previews. They look almost identical, and the
difference between them is a security boundary rather than a style
choice.
("Satellite tables" is the shorthand used in the code and in
`satellites_consistency`, the job that walks both. It is a useful
collective noun once you know what it covers; this page is what it
covers.)
- **`storage.content_derived_blobs`** — things the *server derived from
file content*. Keyed by the BLAKE3 of the source.
- **`storage.file_attached_blobs`** — things a *user attached to one
specific file*. Keyed by `file_id`.
Both point into the same blob store underneath (see
[Backend Storage](./backend-storage.md)). The keying is what separates
them.
```mermaid
erDiagram
FILES ||--o{ ATTACHED : "file_id — per FILE"
FILES }o--|| BLOBS : "blob_hash (its content)"
BLOBS ||--o{ DERIVED : "source_hash — per CONTENT"
DERIVED }o--o| ARTIFACT : "blob_hash (NULL = negative)"
ATTACHED }o--|| ARTIFACT : "blob_hash"
FILES {
uuid id PK
text blob_hash
}
DERIVED {
text source_hash PK
text kind PK
text variant PK
text blob_hash "NULL = not worth deriving"
text content_type "NULL iff blob_hash NULL"
}
ATTACHED {
uuid file_id PK
text kind PK
text variant PK
text blob_hash
uuid uploaded_by "no FK; nil = imported"
}
ARTIFACT {
text hash PK
}
```
Read the two arrows into `ARTIFACT`: `DERIVED` reaches it from
**content**, `ATTACHED` from a **file**. Everything below follows from
that.
## Why two tables and not one with a `kind` column
Content keying means identical bytes share one derivation. Upload the
same photo twice and the server renders its thumbnail **once** — both
files resolve to the same `source_hash`, find the same row, and serve
the same blob. That is the entire point for server-derived artifacts:
the derivation is a pure function of the content, so sharing it is
free and correct.
Apply the same keying to *user-supplied* bytes and it becomes an
attack. If uploaded previews were content-keyed, uploading a file whose
content matches someone else's would let you replace the preview they
see — or read yours in place of theirs. The preview is not derived from
the content; it is an assertion *about* a file, made by whoever uploaded
it, and two people can hold different assertions about identical bytes.
A single table with a `kind` discriminator could not express this. The
key would have to be one thing or the other, and either choice is wrong
for half the rows. The split is the enforcement mechanism, not a
filing convenience — which is why `thumb_derived_import` explicitly
refuses `ext-` filenames and `thumb_attached_import` explicitly refuses
hash-named ones, rather than one job handling both trees.
## `storage.content_derived_blobs`
| column | type | notes |
|---|---|---|
| `source_hash` | `VARCHAR(64)` | PK. BLAKE3 of the **source** content. Dependent reference — holds no refcount; the row is reaped with its source. |
| `kind` | `TEXT` | PK. `thumbnail` \| `transcode` (CHECK-constrained). |
| `variant` | `TEXT` | PK. Opaque rendering discriminator — see [Variant](#variant-holds-every-axis-that-can-vary). |
| `blob_hash` | `VARCHAR(64)` | The derived artifact, **or NULL** — see [Negative rows](#negative-rows). Reference **holder** when present. |
| `content_type` | `TEXT` | MIME of the artifact. NULL exactly when `blob_hash` is NULL. |
| `created_at` | `TIMESTAMPTZ` | |
A CHECK keeps `blob_hash` and `content_type` NULL together: a type
without bytes describes nothing, and bytes without a type cannot be
served.
## `storage.file_attached_blobs`
| column | type | notes |
|---|---|---|
| `file_id` | `UUID` | PK. FK to `storage.files` **ON DELETE CASCADE**. |
| `kind` | `TEXT` | PK. `preview` \| `subtitle` \| `cover_art` (CHECK-constrained). |
| `variant` | `TEXT` | PK. |
| `blob_hash` | `VARCHAR(64)` | `NOT NULL` — there is no negative case here. |
| `content_type` | `TEXT` | `NOT NULL`. |
| `uploaded_by` | `UUID` | `NOT NULL`, and deliberately **no FK**. |
| `created_at` | `TIMESTAMPTZ` | |
`uploaded_by` follows the provenance convention: an FK with
`ON DELETE SET NULL` would erase the audit trail exactly when it matters
most, and without an `ON DELETE` clause it would block deleting a user
at all. Deleting the uploader must not rewrite history, so the id is
kept even once it no longer resolves. Rows created by the migration
carry the all-zeros sentinel — "imported, uploader unknown" — rather
than a fabricated owner such as the file's `created_by`, which could
later be misread as evidence that someone replaced a preview.
## Negative rows
`content_derived_blobs.blob_hash` is nullable, and a NULL row means:
**this derivation was attempted and is known not to be worth storing
for this content.**
The case that motivated it: `ImageTranscodeService` can only discover
that WebP comes out *larger* than the original by doing the full decode
and re-encode. Without a record, every request repeats that work to
throw the result away. The same applies to a source that cannot be
decoded, or one over the decode ceiling.
Only failures **deterministic in the content** may be recorded. A
timeout, a closed semaphore, an I/O error reading the source are
properties of the moment, not the bytes; persisting one marks a
perfectly good image as underivable forever, with nothing to retry it.
The asymmetry sets the default — a wrongly-cached transient is silent
and permanent, a missing negative merely costs repeated work — so **when
in doubt, do not write the row.**
A sentinel hash was considered and rejected: it would stop `blob_hash`
naming a real blob, and every consumer joining on it would need to learn
the exception or silently mishandle it. NULL is already SQL's way of
saying "no blob", and joins drop it naturally.
`file_attached_blobs` has no negative case. There is nothing to attempt
— the bytes either arrived from a client or they did not.
### The NULL trap
This has caused two bugs, both found before shipping, and it will cause
more. SQL comparison against NULL yields NULL, so:
```sql
EXISTS (SELECT 1 FROM storage.blobs b WHERE b.hash = d.blob_hash)
```
is **false** for every negative row. Whether that is right depends
entirely on what you are asking:
- **Refcounts — correct.** A negative row holds no reference, so it must
not contribute. `content_derived_ref_sql` relies on exactly this.
- **Dangling checks — wrong.** `satellites_consistency` reported every
negative row as `derived_dangling_blob` at `data_loss` severity: a row
correctly pointing at nothing, reported as an artifact that had gone
missing. It needs `d.blob_hash IS NULL OR <exists>`.
- **Enumeration — wrong, and it fails loudly.**
`list_referenced_blobs` decodes `blob_hash` into `String`; the first
NULL takes the whole sweep down. It needs `WHERE blob_hash IS NOT NULL`.
Anything joining on `blob_hash` has to decide which of these it is.
## `variant` holds every axis that can vary
`variant` is an opaque discriminator, and the schema states the rule:
new axes go **inside this string, never into new columns**.
The two tables therefore look asymmetric, and correctly so:
| table | variant | why |
|---|---|---|
| `content_derived_blobs` | `icon.webp`, `preview.jpg` | size **and** format |
| `file_attached_blobs` | `icon` | size only |
A single source legitimately has two thumbnails at one size — WebP for
capable clients, JPEG for the rest — and since the PK is
`(source_hash, kind, variant)`, the format must be inside `variant` or
those rows collide and only one can exist. Uploaded previews have no
format axis: `store_external_thumbnail` re-encodes to JPEG on write, so
`image/jpeg` is a constant and `.jpg` in every variant would carry no
information.
**Rejected alternative: key on `content_type` instead.** Making
`content_type` part of the key would express the same thing, and it was
reasonable until negative rows landed. It is now foreclosed —
`content_type` must be nullable for negative rows, and PostgreSQL does
not allow a nullable column in a primary key. A UNIQUE constraint would
not rescue it either: NULLs compare as *distinct* in a unique index, so
duplicate negative rows for one `(source_hash, kind, variant)` would
become possible, and that row's singularity is what the mechanism
depends on. Two softer objections stand regardless — `content_type` is a
presentation value, and MIME strings are not canonical (`image/jpg` and
`image/jpeg` name one format).
**If the assumption changes**, the migration has a known shape.
`20261022000000_derived_variant_encodes_format.sql` did it once for the
derived table: append the format to existing variants, update the
callsites that build the string. Nothing is lost in the meantime —
`content_type` records the real format — it simply is not part of the
key, so two formats cannot coexist until it is.
## Worked examples
**The same image uploaded twice.** Two `storage.files` rows, one
`blob_hash` between them, **one** `content_derived_blobs` row per
`(kind, variant)`, one thumbnail blob. The second upload renders
nothing; it finds the existing row. Copying either file adds no row at
all — the copy shares the source hash, so it resolves to the same
derivation.
**A PDF with a client-uploaded preview.** One `file_attached_blobs` row
keyed by that `file_id`. Copy the file and the row is **duplicated** for
the new id (`storage.copy_file_satellites`), because the preview belongs
to the file, not to the content. Without that duplication the copy
silently loses its preview — and a PDF has no server-side render path,
so nothing regenerates it.
**A screenshot WebP cannot shrink.** One row, `blob_hash` and
`content_type` both NULL. A reader concludes: the transcode was
attempted, it is known not to help *for this content*, serve the
original and do not retry. Every file sharing those bytes inherits the
verdict.
**One source, thumbnailed and transcoded.** Two rows, same
`source_hash`, `kind` of `thumbnail` and `transcode`. Add a JPEG
fallback thumbnail and it is a third row, differing only in `variant`
(`icon.webp` vs `icon.jpg`).
## Lifecycle
**References.** A positive `blob_hash` is a reference *holder* — it
bumps `chunk_manifests.ref_count` through `DedupService::add_reference`,
so `dedup_gc` cannot reap an artifact a satellite still points at. A
negative row holds none. `source_hash` is a *dependent* reference and
holds nothing.
**Reaping.** Derived rows are removed by `purge_derived_blobs` when
their source blob is reaped. Attached rows vanish by
`ON DELETE CASCADE` when their file is deleted — which happens **inside
the database**, where the Rust lifecycle hooks cannot observe it, so
`trg_file_attached_blobs_decrement_blob_ref` releases the blob reference
on `DELETE`. The trigger fires on DELETE only; replacing a preview
updates `blob_hash` in place and the Rust path handles that reference
swap.
**Writing a derived row requires its source to exist.**
`store_derived_blob` guards the insert with an `EXISTS` on
`chunk_manifests`/`blobs`. Without it, a row written just after its
source was reaped would pin its artifact forever: nothing would ever
reap that `source_hash` again, so `purge_derived_blobs` could never
fire. That is not hypothetical — it shipped once, as a permanent blob
leak.
**Consistency coverage.** `satellites_consistency` is the only job that
walks these tables, and it exists because of a gap the others cannot
close: a satellite row whose *source* is gone breaks no invariant any
other check looks at. The reference is valid, the refcount is correct,
the bytes are present — every Blob-centric job agrees the system is
healthy while the artifact is pinned forever. Blob-side integrity
(missing bytes, orphans, bit-rot) belongs to `backend_consistency`;
refcount arithmetic to `blobs_consistency` and
`manifests_consistency`.
## Adding a third artifact type
Ask one question first: **is it derived from the content, or asserted
about a file?**
Derived from content — a waveform, an extracted page count, an OCR
layer — goes in `content_derived_blobs` under a new `kind`, and shares
across identical content for free.
Supplied by a user — a custom cover image, a hand-authored subtitle
track — goes in `file_attached_blobs`, and must be duplicated on copy
rather than shared.
Getting that backwards is not a performance mistake. Putting
user-supplied bytes in the content-keyed table means one user's upload
is served to everyone whose file happens to match.
docs/architecture/index.md
+1
View File
@@ -75,4 +75,5 @@ src/
- [Resource Listing API →](/architecture/resource-listing)
- [Storage Quotas →](/architecture/storage-quotas)
- [Backend Storage →](/architecture/backend-storage)
- [Derived and Attached Blobs →](/architecture/derived-and-attached-blobs) — thumbnails, transcodes and uploaded previews: why content-keyed and file-keyed artifacts need separate tables
- [Background Jobs →](/architecture/jobs)
docs/plan/derived-blobs.md
+106
View File
@@ -1567,6 +1567,112 @@ hardcoded SQL). New sources bolt on independently.
land at any point; last is easiest, since every earlier slice
would otherwise rebase across it.
12. **Document the two satellite tables** — `storage.content_derived_blobs`
and `storage.file_attached_blobs`, with worked examples. **Not
started.**
The tables exist and each carries `COMMENT ON` text, but nothing
explains the pair *together*, and the thing that matters about them
is the relationship: they hold the same kind of artifact under two
different keys, and **the keying difference is the security
boundary**. Someone adding a third artifact type has to get that
right, and today the only place it is written down is prose
scattered through this plan and two job doc-comments.
What the page needs:
* **Column-by-column structure** of both tables, including the
parts that are easy to get wrong: `blob_hash` is nullable on
`content_derived_blobs` (a NULL is a NEGATIVE row — the
derivation was attempted and is known not to be worth storing),
paired with `content_type` by a CHECK; `uploaded_by` on
`file_attached_blobs` is `NOT NULL` with no FK, so provenance
survives a user deletion, and imported rows carry the nil-UUID
sentinel rather than a fabricated owner.
* **Why two tables and not one with a discriminator.**
`content_derived_blobs` is keyed by the BLAKE3 of the SOURCE, so
identical content shares one derivation — which is exactly what
must NOT happen for user-supplied bytes, or one user's uploaded
preview would be served for every file with matching content.
That is the poisoning vector; the split is what prevents it, and
a `kind` column on a single table would not.
* **Worked examples**, which is what turns the rule into something
checkable:
- the same image uploaded twice → one `content_derived_blobs` row,
two files, one thumbnail blob;
- a PDF with a client-uploaded preview → one `file_attached_blobs`
row per file, duplicated on copy, never shared;
- a screenshot WebP cannot shrink → a negative row with NULL
`blob_hash`, and what a reader is expected to conclude from it;
- the same content transcoded and thumbnailed → two rows, same
`source_hash`, different `kind`.
* **Lifecycle**: which rows hold a reference (positive
`blob_hash` bumps `chunk_manifests.ref_count`; a negative row
holds none), what reaps them (`purge_derived_blobs` on source
reap; the decrement trigger on `file_attached_blobs`), and which
consistency job covers which failure — the point being that a
satellite row whose SOURCE is gone breaks no refcount invariant,
which is why `satellites_consistency` had to exist at all.
* **The NULL-handling trap**, worth a paragraph because it has
already bitten twice: SQL comparison against NULL is NULL, so
`EXISTS (… WHERE blob_hash = h)` silently excludes negative rows.
That is correct for refcounts (a negative row holds none) and
WRONG for dangling checks (it reported every negative row as
`data_loss`). Anything joining on `blob_hash` must decide which
of the two it wants.
* **A diagram**, because the security boundary is a shape before it
is a rule — two arrows starting from different places is the
whole argument, and it lands faster than any paragraph:
```mermaid
erDiagram
FILES ||--o{ ATTACHED : "file_id — per FILE"
FILES }o--|| BLOBS : "blob_hash (content)"
BLOBS ||--o{ DERIVED : "source_hash — per CONTENT"
DERIVED }o--o| ARTIFACT : "blob_hash (NULL = negative)"
ATTACHED }o--|| ARTIFACT : "blob_hash"
FILES {
uuid id
text blob_hash
}
DERIVED {
text source_hash PK
text kind PK
text variant PK
text blob_hash "NULL = not worth deriving"
text content_type "NULL iff blob_hash NULL"
}
ATTACHED {
uuid file_id PK
text kind PK
text variant PK
text blob_hash
uuid uploaded_by "no FK; nil = imported"
}
ARTIFACT {
text hash PK
}
```
What a reader should take from it: `DERIVED` hangs off **content**,
so two files with identical bytes reach the same row — good for a
render, catastrophic for an upload. `ATTACHED` hangs off the
**file**, so those rows are duplicated on copy and never shared.
Both point at the same artifact space, which is why one table
could not simply be folded into the other with a `kind` column.
**Landed** as `docs/architecture/derived-and-attached-blobs.md`,
beside `backend-storage.md`, which documents the blob layer these
sit on top of. Operator-facing rather than end-user, so schema and
SQL belong here in a way they do not in `docs/guide/`.
Named for the two things rather than for "satellite tables" —
that is internal shorthand nobody would search for, whereas
*derived* and *attached* are the words the schema and the import
jobs already use.
Tracked separately, **not** part of this plan: the
`storage.copy_folder_tree` refcount bug (see the copy section). It is
a production data-loss bug on a path this plan doesn't otherwise