From 4cb73eaf3982349e9ed90728d65f62b4052de252 Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Sat, 1 Aug 2026 20:42:24 +0200 Subject: [PATCH] plan(storage-key-rotation): add a key rotation + header version blob --- docs/plan/storage-key-rotation.md | 609 ++++++++++++++++++++++++++++++ 1 file changed, 609 insertions(+) create mode 100644 docs/plan/storage-key-rotation.md diff --git a/docs/plan/storage-key-rotation.md b/docs/plan/storage-key-rotation.md new file mode 100644 index 00000000..b75098b2 --- /dev/null +++ b/docs/plan/storage-key-rotation.md @@ -0,0 +1,609 @@ +# Plan — Storage encryption-key rotation + +## Context + +Today an encrypted storage entry declares a single key: + +```env +OXICLOUD_STORAGE_s3_prod_BACKEND=s3 +OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY= +OXICLOUD_STORAGE_s3_prod_ENCRYPTION_CIPHER=aes-256-gcm # optional, defaults to aes-256-gcm +``` + +Blobs land at `.blob` as raw AES-GCM output (`nonce | ciphertext | tag`) — +no discriminating prefix. That key can never change. Any real deployment needs +to rotate keys — after a suspected leak, on a periodic policy, or after a staff +turnover. The current answer is "create a second entry and run a migration to +it," which: + +* forces the admin to provision a second bucket / directory, +* churns object storage costs, +* and does not scale to routine rotations. + +The proper fix is **in-place key rotation** inside the same entry. The storage +config holds a *list* of keys, the server writes with the newest one, reads +with any of them, and a background job re-encrypts every blob under the newest +key. When the job completes, the admin removes the old key from the list. + +At the same time we solve two related on-disk problems that today's format +leaves open: + +1. **No self-description.** A `.blob` today is ambiguous: could be raw + plaintext (unencrypted deployment) or raw AES-GCM output (encrypted + deployment). Reads guess based on the entry config; a misconfigured key or + a wrong-cipher default silently returns garbage. +2. **No path from encrypted to plaintext.** Once encryption is on for an + entry, the only way off is via a second entry and full migration — same + ergonomics problem as key rotation. + +Both fall out of introducing a self-describing on-blob header (`v1`). The +object-key suffix stays `.blob` for both eras — the magic bytes at the +top of every v1 blob are the discriminator, so legacy files stay readable +byte-identically and new files coexist alongside them at the same suffix. +Migration is lazy. + +Related memory (this plan supersedes it): + +> **[Encryption key rotation in place — deferred, unsafe today]** — the previous +> "namespace object keys by encryption generation" direction (`.k2.blob`) +> is DROPPED in favour of the pair-list + v1-header approach below. That +> approach fought content-addressability by generation-namespacing per key; +> this one keeps a stable content-addressable object key and encodes generation +> in a compact on-blob header. + +## Design + +### The pair-list config + +Replace the singular `_ENCRYPTION_KEY` + `_ENCRYPTION_CIPHER` with a single +comma-separated list of pairs on `_ENCRYPTION_KEY` alone: + +```env +OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:,aes-256-gcm: +``` + +Rules: + +* **Format** — `[:]` per pair, comma-separated. Whitespace + around commas / colons tolerated. +* **Cipher optional.** Only one real cipher (`aes-256-gcm`) exists today, so + `` on its own is legal and behaves as `aes-256-gcm:`. When a + second cipher lands, the colon-prefixed form is the disambiguator. +* **The `none` cipher.** Legal, has no key material (`none:` — trailing colon + with an empty key). Enables migrating BOTH ways: encrypt a previously-plaintext + deployment (`none:,aes-256-gcm:`), or decrypt an encrypted deployment + (`aes-256-gcm:,none:`). Correctness relies on the v1 header (see + *Encrypted-blob format*) discriminating encrypted vs plaintext, NOT on + fallback ordering. A `none` pair by itself in the list is equivalent to + omitting `_ENCRYPTION_KEY` entirely (kept for symmetry). +* **Order matters. Last pair wins on writes.** For v1-header reads, the + `key_fp` in the header selects the exact pair; the list order is irrelevant + at read time. For legacy reads (see *Coexistence*), the head pair is used + because pre-v1 deployments had only one key by construction. +* **At least one pair.** Empty list → boot aborts. +* **Uniqueness.** Same key appearing twice = boot aborts (config drift smell). + `none` may appear at most once. +* **Fingerprint.** For each real-cipher pair, log a truncated SHA-256 of the + key material at boot (12 hex chars) so a swap-order accident is loud in the + audit stream. `none` logs as `none:—`. + +### Retire `_ENCRYPTION_CIPHER` + +`_ENCRYPTION_CIPHER` was added recently and has not shipped in a release. Drop +it entirely in the same slice — the pair-list makes it redundant, and the on-blob +v1 header carries the cipher choice explicitly per blob anyway. No back-compat +shim needed. + +### Encrypted-blob format (v1 header) + +Every v1 blob starts with a fixed self-describing header. Layout: + +``` +"OXCPT" 5 bytes — magic marker (0x4F 0x58 0x43 0x50 0x54) + 2 bytes — big-endian u16; v1 = 0x0001 + 8 bytes — sha256(key material)[..8]; identifies which pair + decrypts. For a `none` (plaintext) v1 blob this is + eight zero bytes and the fields below are absent. + 12 bytes — random per blob (AES-GCM standard nonce length) + N bytes — encrypted payload; same length as the plaintext + 16 bytes — AEAD authentication tag + ──── +fixed overhead: 43 bytes for encrypted v1; 15 bytes for plaintext v1 +``` + +* **The magic `OXCPT`** unambiguously distinguishes a v1 blob from legacy raw + bytes. It's the sole discriminator — no filename convention, no DB column, + the first five bytes tell the reader everything. +* **``** is the only field that can change format without touching + every blob. v1 is what this plan ships; v2 slots are reserved + (see *Forward compatibility*). +* **``** eliminates guess-work on read. The pair-list lookup becomes + O(1) — no fallback tag-check loop. A key-fp with no matching pair means + `NoKeyForBlob` (500 with a clear message), NEVER silent garbage. +* **``** and **``** are the AES-GCM primitive's own outputs. + Their length is fixed by v1 = AES-256-GCM. If a future v2 uses a different + AEAD with the same shape (say ChaCha20-Poly1305, tag = 16 B, nonce = 12 B) + v2 can keep the layout; a truly-different AEAD triggers a new version. +* **Plaintext v1 blobs** (produced when the entry's head pair is `none`) use + the same magic + version + all-zero key_fp, then the raw plaintext bytes + directly (no nonce, no ciphertext framing, no tag). The header alone is + enough to say "this is plaintext v1", cleanly distinguished from encrypted + v1 and from legacy raw-plaintext. + +**Why no CRC.** The AEAD tag covers ciphertext + nonce integrity. Content +addressability (`` = BLAKE3(plaintext)) covers whole-blob integrity for +either format. A CRC over the header adds nothing — a corrupt version byte +fails on decode; a corrupt key_fp fails to find a pair. Both fail loud. + +### Coexistence with legacy blobs + +Legacy and v1 blobs share the `.blob` object-key namespace. The magic +bytes at position 0 discriminate them safely: + +| First 5 bytes | Interpretation | Read path | +|---|---|---| +| `"OXCPT"` | v1 blob | verify magic, dispatch on version + key_fp | +| anything else | legacy | apply entry config: raw plaintext if no encryption pair, single-key AES-GCM decode with the head pair if encryption is set (by construction pre-v1 used one key, and any pair-list produced by upgrading from a pre-v1 config keeps that key at the head) | + +Collision probability of legacy bytes accidentally matching `"OXCPT"` at +position 0 is 2⁻⁴⁰ per blob. If a match does happen, the read continues into +the v1 path, fails at version/key_fp check with `UnsupportedBlobVersion` or +`NoKeyForBlob`, and returns a hard error. **Never silent corruption.** For +context: on a 10 M-blob deployment, expected number of collision hard-errors +across the lifetime of legacy files is ~10⁻⁵ blobs. Effectively never. + +Properties this buys us: + +* **No forced migration.** Legacy `.blob` files stay readable + indefinitely. Existing deployments upgrade to v1 code with zero + format-conversion work. +* **Lazy conversion on hot paths.** Any COW overwrite (WebDAV MOVE, PUT-over, + content-hash re-upload) naturally lands as v1 at the same object key. +* **Explicit conversion via `storage_rotate`.** The rotate job walks + `storage.blobs`, reads each blob via the magic-byte dispatch, and if the + blob is not already v1 with the head-pair key, PUTs it back as v1 with + the head pair — in place, same object key. +* **No schema changes.** Nothing new in `storage.blobs`; nothing for + `blobs_consistency`, `storage_migrate`, or any other sibling job to learn. + +### Legacy-pair guardrail + +Once any legacy blob remains on disk, the pre-v1 single key that produced +those blobs MUST remain in the pair-list at the head position (or in the list +somewhere the legacy read path can find it). Removing it early breaks every +legacy read. + +Guardrail: + +* Boot logs a `warn` line if any pair-list has more than one entry AND the + head pair's fingerprint differs from the second entry's — signals "you + added new keys but haven't rotated legacy blobs yet". +* A **legacy-blob counter** is surfaced in the admin panel per storage entry. + The counter is maintained by `blobs_consistency`: during its normal walk it + branches on the magic-byte check and records the legacy count as a run + statistic on `jobs.recoverable_runs` (existing surface, no schema hit). The + admin panel reads the most recent count and displays it. Refresh cadence is + whatever consistency scan cadence the deployment has (weekly by default; on + demand from the admin panel). +* The *"Rotation complete — safe to remove the old key"* hint appears in the + entry card only when the last `storage_rotate` run completed with zero + findings AND the most recent consistency scan reported zero legacy blobs. +* Nothing enforces removal at code level — the admin is trusted, given a + clear signal, and warned. + +### Read path + +Per blob read: + +1. Fetch `.blob` from the backend. +2. Check first 5 bytes. +3. If `"OXCPT"` → v1 read path: + * Read ``. Not `0x0001` → `UnsupportedBlobVersion`. + * Read ``. If zero → plaintext-v1; return raw bytes (post-header + payload). Else find the pair whose `sha256(key)[..8]` matches; not + found → `NoKeyForBlob`; found → AES-GCM decrypt with `` / + `` / `` and return plaintext. +4. Otherwise → legacy read path: + * If entry has any real-cipher pair, attempt AES-GCM decrypt with the + head pair's key; return plaintext on success, `DecryptFailed` on tag + failure. + * If entry has only `none` (or no pair at all), return raw bytes as + plaintext. + +Never falls through silently. Every failure is a distinct typed error the +handler can map to a 500 with an actionable message. + +### Write path + +1. Always writes to `.blob`. +2. Head pair is `none` → write v1 header (magic + version + zero key_fp) + + raw plaintext. +3. Head pair is a real cipher → compute AEAD; write v1 header (magic + + version + key_fp) + nonce + ciphertext + auth_tag. + +v1 code never produces a legacy-format blob. Any leftover legacy blobs on +disk pre-date the v1-code deployment. + +### The rotation job + +New `RecoverableJobHandler` tenant, `storage_rotate`. Mirrors +`storage_migration`'s shape: + +* **Iterates `storage.blobs`** in hash-lex order. Cursor is the last-processed + hash (64 hex chars). Same cursor encoding as `storage_migration` and + `blobs_consistency`. +* **Per blob:** + 1. Fetch `.blob` and dispatch via the standard read path. + 2. Decide if a rewrite is needed based on what actually decoded: + * Legacy blob (no v1 magic) → always rewrite (upgrade to v1 with head + pair). + * v1 encrypted, decrypted under a pair-index other than head → rewrite + (key rotation). + * v1 encrypted, already under head — skip. + * v1 plaintext, head pair is `none` — skip. + * v1 plaintext, head pair is a real cipher — rewrite + (encrypt-in-place upgrade). + * v1 encrypted, head pair is `none` — rewrite + (decrypt-in-place downgrade). + 3. Write via the standard v1 write path — same object key, atomic + replace. + 4. Checkpoint. On per-blob failure, record a `rotation_failed` finding + with severity `data_loss` (bytes may not have crossed), continue. +* **Concurrency-safe by construction.** + * A concurrent upload during rotation writes v1 with the head key. The + rotate walk will short-circuit on that hash if it reaches it later. + * A COW overwrite is the same story. + * The v1 write is atomic at object-storage level (S3 replace, Local + rename-into-place). A concurrent reader sees either state. + * No readonly mode. This is a critical improvement over + `storage_migration`: rotation is per-blob idempotent, so we don't need + to freeze writes. +* **Restart-survivable** — same boot-time sweep as every other recoverable + handler. +* **`?deep=true` unused** — rotation has no "slow variant" mode. Parameter + accepted for uniformity, ignored. + +### Admin UX + +In the admin storage-panel entry card, add a **Rotate encryption key** action. +Preconditions: + +* Entry has encryption enabled with ≥ 2 pairs OR entry has legacy blobs + outstanding OR the pair-list has otherwise changed since the last rotation. +* Otherwise the button is disabled with the tooltip *"Add a second pair in + `OXICLOUD_STORAGE__ENCRYPTION_KEY` first, or wait for legacy blobs to + accumulate — nothing to rotate right now."* + +Clicking the button dispatches the `storage_rotate` job for that entry. The +job's progress rides on the same `X-Server-Status` header infrastructure the +maintenance banner uses — but this time WITHOUT engaging read-only mode. +Banner variant reads *"Rotating encryption key on `` — X% (Y / Z +blobs). All operations continue normally."* and disappears on completion. + +The entry card shows a **legacy-blob counter** sourced from the most recent +`blobs_consistency` run: *"N legacy-format blobs remain (upgrade included in +next rotation)"*. Refresh-on-demand button next to it triggers a targeted +`blobs_consistency` scan (already available via the admin surface). The +*"Rotation complete — safe to remove the old key"* hint appears only when +N = 0 and the last `storage_rotate` run completed with zero findings. + +### Removing the old pair + +Not automated. The admin edits `.env`, restarts. Rationale: + +* We do not want the running server to silently mutate its own `.env`. +* A retained old pair is a benign cost (nothing hits it on read since + `key_fp` lookup is O(1)). +* Explicit human step matches the "add a new pair → restart" symmetry. + +## Deployment flow + +### First-time upgrade to v1 + +Zero admin work required. On upgrade: + +* v1 code deploys. +* New blobs are written in v1 format at the existing `.blob` object + key. +* Existing legacy blobs stay readable via the magic-byte dispatch — the + legacy read path is preserved verbatim. +* Admin can optionally trigger a `storage_rotate` run to consolidate every + legacy blob into v1 format. Not required — legacy blobs migrate + opportunistically via COW overwrites and stay readable indefinitely + otherwise. + +### Rotating an encryption key + +The user-facing recipe (goes verbatim into `docs/guide/backend-storage.md`): + +``` +1. Generate a new key: + openssl rand -base64 32 + +2. Append it to the entry's key list. Order matters — the NEW key goes LAST: + OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm:,aes-256-gcm: + +3. Restart OxiCloud. New uploads are now encrypted with the new key; existing + blobs still decrypt with the old one. + +4. In the admin panel, click "Rotate encryption key" on the entry. This + dispatches a background job that re-encrypts every existing blob under + the new key AND upgrades any remaining legacy-format blobs to v1. All + operations keep working during rotation. + +5. Wait for the job to complete (progress shows in the top banner and in + the Jobs admin page). + +6. Remove the OLD key from the list: + OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY=aes-256-gcm: + +7. Restart OxiCloud. Rotation is complete. +``` + +### Encrypting a previously-plaintext deployment + +``` +1. Generate a new key (as above). +2. Add it AFTER `none`: + OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=none:,aes-256-gcm: +3. Restart. New uploads are encrypted; existing plaintext blobs stay readable. +4. Run `storage_rotate` to encrypt existing blobs in place. +5. Remove `none:` from the list; restart. +``` + +### Decrypting an encrypted deployment + +``` +1. Add `none:` AFTER the current encryption key: + OXICLOUD_STORAGE_local_main_ENCRYPTION_KEY=aes-256-gcm:,none: +2. Restart. New uploads are plaintext; existing encrypted blobs stay readable. +3. Run `storage_rotate` to decrypt existing blobs in place. +4. Remove the key pair, keep `none:` only (or drop `_ENCRYPTION_KEY` entirely); + restart. +``` + +## Testing strategy + +**Most of this ships as Rust tests, not Hurl.** The rotation surface is +byte-level, cryptographic, and stateful across restart boundaries — none of +which Hurl can meaningfully assert. Push almost everything down into +`#[cfg(test)]` blocks alongside the code, following the codebase's existing +inline-tests convention (see AGENTS.md — "tests are primarily `#[cfg(test)]` +modules within source files"). + +**What lives in Rust (unit + integration):** + +* **Pair-list parser.** Unit tests in `config.rs` on `_ENCRYPTION_KEY` parsing: + 1-pair, 2-pair, cipher-optional shape, `none:` alone, `none:,aes:K`, + `aes:K,none:`, whitespace tolerance, duplicate rejection, multiple-`none` + rejection, empty-list rejection, retired-`_ENCRYPTION_CIPHER` rejection, + base64 length validation. Table-driven; fast. +* **v1 header round-trip.** Unit tests in + `encrypted_blob_backend.rs::tests`: encrypt payload → verify header bytes + (magic, version, key_fp) → decrypt → assert plaintext identity. One test per + format flavour (encrypted, plaintext-with-`none`-head). Verify `` = + BLAKE3(plaintext) with no offset for plaintext-v1 (the "recovery tool" + invariant). +* **Legacy read-path preservation.** Fixture: a hand-crafted raw AES-GCM + ciphertext (`nonce | ct | tag`) produced by the pre-v1 code path. Assert + that magic-byte dispatch falls through to the legacy branch and decrypts + correctly. Fixture stays in-tree as `tests/fixtures/legacy_blob.bin` so + regressions on the legacy path can never sneak through. +* **`OXCPT` collision on random data.** Property test: generate random N-byte + payloads, feed through legacy read path. Assert the ~2⁻⁴⁰ magic-collision + case fails with `UnsupportedBlobVersion` or `NoKeyForBlob` — hard error, + never silent misread. Keeps the "collisions can't silently corrupt" claim + in the plan honest. +* **Rotation decision tree.** Unit tests on the per-blob `decide()` helper of + `storage_rotate_service.rs`. All six cases from *The rotation job* section + as separate tests with clear names (`legacy_upgrades_to_v1`, + `v1_encrypted_under_head_skips`, + `v1_encrypted_under_older_pair_rewrites`, + `v1_plaintext_with_none_head_skips`, + `v1_plaintext_encrypts_when_head_is_cipher`, + `v1_encrypted_decrypts_when_head_is_none`). +* **Recoverable-job round-trip.** Integration test in + `storage_rotate_service::tests` using the existing recoverable-run harness: + seed N blobs (mix of legacy + v1-under-old-key), trigger rotation, assert + every blob ends v1-with-head, `format_generation` (if we add it later) or + the consistency-scan count reports zero legacy remaining, findings=0. +* **Crash recovery.** Same harness: interrupt mid-run at cursor position K, + restart, assert resume from K and eventual completion with correct final + state. Same discipline `storage_migration` already uses. +* **Concurrency safety.** Test that a `put_blob` call during a rotation + targeting the same hash produces exactly one v1 blob at end-state (either + the rotate's or the concurrent write's — both are head-format so the final + state is indistinguishable). Verifies the "per-blob idempotent" claim. +* **Config-restart semantics.** Table-driven test on the pair-list state + machine: start with pair-list [K1], add K2 → [K1,K2], run rotation, + drop K1 → [K2], assert every blob still decrypts. Exercises the "safe to + remove old key" transition end-to-end without going through the admin UI. + +**What Hurl covers (thin — the API surface, not the mechanics):** + +* Trigger endpoint auth: `POST /api/admin/storage/entries/{name}/rotate` is + admin-only (403 for non-admins, 401 for unauthenticated). +* Trigger endpoint preconditions: 400 or similar when pairs < 2 AND no + legacy blobs exist AND pair-list hasn't changed. +* Trigger endpoint concurrency: second POST while a run is Active returns + 409. +* Progress header: `X-Server-Status` includes a `rotation` payload during a + running job and drops it on completion. + +Hurl does NOT try to: +* Inspect on-disk bytes. +* Restart the server with a new pair-list. +* Seed a legacy blob directly. +* Verify decryption after old-key removal. + +Those all belong in Rust tests where we can hold the pool, the backend, and +the config in the same test's memory. + +**Test data.** Legacy-blob fixtures are byte-frozen (`tests/fixtures/*.bin`) +and committed. Don't regenerate them from live code — the whole point is that +they were produced by pre-v1 code and won't ever again be. Regeneration +scripts (kept outside `tests/`) are OK for one-time refreshes if the format +changes. + +## Slices + +Same discipline as `storage-multi-entry.md`. Each slice is a mergeable +increment; no slice depends on a later one. + +### Slice K1 — Pair-list config parsing + +**Scope.** Config layer only. No behaviour change on the write / read path +yet — the parser produces `Vec` and the existing code keeps calling +`pairs.last()`. + +* `NamedStorageEntry.encryption` becomes `Option>` where + `KeyPair = { cipher: CipherKind, key_material: Option<[u8; 32]> }`. + `CipherKind` = `{ AesGcm256, None }`; `None` carries no key. +* Parser: split on `,`, per-pair split on `:` (1 or 2 parts), decode base64 + when a real cipher, reject empty list, reject duplicates, reject multiple + `none`. +* Retire `OXICLOUD_STORAGE__ENCRYPTION_CIPHER` — parser errors on it with + guidance to move the cipher into the pair. +* Boot logs each pair's fingerprint (`sha256(key)[..12]` for real ciphers, + `—` for `none`) and marks the head. Warn line if head fp differs from any + non-head fp (rotation window signal). +* Docs: `example.env`, `docs/config/env.md`, `docs/guide/backend-storage.md` + updated with the pair syntax and the three recipes (rotate / encrypt / + decrypt). + +**Exit criteria.** Boot with a 1-pair config behaves identically to today. +Boot with a 2-pair config succeeds and logs both fingerprints. Boot with +`_ENCRYPTION_CIPHER` alongside `_ENCRYPTION_KEY` fails with a migration hint. + +### Slice K2 — v1 read/write paths in `EncryptedBlobBackend` + +**Scope.** The encrypted-backend decorator learns the v1 header format. Reads +dispatch on magic bytes; writes always emit v1 headers. Legacy read path +preserved verbatim. `blobs_consistency` gains a magic-byte branch to track +the legacy-blob count. + +* `EncryptedBlobBackend` refactored around `BlobFormat::V1` writer + reader. + Legacy path preserved but read-only from new code. +* Read dispatches on magic; write always emits v1 header at `.blob`. +* `blobs_consistency`: during its normal walk, per-blob magic-byte check; + legacy count recorded on the run's `stats` JSON. No schema hit — reuses + the existing `stats` bag on `jobs.recoverable_runs`. +* `storage_migrate`: no changes needed. It copies raw bytes between backends; + format is preserved on the target automatically because the object bytes + are opaque to it. + +**Exit criteria.** A brand-new deployment writes only v1 blobs. An upgraded +deployment reads existing legacy blobs and writes new v1 blobs at the same +object-key. `blobs_consistency` reports a legacy-blob count in its run stats. + +### Slice K3 — The `storage_rotate` recoverable job + +**Scope.** New handler, admin-triggered, iterates blobs, per-blob decision +tree (legacy → v1 upgrade, v1 with old key → v1 with head key, plaintext ↔ +encrypted where applicable), records findings. + +* New file `src/infrastructure/services/storage_rotate_service.rs`. +* Registered in `JobRegistry` as `storage_rotate`. Runs on the same + recoverable-runs engine (crash recovery, cursor persistence, pause/resume). +* Trigger endpoint: `POST /api/admin/storage/entries/{name}/rotate`. + Requires admin. Refuses if no work would happen (all blobs already at + head format + head key). Refuses if a `storage_rotate` or + `storage_migration` run is already Active for any entry. +* Per-blob decision tree per *The rotation job* section above. In-place + atomic replace at the same `.blob` object key. +* No readonly mode engaged. `X-Server-Status` header payload gains a + `rotation` field alongside `migration`. + +**Exit criteria.** On a filled sandbox: (1) legacy-only entry rotates to +v1; (2) encrypted entry with 2 pairs rotates so head-pair-only remains +decryptable; (3) plaintext entry with `none,cipher` head rotates to +encrypted-v1; (4) encrypted entry with `cipher,none` head rotates to +plaintext-v1. Each round-trip verified by dropping the retired pair from +config and successfully reading every blob. + +### Slice K4 — Admin panel action + banner + docs + +**Scope.** UI wiring, banner variant, guide docs, legacy-blob counter. + +* Entry card: **Rotate encryption key** button + tooltip states as above. +* Legacy-blob counter chip on each entry card. Sources the count from the + most recent `blobs_consistency` run's stats (already surfaced in the Jobs + admin page). Refresh button triggers a fresh scan. +* `ReadOnlyBanner.svelte` gains a third variant `variant="rotating"` — same + shape, milder tone (info not warning), copy makes it clear that operations + continue. +* Server-status header payload gains `rotation?: {entry, migrated, total, + percent}` alongside the existing `migration?:` field. `readonly` stays + false during rotation. +* `docs/guide/backend-storage.md`: new **Rotating an encryption key**, + **Encrypting a plaintext deployment**, and **Decrypting an encrypted + deployment** sections with the recipes above. +* `docs/config/env.md`: pair syntax section under `_ENCRYPTION_KEY`. +* Admin panel *"Rotation complete — safe to remove the old key"* hint gated + on findings=0 AND legacy count=0. + +**Exit criteria.** Round-trip on a real deployment (Ed's OVH S3): add pair +→ restart → rotate → verify blobs (including legacy) → remove old pair → +restart → blobs still readable. + +## Forward compatibility (v2 and beyond) + +The `` field is the single load-bearing knob for future format +changes. Reserved slots: + +* `0x0001` — this plan. AES-256-GCM, server-side keys, individual blob + integrity via AEAD tag + content-hash. +* `0x0002…0x00FF` — reserved for server-side format bumps: new AEAD, + chain-authenticated CDC chunks (either *manifest HMAC* — no header changes, + `chunk_manifests` gains a server-computed HMAC over the ordered chunk-hash + list — or *Merkle root in header* — v2 grows to carry `manifest_root` + + `chunk_index`, letting any streamed chunk be verified in isolation). Both + defend against manifest reorder / truncation / injection attacks that + content-addressability alone doesn't cover. +* `0x0100…0x01FF` — reserved for client-side encryption (E2E) variants. + `` becomes the client-key fingerprint; the server can no longer + decrypt and passes ciphertext through the streaming path unchanged. + +v1 and v2 coexist in the same storage indefinitely — the magic-byte read +dispatch handles arbitrary versions at position 5-6. Migration between +generations reuses `storage_rotate`'s pattern: rewrite each blob with the +new-generation writer, in-place at the same object key. + +## Non-goals + +* **Asymmetric / KMS-backed keys.** Out of scope. Symmetric AES-256-GCM only. + A KMS-backed variant is a separate slice built on top of this one. +* **Chunk-chain integrity / manifest tampering defense.** Recognised as a + real gap (chunk reorder / truncate / inject via DB write). Deferred to the + v2 header slice — see *Forward compatibility*. +* **Client-side encryption (E2E).** Same — v2 slot reserved, separate slice. +* **Automatic old-key removal.** Explicit human step, see above. +* **Rotation of a key on the "in-transit" side** (client → server TLS). + Handled by the reverse proxy; unrelated. +* **Per-blob key derivation** (KDF from a master key + blob hash). A + legitimate hardening path but orthogonal to the rotation story; folds in as + a future slice under this same pair-list config. +* **Rollback safety across format generations.** If v1 code is rolled back + to pre-v1 after new v1 blobs have been written, old code will 500 on those + blobs (AES-GCM tag fails on OXCPT-prefixed bytes). Not addressed by this + plan; the safety net is restore-from-backup. Deploying an + incompatibility-breaking format change is a pre-planned event, not a hot + rollback scenario. + +## Open questions + +* **Should we throttle the rotate job?** Same question `storage_migration` had. + Answer: not in v1. If throughput bites, add a `_ROTATE_MAX_MB_PER_SEC` on + the entry later. +* **Should rotation be idempotent under repeat trigger?** Yes. Running it a + second time with the same config is a no-op walk (every blob is already at + head format + head key). +* **Should we ship a `format_generation` column on `storage.blobs` as a + query-optimizer?** No in v1. Consistency-scan stats cover the "how many + legacy blobs remain" question at admin cadence, and per-read magic-byte + dispatch is a 5-byte compare. If a real hot-path or admin-panel latency + need emerges, the column is a small back-fill migration to add later. +* **Should the legacy-blob counter block key removal at code level?** No — + keeping the trust model consistent with "admin edits .env, we don't fight + them" is the current default. The counter + hint is enough. If real + incidents happen we can escalate to a hard guard later.