# Plan — Multi-entry storage config + name-selected active backend ## Context Today OxiCloud has a single storage backend, configured via a flat set of env vars (`OXICLOUD_STORAGE_BACKEND`, `OXICLOUD_S3_*`, `OXICLOUD_STORAGE_ENCRYPTION_KEY`, …). The admin panel has a second, parallel storage-config surface backed by `admin_settings.storage.*` rows, used by the migration tool to pick a target. Priority is `env > DB > defaults`, so the DB config effectively acts as a staging area for "what the migration should copy INTO" but never wins at boot. Two chronic problems fall out: 1. **Split-brain config.** Admin edits DB via the panel; app boot ignores DB. Migration completes; live backend hasn't moved. Admin has to remember to copy env vars into `.env` and restart. Two sources of truth for the same setting. Cutover is a manual multi-step flow; users routinely get it wrong. 2. **Migration data-loss window on concurrent writes.** The copy walks `storage.blobs` in hash order. A blob whose hash is lex-lower than the current cursor, written to source AFTER migration passed it, is never copied to target. `passed=true, findings=0` completion does NOT guarantee target has every blob. Silent. 3. **Migration target selection is fragile.** DTO passes the whole S3 config at trigger time; secrets sit plaintext in `admin_settings`. Any future pluggable-storage story compounds this (Azure, GCS, WebDAV-as-source, …). This plan replaces the split-brain model with a single-source-of-truth architecture: - `.env` declares **N named storage entries** (immutable per-deploy). - `admin_settings.storage.active_backend_name` holds ONE row — which named entry the app currently runs on. That's the whole runtime config. - Migration is the atomic transition from one active entry to another. Server is put in read-only mode for the copy window; on completion, the active pointer flips; a restart cuts over. Named entries also solve two adjacent problems Ed flagged during design: - **Per-entry encryption keys** enable `local (raw) → s3 (encrypted K1)` moves AND `s3 (K1) → s3-new-bucket (K2)` key-rotation moves. - **`?storage=` on `blobs_consistency`/`backend_consistency`** lets operators audit any registered entry (target verification, pre-decommission check, etc.), replacing the sample-based `verify_migration` endpoint with a full-walk audit. ## Design decisions ### Named entries in `.env` — explicit allowlist ``` OXICLOUD_STORAGE_ENTRIES=local_main,s3_prod OXICLOUD_STORAGE_local_main_BACKEND=local OXICLOUD_STORAGE_local_main_ROOT_DIR=/data OXICLOUD_STORAGE_s3_prod_BACKEND=s3 OXICLOUD_STORAGE_s3_prod_S3_BUCKET=my-bucket OXICLOUD_STORAGE_s3_prod_S3_ENDPOINT_URL=https://s3.example.com OXICLOUD_STORAGE_s3_prod_S3_REGION=us-east-1 OXICLOUD_STORAGE_s3_prod_S3_ACCESS_KEY=... OXICLOUD_STORAGE_s3_prod_S3_SECRET_KEY=... OXICLOUD_STORAGE_s3_prod_S3_FORCE_PATH_STYLE=true OXICLOUD_STORAGE_s3_prod_ENCRYPTION_KEY= ``` - **Explicit `_ENTRIES` allowlist** — order-independent, admin-authored names. Env-pattern-matching was considered and rejected: too fragile (someone typos `OXICLOUD_STORAGE_s3_prud_BUCKET`, gets a silently-registered ghost entry). Explicit list forces a declaration. - **Names** are admin-chosen strings matching `[a-z0-9_-]{1,32}`, unique within the list. Parsed at boot; unparseable → fail-fast with the offending name in the error. - **Order does not matter** — DB says which is active. Reordering entries in `.env` never changes runtime behaviour. ### Legacy flat-var interaction — three states, one is fail-fast Multi-entry lives alongside the pre-existing single-backend flat vars (`OXICLOUD_STORAGE_BACKEND`, `OXICLOUD_S3_*`, `OXICLOUD_AZURE_*`, `OXICLOUD_STORAGE_ENCRYPTION_KEY`, `_ENABLED`). The parser resolves the interaction as follows: | `_ENTRIES` | Legacy storage-backend vars present | Behaviour | |---|---|---| | Unset / empty | Absent | `storage_entries = []`. Boot uses framework defaults (Local at `storage/`). | | Unset / empty | Present | **Synthesize** a single entry named `default` from the legacy vars. Preserves upgrade path — existing deployments keep working without touching `.env`. | | Set (e.g. `foo,bar`) | Absent | Parse each named entry from `_STORAGE__*` vars. Fail-fast if any declared entry is missing its required per-name fields. | | Set (e.g. `foo,bar`) | **Present** | **FAIL FAST**. Boot aborts with an error listing every legacy var found. Admin must remove them or migrate them into per-entry `_STORAGE__*` form. | **Why fail-fast on the "both set" case:** without it, admin state is ambiguous — someone edits `OXICLOUD_S3_BUCKET` expecting it to matter, but it's silently ignored because `_ENTRIES` won. The subtle-bug cost is much higher than the one-time cleanup cost. Refusing to boot forces the conversion once, with a clear message naming the exact vars to remove. **Set of "legacy storage-backend vars"** counted for the conflict check: `OXICLOUD_STORAGE_BACKEND`, all seven `OXICLOUD_S3_*`, all five `OXICLOUD_AZURE_*`, `OXICLOUD_STORAGE_ENCRYPTION_ENABLED`, and `OXICLOUD_STORAGE_ENCRYPTION_KEY`. `OXICLOUD_STORAGE_PATH` is NOT in this set — it drives multiple non-backend things (chunk dir default, etc.) and remains a valid ambient path; per-entry `_ROOT_DIR` falls back to it for Local entries when unset. ### One DB row: `active_backend_name` Single setting in `admin_settings`: ``` storage.active_backend_name = "local_main" ``` - **Boot logic**: 1. Parse `AppConfig.storage_entries` from env. 2. Read `active_backend_name` from `admin_settings`. 3. If unset (fresh install): use the FIRST name in `_ENTRIES`. Log which one, don't fail. 4. Look up the entry by name. Build `blob_backend` from it. 5. If the name doesn't exist in current env (deploy drift — someone removed an entry): **fail-fast at boot** with a clear error naming the missing entry AND listing the available ones. Admin fixes env or overrides the setting via a fallback CLI (see §Fallback). - Everything the admin panel currently writes about storage (`s3.bucket`, `s3.access_key`, …) is **removed** from DB. `save_storage_settings` is deleted along with those rows. ### Encryption is per-entry `OXICLOUD_STORAGE__ENCRYPTION_KEY` (base64 of exactly 32 bytes) on an entry → that entry's backend is wrapped in `EncryptedBlobBackend` at build time. Absent → raw backend. - **Presence-implies-enabled.** No separate `_ENCRYPTION_ENABLED` toggle — one env var per entry is enough. - **Fail-fast on invalid key**: bad base64, wrong decoded length → boot aborts with the entry name in the error message. A bad key is a real deployment error, must be caught at boot, never silently disabled. - **Legacy `OXICLOUD_STORAGE_ENCRYPTION_KEY`** remains honoured for the synthesized `default` entry when `_ENTRIES` is empty (upgrade path). Cross-entry encryption combinations work out of the box because `EncryptedBlobBackend` is a decorator and `copy_blob` in the migration handler always spools plaintext to a tmp file: | Source | Target | Migration behaviour | |---|---|---| | Raw local | Encrypted S3 (K1) | Read plaintext → write encrypts with K1 | | Encrypted S3 (K1) | Raw local | Read decrypts with K1 → write plaintext | | Encrypted S3 (K1) | Encrypted S3 (K2) on different bucket | Read decrypts K1 → write encrypts K2 (rotation via new bucket) | | Encrypted S3 (K1) | Encrypted S3 (K2) on SAME bucket | **REFUSED** — see below | **In-place key rotation is refused.** Same physical bucket + different encryption key would have the migration overwrite `.blob` with K2 ciphertext while the LIVE backend is still K1-configured → readers get K1-decrypt-of-K2-bytes → 500. Silent data-loss window. The `is_source_target_identical` guard (see §Migration flow) catches this because `storage_identity` deliberately excludes the encryption key. The refusal message spells the case out and recommends the two-step workaround (rotate via a temp bucket). **Proper in-place rotation (deferred future slice)** The safe implementation is to namespace object keys by encryption generation — `.k2.blob` or `k2/.blob` (backend-specific convention). Then: - Both key generations coexist in the same bucket during rotation. - Reads continue against K1 (LIVE backend) — old keys untouched. - Target writes go to K2 keys. - Cutover restarts with K2 as live; K1 objects can be reaped async by a follow-up sweep. - The `is_source_target_identical` guard's `storage_identity` string changes to INCLUDE the encryption generation — same-bucket different-generation then correctly registers as a legitimate migration, no longer refused. Schema changes required: - `EncryptedBlobBackend` gains a `generation: u32` field. - Object keys become `.k.blob` (or backend-specific equivalent — S3 key naming, Azure blob naming). - Read path tries current-gen first, falls back to prior-gen for the rotation window (bounded by cutover-completion + async-reap duration). Not built now — real key-rotation demand is rare enough that the two-step workaround via temp bucket is acceptable. File this as a follow-up slice AFTER multi-entry lands; the named-entry infrastructure is a prerequisite (generation-versioned keys only make sense when there's an entry model to hold "which generation is active" as configuration). ### Read-only mode reuses the existing AuthZ short-circuit `DrivePolicies.read_only` already gates writes at `PgAclEngine::check_inner` — short-circuits `Create|Update|Delete|Share` Permission checks on any resource in a read-only drive. We extend that clause with a global check: ```rust // Inside PgAclEngine::check_inner, before per-drive read_only check: if permission.is_write() && self.migration_readonly.load(Ordering::Relaxed) { return AclDecision::Denied("server in migration read-only mode"); } ``` - The flag is a `AtomicBool` on `AppState`, backed by `admin_settings.storage.migration_readonly` so it **survives restart** (server crashes mid-migration → boots read-only → admin retriggers → still safe). - **Admin operations bypass** as they already do — admin can still exit read-only, cancel migration, restart the server. - **Reads are unaffected**. Users can still browse and download during migration. - **Boot-time clearing**: if boot detects `migration_readonly=true` AND no in-flight `storage_migration` row (no `Running`/`Paused`) AND `active_backend_name` matches the entry the app booted onto → assume successful cutover completed on prior boot, clear the flag. Otherwise leave it set; admin knows they still need to finish something. ### Migration flow — atomic + restart-proof ``` 1. Admin picks target entry from a dropdown → clicks "Migrate to s3_prod" 2. Backend: - Verify target_name exists in AppConfig.storage_entries - Verify target_name != active_backend_name (no-op guard; existing `is_source_target_identical` refactored to compare NAMES not identity strings — but the identity check still runs as a second-line defence against the encryption-in-place case) - Write admin_settings.storage.migration_readonly = true - Trigger `storage_migration` recoverable job with params = { source_name: "local_main", target_name: "s3_prod" } 3. Migration runs — target resolved fresh each batch by NAME lookup, so the run's params carries only the name. No secrets in params. If the process restarts mid-run, the resume path re-resolves the entry from env by the persisted name. Env is the source of truth for credentials. 4. On Completed: - Write admin_settings.storage.active_backend_name = "s3_prod" - Log "Cutover complete — restart the server to switch to the new backend" - LEAVE read-only mode on. The app is still running with local_main as the live backend; if we lifted read-only now, writes would go to local_main even though the DB pointer says s3_prod. Forces the operator restart, which resolves the ambiguity. 5. Admin restarts the server: - Boot reads active_backend_name = "s3_prod" → live backend is now S3 - Boot's read-only-clear rule fires (no in-flight migration + active matches booted entry) → migration_readonly cleared - Server writable, on the new backend. Cutover complete. ``` **Handling in-progress restart (server dies while migration is running)**: - Boot sweep flips `Running` → `Paused` on the migration row (existing Part 2 machinery). - `active_backend_name` unchanged. Boots on old backend. - `migration_readonly` stays true (in-flight migration → boot-time clear rule DOESN'T fire). - Admin retriggers migration → run_or_resume reads params.target_name → resumes from cursor with the same target. **No secret ever reaches the DB.** Params holds only the entry names. Credentials stay in env; migration resolves them at each batch by name. ### Concurrent-write safety — read-only for the copy window The full-quiesce trade-off: users can browse/download during migration but cannot upload, rename, delete, or share. For a multi-hour migration this is noticeable; the alternative (dual-write decorator) is genuinely a week of work (runtime backend swapping, failure-mode reconciliation, `MigrationBlobBackend` rebuild) and only justified if migration is a routine op. Read-only is the honest v1 answer — pick a low-traffic window, run the migration, restart. Read-only is engaged at trigger time and cleared at boot after cutover. Nothing more elaborate. Dual-write is filed as a future upgrade if operator demand appears. ### `?storage=` for consistency audits Once entries are named, `?storage=` becomes the generic "probe any registered entry" knob on the two tenants that touch a backend: - `blobs_consistency?storage=` — DB → backend probe. - `backend_consistency?storage=` — backend → DB probe. Unspecified → falls through to the active backend (today's behaviour, preserved). **Use cases this unlocks**: - Pre-cutover verification: `blobs_consistency?storage=s3_prod` after migration completes — full walk (not a sample) proving target has every blob before the .env flip + restart. **Retires `verify_migration`** — sample-check becomes redundant when full audit is one click away. - Pre-decommission verification: `?storage=local_main` after cutover — check the old backend still has every blob DB expects before `rm -rf` the local `.blobs/`. - Ad-hoc audit of any registered entry, backup-restore verification, etc. **Plumbing**: - `JobRunArgs` gains `storage: Option`. - `TriggerJobQuery` on the admin trigger endpoint parses `?storage=`. - `BlobsConsistencyCheck` and `BackendConsistencyCheck` constructors take `Arc` (or a smaller `EntryResolver` port); at run start they use `args.storage` to pick the backend, falling back to the injected active-backend Arc. - **Unknown entry**: fail-fast at HTTP layer (`400` before the run ever starts) with `known: [local_main, s3_prod]` in the response. Cheaper than burning a run row. - **Combined with `?deep=true`**: legit — "full byte-level integrity audit of a named entry, not the live one". Audit log records both. - Params on the run row records `probed_storage: ` (or "active") for post-hoc diagnosis. ### Fallback for the "boot fails on missing entry" case If admin renames an entry in `.env` (or removes one that DB still points at), boot fails fast with a clear error. Operator has two ways out: 1. Fix `.env` — add the missing entry back OR update `_ENTRIES` to include an alternative that DOES exist, plus flip `active_backend_name` before restart. 2. **CLI repair flag on the `oxicloud` binary itself**: ``` oxicloud --select-storage ``` Behaviour: parse `.env`, verify `` exists in `_ENTRIES` (fail-fast with the available names listed if not), connect to DB, UPDATE `admin_settings.storage.active_backend_name`, print confirmation, exit 0. Does NOT continue to boot the server — one-shot repair; admin starts the server normally afterwards. The bare-flag on the shipped binary is chosen over a separate `just` recipe or auxiliary bin because: - **Docker-friendly**: `docker exec oxicloud oxicloud --select-storage foo` — no need to install extra tooling in the container. - **Systemd-friendly**: can be run as a `ExecStartPre=` one-shot before the main service unit. - **No dep on `just`** being installed (dev-machine tool, not typical prod). - **Same binary, same env-parse code path**: the repair uses THE SAME `.env` parser the server does, so "verified present" means the server will succeed on next boot too. No parser drift possible. The boot-time error message points at this flag explicitly, with the exact command line filled in. ### Interaction with existing surfaces - **`/admin/storage` tab** rewritten: - Read-only listing of entries (name, backend type, encryption on/off, is active). - "Migrate to X" dropdown (choose target entry). - Migration progress + verify/cancel (existing). - Read-only banner when `migration_readonly` is on. - The Save form, S3-field editors, .env cutover hint — all deleted (no settings edit here anymore). - **`test_storage_connection`** loses the S3-fields DTO; becomes a round-trip probe against a named entry: `POST .../test?storage=`. - **`verify_migration`** deleted; the "Verify integrity" button on the storage tab is rewired to trigger `blobs_consistency?storage=` against the migration target (or dropped in favour of the standard admin/jobs Run button — TBD in slice 6). ## Slice breakdown Single PR is too big; splitting into a coherent sequence. Slices 1-2 are foundational; the rest layer on top independently within reason. | # | Slice | Depends on | Rough size | |---|---|---|---| | 1 | Config parser: `OXICLOUD_STORAGE_ENTRIES` + per-entry vars + `_ENCRYPTION_KEY` → `Vec` on `AppConfig`. Legacy synthesis for empty `_ENTRIES` (default entry from legacy flat vars). | — | 1 day | | 2 | Boot: read `active_backend_name` from `admin_settings`; look up entry; build `blob_backend` via a shared `build_entry_backend(&NamedStorageEntry)` factory (wraps encryption decorator when key present). Fail-fast on missing entry with actionable message. | 1 | ~half day | | 3 | Migration handler rewrite: params carry `target_name` only. Handler resolves target by name from `storage_settings.build_entry_backend(name)`. `is_source_target_identical` guard refactored to compare NAMES first; keeps physical-identity check as second-line refusal (with encryption-differs-specific message). Retire the migration DTO S3-field body. | 1, 2 | ~half day | | 4 | Global `migration_readonly` flag on `AppState`, backed by `admin_settings.storage.migration_readonly`. One clause added to `PgAclEngine::check_inner`. Boot-time clear rule (no in-flight + active matches booted → clear). | 2 | ~half day | | 5 | Cutover state machine: on migration `Completed`, write `active_backend_name = target_name`, keep read-only on. Boot on new backend after operator restart. | 4 | ~half day | | 6 | Admin storage tab rewrite: list entries, show active, migrate dropdown, read-only banner. Delete Save form + S3 field editors + .env cutover hint. | 1, 3, 4 | 1 day | | 7 | `?storage=` on `blobs_consistency` + `backend_consistency`. `JobRunArgs.storage` plumbing, `TriggerJobQuery.storage`, entry-resolver at run start, params records probed name. Retire `verify_migration` + its DTO + its route + its handler. | 1, 3 | 1 day | | 8 | `oxicloud --select-storage ` bare-flag repair command on the main binary. Parses `.env`, verifies entry exists, UPDATEs DB, exits. Boot-time missing-entry error message points at it. See §Fallback. | 2 | ~quarter day | **Total: ~5-6 days end to end.** Slices 6 and 7 can proceed in parallel with each other once 1-5 land. Slice 8 is an ops nicety, could ship whenever. ## Verification Per slice, plus these end-to-end scenarios in Hurl: 1. **Fresh install, no `_ENTRIES`**: boot uses synthesized `default` entry from legacy vars, `active_backend_name` unset. `GET /admin/settings/storage` returns one entry, active. No cutover UI. 2. **Two entries, no active set**: boot picks first in `_ENTRIES`, logs it, proceeds. Admin panel shows both entries with the picked one marked active. 3. **Change active without migration** (rarely useful but must be safe): `POST /admin/settings/storage/active-name` (or however the pointer is exposed) updates the row; next restart boots on the new entry. Existing blobs on the new entry NOT verified — operator's problem, but `blobs_consistency` catches the drift on next run. 4. **Migration happy path**: two entries, entry A active, migrate to B. Read-only comes on. Copy runs. `active_backend_name` flips to B on completion. Read-only stays on until restart. After restart, live is B, read-only cleared. 5. **Restart mid-migration**: kill server after checkpoint N. Boot: row Paused, `active` still A, `migration_readonly` still true. Admin retriggers → resumes from checkpoint N against target B (resolved by name from params). Complete → pointer flips → restart → live is B. 6. **In-place encryption rotation refused**: two entries, same S3 bucket, different encryption keys. Trigger migration → refuses with the specific error message pointing at the encryption case and the two-step workaround. 7. **`?storage=` on blobs_consistency**: run against `s3_prod` before cutover. Full walk, `probed_storage` in run row. Then cutover, then rerun against `local_main` — verifies old backend still has everything. 8. **Unknown storage name**: `POST /admin/jobs/blobs_consistency/trigger?storage=nope` → 400 with known-names list. No run row created. 9. **Missing entry at boot**: `active_backend_name = "gone"` but `_ENTRIES` doesn't include it → boot aborts with the specific message pointing at `oxicloud --select-storage ` (with the available names filled in). Re-run the binary with `--select-storage local_main` → verifies + updates DB + exits 0. Restart the server → boots cleanly on `local_main`. 10. **Encryption key invalid**: `OXICLOUD_STORAGE__ENCRYPTION_KEY=badbase64` → boot aborts with entry name + reason (not valid base64 / wrong length). 11. **Legacy vars alongside `_ENTRIES`**: set `_ENTRIES=foo` AND leave a stale `OXICLOUD_S3_BUCKET=...` in `.env`. Boot aborts with the full list of legacy vars detected, tells the admin to remove them (or move them into `OXICLOUD_STORAGE__S3_BUCKET` form). Removing the legacy var → next boot succeeds. 12. **Legacy synthesis path**: `_ENTRIES` unset, `OXICLOUD_STORAGE_BACKEND=s3` + `OXICLOUD_S3_BUCKET=...` set. Boot synthesizes one entry named `default` from those vars. `storage_entries.len() == 1`, name is `default`, backend is S3, config carries the flat-var values. ## Out of scope - **Dual-write decorator / zero-downtime migration**: read-only is v1; dual-write is filed under "if operator demand appears". Would rebuild `MigrationBlobBackend` (deleted in the recoverable-migration PR — retained in git for the same reason). - **In-place encryption key rotation** (same-bucket, different key): refused by the identity guard; workaround is two-step via temp bucket. Proper fix (per-generation object naming) is documented inline in the Encryption section above; deferred until real demand appears. - **Runtime backend hot-swap without restart**: not attempted. Requires `Arc>>` indirection at every call site plus per-request coordination; huge blast radius. Read-only + restart is the honest answer. - **Per-user or per-drive storage backends**: everything in this plan is server-scoped. If per-drive storage becomes a real need, the named-entry registry is the right substrate but `blob_backend` on AppState becomes a `EntryResolver` and every call site changes. Not now. - **DB storage config UI**: retired. Admin panel is a controller (list + test + migrate + audit), never a persister. If persisted per-entry knobs become a need (retention days per entry, quota per entry, …), those live in DB rows keyed by entry name — a small extension, not a return to the old model. - **Secret encryption in DB**: `admin_settings.storage.*` secret rows are deleted along with `save_storage_settings`. If the migration params approach ever grows to store secrets (e.g., a future "supply the target creds inline for one-off migrations"), those get encrypted at rest with a KMS-provided key. Not needed for this plan — params holds only names. ## Related memory notes - `feedback_no_abbreviated_env_vars` — full-word env var names (`OXICLOUD_STORAGE_local_main_S3_ENDPOINT_URL`, not `OXICLOUD_STORAGE_local_main_S3_EP`). - `feedback_config_file_overrides_shell` — `dotenvy` behaviour on explicit config path. Matters when `_ENTRIES` values are loaded from `--config` vs shell. - `project_admin_middleware_layer` — admin ops bypass read-only via existing middleware; the AuthZ short-circuit added in slice 4 doesn't need any new admin carve-out. - `bug_drive_rename_editor_can_do_it` — reminder that `PgAclEngine::check_inner` is where write-permission short-circuits live; the new global read-only clause lands next to the per-drive one. - `docs/plan/job-registry.md` Part 2 — recoverable-run engine that `storage_migration` runs on; `params` field, resume semantics, boot sweep.