diff --git a/docs/config/env.md b/docs/config/env.md index ec158e84..7ddee4f7 100644 --- a/docs/config/env.md +++ b/docs/config/env.md @@ -18,6 +18,7 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator | `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Direct (non-chunked) uploads stream straight into the blob store and need no spool directory. Placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). | | `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. | | `OXICLOUD_METRICS_LISTEN` | (unset) | Prometheus `/metrics` listener address (e.g. `127.0.0.1:9090`, IPv6 allowed as `[::1]:9090`). **Unset = disabled**: no `/metrics` endpoint is bound and no metrics recorder is installed (zero runtime cost). When set, a separate HTTP listener on this address serves the text-format scrape. **Deliberately NOT merged into the main API** — no auth, CSRF, or DPoP layer in front. Bind to loopback or a private interface unless you intend to expose metrics publicly. Starter counters: `oxicloud_dpop_verify_failed_total{reason}`, `oxicloud_dpop_proof_missing_total`, `oxicloud_dpop_header_missing_on_bound_session_total`, `oxicloud_dpop_replay_detected_total`, `oxicloud_dpop_nonce_challenges_issued_total`. | +| `OXICLOUD_STARTUP_JOBS` | `thumb_derived_import?repair=true,thumb_attached_import?repair=true` | Background jobs dispatched once at boot, comma-separated, each `name` or `name?flag=true` using the same syntax as `POST /api/admin/jobs/{name}/trigger`. Flags: `force`, `deep`, `repair`, `storage`. **The default migrates thumbnails out of the legacy `.thumbnails/` directory and deletes the originals**, so the migration completes without anyone triggering it from the admin panel; each sidecar is read back through the normal stack before it is unlinked, and every deletion is audited. An explicit value **replaces** the default; set it empty (`OXICLOUD_STARTUP_JOBS=`) to disable startup jobs, or to `thumb_derived_import,thumb_attached_import` to import without deleting. **Non-blocking** — readiness never waits on a job; entries run sequentially in the background. **Fail-fast** — an unknown job name or flag panics at boot, because a silently-dropped entry means a migration that never runs. A run interrupted by a restart resumes from its cursor on the next boot, so a long migration finishes across restarts. Safe to leave at the default: the jobs are idempotent, and once drained a run does nothing. See [Thumbnail Migration](./thumbnail-migration.md) for the upgrade runbook. | ## Database diff --git a/docs/config/index.md b/docs/config/index.md index 35d424f2..c47a6cf1 100644 --- a/docs/config/index.md +++ b/docs/config/index.md @@ -7,6 +7,7 @@ OxiCloud is configured entirely via **environment variables** (no config files n - [Deployment & Docker](/config/deployment) — Docker Compose, Kubernetes Helm chart, image details - [Environment Variables](/config/env) — complete reference of all `OXICLOUD_*` variables - [Storage Fine Tuning](/config/storage-fine-tuning) — sizing the upload caps + spool directories; tmpfs vs real disk; NVMe split layouts +- [Thumbnail Migration](/config/thumbnail-migration) — upgrading past `.thumbnails/`: what runs on first boot, taking a snapshot first, verifying afterwards - [Authentication](/config/authentication) — JWT auth, login, refresh, password changes, and auth status - [OIDC / SSO](/config/oidc) — single sign-on with Keycloak, Authentik, Authelia, Google, Azure AD - [WOPI (Office Editing)](/config/wopi) — Collabora Online / OnlyOffice integration diff --git a/docs/config/thumbnail-migration.md b/docs/config/thumbnail-migration.md new file mode 100644 index 00000000..c9c762a7 --- /dev/null +++ b/docs/config/thumbnail-migration.md @@ -0,0 +1,153 @@ +# Thumbnail migration runbook + +Thumbnails used to live as files under `{STORAGE_PATH}/.thumbnails/`. +They now live in the content-addressed blob store, alongside file +content. This page is for operators upgrading across that change. + +**You do not have to do anything.** The migration runs itself, in the +background, on the first boot after the upgrade. The rest of this page +is for operators who want to verify it, take a safety net first, or +understand what it did. + +## What runs, and when + +Two background jobs, dispatched once at startup and daily thereafter: + +| Job | Migrates | Regenerable if lost? | +|---|---|---| +| `thumb_derived_import` | Thumbnails the server rendered from file content | Yes — the next request re-renders | +| `thumb_attached_import` | Previews a client uploaded (`ext-{file_id}.jpg`) | **No** — there is no render path for these | + +Both import each sidecar into blob storage, read it back to confirm the +copy is byte-identical, and only then delete the original. When the +directory is empty it is removed, and `.thumbnails/` stops existing. + +Startup dispatch is non-blocking — the server is ready immediately and +the migration proceeds behind it. A run interrupted by a restart resumes +from where it stopped, so a large installation finishes over several +restarts rather than starting again each time. + +This is controlled by `OXICLOUD_STARTUP_JOBS`, which defaults to: + +``` +OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true +``` + +To **import without deleting** — migrate now, inspect, delete later: + +``` +OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import +``` + +The sidecars then stay on disk. Trigger the deletion when you are ready +from **Admin → Jobs**, using each job's Repair action. + +To disable startup jobs entirely, set the variable to an empty value. + +## Taking a safety net first + +Recommended for any installation where the uploaded previews matter, and +cheap enough to be worth it regardless. Both parts must be captured +together — a database that references blobs a storage snapshot predates +is worse than neither. + +**1. Stop the server.** A snapshot taken while writes are in flight can +catch a blob that exists on disk without its database row, or the +reverse. + +```bash +systemctl stop oxicloud # or: docker compose stop oxicloud +``` + +**2. Snapshot the database.** + +```bash +pg_dump --format=custom --file=oxicloud-preflight.dump "$DATABASE_URL" +``` + +Use `--format=custom`; restoring it needs `pg_restore --disable-triggers`, +because the folder table carries a self-referencing foreign key that a +plain SQL restore cannot order correctly. + +**3. Snapshot the storage directory.** At minimum `.thumbnails/`, which +is what the migration touches: + +```bash +tar -czf oxicloud-thumbnails-preflight.tar.gz -C "$STORAGE_PATH" .thumbnails +``` + +A whole-directory snapshot is better if you have the space — filesystem +or volume snapshots (ZFS, LVM, EBS) are ideal, since they are atomic and +near-instant: + +```bash +zfs snapshot tank/oxicloud@preflight +``` + +**4. Start the server.** The migration begins in the background. + +Keep both snapshots until you have run the verification below and are +satisfied. + +## Verifying the migration + +Two checks, both from **Admin → Jobs** or the API. Run them after the +migration reports no remaining work. + +**1. Every mapping points at a blob that exists.** Run +`satellites_consistency`. It walks both thumbnail tables and reports any +row whose blob or source is gone. A clean run means nothing was lost in +the bookkeeping. + +``` +POST /api/admin/jobs/satellites_consistency/trigger +``` + +**2. Every blob still hashes to what it claims.** Run +`backend_consistency` with `?deep=true`. It reads every blob back from +storage and re-hashes it, which covers the migrated thumbnails along +with everything else. This is a full read of your storage and can take +hours on a large installation — schedule it accordingly. + +``` +POST /api/admin/jobs/backend_consistency/trigger?deep=true +``` + +A clean pass on both means the thumbnails are readable, correctly +referenced, and byte-intact in their new home. At that point the +snapshots can be discarded. + +## Checking it finished + +`.thumbnails/` is gone. That is the whole test: + +```bash +ls -d "$STORAGE_PATH/.thumbnails" # No such file or directory +``` + +If you instead find `.thumbnails.migrated/`, the migration completed but +could not remove the directory, because something that is not a +thumbnail was inside it — a `.DS_Store` from macOS Finder is the usual +culprit. The tree was moved aside instead of deleted. Its contents are +no longer used and it is safe to remove by hand once you have looked at +what is in there. + +While either directory is absent, the server skips the legacy read path +entirely, at no cost. While `.thumbnails/` is present, reads fall back +to it on a miss, which is what makes the migration invisible to users +while it runs. + +## If something looks wrong + +Every deletion is written to the audit log, naming the job, the file +removed and the blob that replaced it. To review what a migration +removed: + +```bash +journalctl -u oxicloud | grep sidecar_deleted +``` + +A sidecar is only ever deleted after its replacement has been read back +and compared byte-for-byte, so a file that failed that check is still on +disk. Those show up as findings on the job's run in **Admin → Jobs**, +with the reason recorded per file. diff --git a/docs/plan/job-registry.md b/docs/plan/job-registry.md index 98c40617..b9845953 100644 --- a/docs/plan/job-registry.md +++ b/docs/plan/job-registry.md @@ -737,6 +737,98 @@ trigger) resumes any `Paused` row per the normal flow. Consistency-check.md's existing consistency-scoped sweep collapses into this general one. +### Startup jobs — `OXICLOUD_STARTUP_JOBS` + +A comma-separated list of jobs to dispatch once, in the background, +after the scheduler is ready. Each entry is a registered job name, +optionally with the same query syntax the admin trigger URL uses. + +**The default is both migration jobs, in repair mode:** + +``` +OXICLOUD_STARTUP_JOBS=thumb_derived_import?repair=true,thumb_attached_import?repair=true +``` + +An explicit value replaces that list; an empty value disables startup +jobs entirely. + +**Why it exists.** Scheduled ticks deliberately never pass `repair` — a +job that deletes on its default setting is what no-silent-auto-repair +forbids. But that left the migration jobs unable to finish on their +own: a deployment whose operator never opens the admin panel re-imports +sidecars it already imported, forever, and never drains the directory. + +**Why the default deletes anyway.** Relying on operators to edit `.env` +has the same failure mode one level up — the ones who never edit it are +exactly the ones whose migration never completes. So this is a +deliberate exception to no-silent-auto-repair, and it rests on three +properties that must keep holding: + +- **Nothing is deleted before its replacement has been read back.** + `verify_and_unlink` imports, reads the blob back through the normal + stack, and only then unlinks; a store that reported success but landed + unreadable keeps its sidecar. This matters most for + `thumb_attached_import`, whose bytes are user-uploaded previews with + no render path — a wrong deletion there is permanent, where a wrong + deletion of a server-rendered thumbnail costs a re-render. +- **Sidecars whose source is gone are deleted without a readback**, + because there is nothing to read back and nothing can reference them + again. Unrecoverable and unreachable are different things; these are + both. +- **Every deletion is audited**, so what a boot removed, and from which + source, is reconstructable afterwards. + +The consequence to hold in mind: an upgrade deletes on first boot, in +every deployment at once, with no operator action. A regression in the +readback path would be simultaneous and unrecoverable, so that code is +load-bearing. Operators who want to inspect before committing set +`OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import` — +same jobs, import only. + +It is not a "run everything in repair mode" switch. Each job is named +individually and carries its own flags. + +**Validation is fail-fast.** An unknown job name panics at boot — the +registry is fully populated by then, so a name that doesn't resolve is a +typo or a stale rename, and ignoring it would leave a migration that +silently never runs. Unknown flags panic too: a dropped `?repare=true` +would leave the job in discovery-only mode while the operator believed +the tier was draining, and the symptom ("it never finished") surfaces +months later with nothing pointing back at the config. + +**Dispatch is non-blocking.** `tokio::spawn`, so readiness never waits +on a job that may walk a filesystem for hours. Jobs in the list run +sequentially within that task, not concurrently: they contend for the +same directories and pool, and the exclusivity gate would turn overlap +into a *skipped* run rather than a queued one. + +**Interrupted runs resume.** The boot recovery sweep above runs first +and flips every abandoned `Running` row to `Paused` with its cursor +intact; `run_or_resume` then picks Resume over a fresh start. So a +migration killed by a restart continues where it stopped, and completes +across however many restarts it takes. + +That is a deliberate exception to "do NOT auto-resume" — scoped to the +named jobs only. The rule protects against a restart silently resuming +work nobody asked for; here somebody did ask, in configuration, and not +having to ask again is the entire point. Every other paused run still +waits for an operator. + +A resumed run keeps the flags it started with (`repair` / `deep` are +persisted to `params` on the fresh open and read back on resume), so +editing the config mid-migration does not retroactively change a run +already in flight. + +**Safe to leave set.** Each job is idempotent and resumable; once the +tier has drained, a run is a `read_dir` over three directories that +returns nothing — and after the directory is removed, not even that. + +**Visible in the admin panel.** These are ordinary registered jobs: +they appear in `GET /api/admin/jobs`, are triggerable by hand, and +record the same runs and findings. Rows named here additionally carry a +`startup` object with the configured flags, so an operator can see that +a job deletes files on every boot rather than only when someone clicks. + ### Admin surface (recoverable runs) Same URL taxonomy as Part 1 — resource-first, action second, all diff --git a/example.env b/example.env index 6c5baf05..9fa80904 100644 --- a/example.env +++ b/example.env @@ -55,6 +55,36 @@ OXICLOUD_SERVER_HOST=0.0.0.0 # Recommended: 127.0.0.1:9090 with node_exporter-style scrapers. #OXICLOUD_METRICS_LISTEN=127.0.0.1:9090 +# ── Startup jobs ────────────────────────────────────────────────────── +# Background jobs dispatched once, after the scheduler is ready. +# Comma-separated; each entry is a registered job name, optionally with +# the same flags the admin trigger URL takes (force, deep, repair, +# storage). +# +# DEFAULT (applied when this variable is unset): +# thumb_derived_import?repair=true,thumb_attached_import?repair=true +# +# Those two migrate thumbnails out of the legacy .thumbnails/ directory +# into blob storage and then delete the originals, so the migration +# completes without anyone having to trigger it from the admin panel. +# Each sidecar is read back through the normal stack before it is +# unlinked, and every deletion is written to the audit log. +# +# Dispatch is non-blocking — startup never waits on a job. A run +# interrupted by a restart resumes from its cursor on the next boot, so +# a long migration finishes across restarts. Safe to leave at the +# default: the jobs are idempotent, and once the directory is drained a +# run does nothing at all. +# +# An unknown job name or flag is a FATAL error at boot, not a warning — +# a silently ignored entry means a migration that never runs. +# +# To disable every startup job, set this to the empty value: +#OXICLOUD_STARTUP_JOBS= +# +# To import without deleting (inspect first, delete later by hand): +#OXICLOUD_STARTUP_JOBS=thumb_derived_import,thumb_attached_import + # ── Upload size caps ────────────────────────────────────────────────── # See docs/config/storage-fine-tuning.md for sizing guidance. diff --git a/frontend/src/lib/api/types.ts b/frontend/src/lib/api/types.ts index e3757566..b3f44d80 100644 --- a/frontend/src/lib/api/types.ts +++ b/frontend/src/lib/api/types.ts @@ -663,6 +663,19 @@ export interface JobSummary { * for this job. Distinct from `running` — a paused run is * resumable via the same trigger endpoint. */ paused_run?: PausedRunBrief; + /** Present iff `OXICLOUD_STARTUP_JOBS` names this job — the flags it + * is dispatched with at every boot. Worth showing: a job configured + * with `repair: true` deletes on every restart, and the row would + * otherwise suggest that only happens when someone clicks Run. */ + startup?: StartupTrigger; +} + +/** Flags a job configured in `OXICLOUD_STARTUP_JOBS` runs with. */ +export interface StartupTrigger { + force: boolean; + deep: boolean; + repair: boolean; + storage?: string; } /** diff --git a/frontend/src/lib/components/AdminJobsPanel.svelte b/frontend/src/lib/components/AdminJobsPanel.svelte index 60fee78d..66628b64 100644 --- a/frontend/src/lib/components/AdminJobsPanel.svelte +++ b/frontend/src/lib/components/AdminJobsPanel.svelte @@ -860,6 +860,31 @@ {mutatesLabel(job)} {/if} + + {#if job.startup} + + {job.startup.repair + ? t('admin.jobs.startup_repair', 'at boot · repair') + : t('admin.jobs.startup', 'at boot')} + + {/if} {cadenceLabel(job)} {timeAgo(job.last_run_at)} diff --git a/frontend/static/locales/en.json b/frontend/static/locales/en.json index 0e314f87..cac7c577 100644 --- a/frontend/static/locales/en.json +++ b/frontend/static/locales/en.json @@ -1282,6 +1282,10 @@ "run_mutating_confirm_body": "This job changes stored state when it runs.", "mutates_never": "read-only", "mutates_on_repair_only": "read-only unless repaired", + "startup": "at boot", + "startup_repair": "at boot · repair", + "startup_tooltip": "Configured in OXICLOUD_STARTUP_JOBS to run at every boot.", + "startup_repair_tooltip": "Configured in OXICLOUD_STARTUP_JOBS to run in repair mode at every boot.", "run_variants_menu": "Run variants menu", "run_repair_confirm": "Repair", "triggered_ok_repair": "{{name}}: {{n}} counter(s) repaired", diff --git a/src/common/config.rs b/src/common/config.rs index 6efc7db5..31f4ac94 100644 --- a/src/common/config.rs +++ b/src/common/config.rs @@ -2,6 +2,8 @@ use std::env; use std::path::PathBuf; use std::time::Duration; +use crate::infrastructure::scheduler::JobRunArgs; + /// Cache configuration #[derive(Debug, Clone)] pub struct CacheConfig { @@ -2280,6 +2282,148 @@ pub struct GrantCleanupConfig { pub interval_hours: u64, } +/// One job to dispatch once at startup, parsed from an entry of +/// `OXICLOUD_STARTUP_JOBS`. +/// +/// **Why this exists.** Scheduled ticks deliberately never pass +/// `repair` — a job that deletes on its default setting is the thing +/// no-silent-auto-repair forbids. But that leaves the migration jobs in +/// a state where an operator who never opens the admin panel imports +/// forever and never drains: the sidecars are fully redundant, and +/// nothing removes them. Naming the job in configuration IS the +/// deliberate operator action; it just gets taken once, at boot, +/// instead of every time. +/// +/// Not a general "run everything in repair mode" switch. Each job is +/// named individually, and the flags are per job. +/// Holds a [`JobRunArgs`] rather than re-listing its fields. They are +/// the same four flags with the same meanings, and a copy here would +/// have to be found and updated the next time the scheduler grows a +/// fifth — silently ignoring it in configuration until someone noticed. +#[derive(Debug, Clone, Default)] +pub struct StartupJob { + /// Registered job name — must match `JobHandler::name`. + pub name: String, + /// Forwarded verbatim to `JobRegistry::trigger`. + pub args: JobRunArgs, +} + +/// Parse one `OXICLOUD_STARTUP_JOBS` entry: `name`, or +/// `name?repair=true&deep=true`. +/// +/// The query syntax is the one an operator already types at +/// `POST /api/admin/jobs/{name}/trigger?repair=true`, so the value is +/// literally the request they would otherwise make by hand. +/// +/// **Errors on anything it does not recognise**, rather than ignoring +/// it. A silently-dropped `?repare=true` typo would leave the job +/// running in discovery-only mode forever while the operator believed +/// the tier was draining — the failure would surface as "the migration +/// never finishes" months later, with nothing in the logs pointing at +/// the config. Same reasoning as fail-fast on any broken config. +fn parse_startup_job(raw: &str) -> Result { + let raw = raw.trim(); + let (name, query) = match raw.split_once('?') { + Some((n, q)) => (n.trim(), q), + None => (raw, ""), + }; + if name.is_empty() { + return Err("empty job name".to_string()); + } + + let mut job = StartupJob { + name: name.to_string(), + args: JobRunArgs::default(), + }; + + for pair in query.split('&').filter(|p| !p.is_empty()) { + let (key, value) = pair + .split_once('=') + .ok_or_else(|| format!("`{pair}` is not key=value (job `{name}`)"))?; + // Booleans accept only `true`/`false` — the same rule the HTTP + // trigger enforces, so a value that works in one place works in + // the other. See memory `bug_axum_query_bool_only_accepts_true_false`. + let as_bool = || match value { + "true" => Ok(true), + "false" => Ok(false), + other => Err(format!( + "`{key}={other}` on job `{name}`: expected true or false" + )), + }; + match key { + "force" => job.args.force = as_bool()?, + "deep" => job.args.deep = as_bool()?, + "repair" => job.args.repair = as_bool()?, + "storage" => job.args.storage = Some(value.to_string()), + other => { + return Err(format!( + "unknown flag `{other}` on job `{name}`: expected force, deep, repair \ + or storage" + )); + } + } + } + Ok(job) +} + +/// What runs at boot when `OXICLOUD_STARTUP_JOBS` is unset. +/// +/// **Both migration jobs, both in repair mode** — they import their +/// sidecars and then delete them. Chosen deliberately: an operator who +/// never edits `.env` is the normal case, and a migration nobody +/// triggers never finishes, so a default that only imports would leave +/// every untouched deployment carrying a fully-redundant `.thumbnails/` +/// forever. +/// +/// This is a destructive default, which is a real exception to +/// no-silent-auto-repair, so what makes it safe has to hold: +/// +/// * **Nothing is deleted before its replacement has been read back.** +/// `verify_and_unlink` imports, reads the blob back through the normal +/// stack, and only then unlinks. A store that reported success but +/// landed unreadable keeps its sidecar. That readback is the whole +/// safety argument — it matters most for `thumb_attached_import`, +/// whose bytes are user-uploaded previews with no render path, so a +/// wrong deletion there is permanent where a wrong deletion of a +/// server-rendered thumbnail costs only a re-render. +/// * **Sidecars whose source is gone are deleted without a readback**, +/// because there is nothing to read back and nothing can ever +/// reference them again. Unrecoverable and unreachable are different +/// things; these are both. +/// * **Every deletion is audited**, so an operator can reconstruct what +/// a boot removed and from which source. +/// +/// The consequence to be aware of when changing this: an upgrade +/// deletes on first boot, in every deployment at once, with no operator +/// action. A regression in the readback path would therefore be +/// simultaneous and unrecoverable. Treat that code as load-bearing. +/// +/// Set `OXICLOUD_STARTUP_JOBS=` (empty) to disable startup jobs +/// entirely; any explicit value replaces this list rather than adding +/// to it. +const DEFAULT_STARTUP_JOBS: &str = + "thumb_derived_import?repair=true,thumb_attached_import?repair=true"; + +/// Parse the whole `OXICLOUD_STARTUP_JOBS` value. Empty → no startup +/// jobs (an explicit opt-out); unset → [`DEFAULT_STARTUP_JOBS`]. +/// +/// # Panics +/// +/// On any malformed entry. A startup-job list that half-parses is worse +/// than one that fails: the server would come up looking healthy with a +/// migration that never runs. +fn parse_startup_jobs(raw: &str) -> Vec { + raw.split(',') + .map(str::trim) + .filter(|s| !s.is_empty()) + .map(|entry| { + parse_startup_job(entry).unwrap_or_else(|e| { + panic!("OXICLOUD_STARTUP_JOBS: {e}"); + }) + }) + .collect() +} + impl Default for GrantCleanupConfig { fn default() -> Self { Self { @@ -2543,6 +2687,17 @@ pub struct AppConfig { /// bind to loopback / a private interface without exposing /// metrics publicly. pub metrics_listen: Option, + /// Jobs to dispatch once, in the background, after the scheduler is + /// ready. Env: `OXICLOUD_STARTUP_JOBS` — comma-separated, each entry + /// `name` or `name?repair=true`, mirroring the admin trigger URL. + /// + /// Empty by default. Intended for the migration jobs, whose + /// scheduled ticks import but deliberately never delete: naming one + /// here is the operator's standing consent to the deletion, given + /// once in configuration instead of per run in the panel. + /// + /// Dispatch is non-blocking — readiness never waits on a job. + pub startup_jobs: Vec, /// Cache configuration pub cache: CacheConfig, /// Timeout configuration @@ -2671,6 +2826,7 @@ impl Default for AppConfig { plugins: PluginConfig::default(), faces: FacesConfig::default(), metrics_listen: None, + startup_jobs: parse_startup_jobs(DEFAULT_STARTUP_JOBS), } } } @@ -2718,6 +2874,19 @@ impl AppConfig { } } + // Jobs to fire once at boot. Unset keeps DEFAULT_STARTUP_JOBS (set + // by `Default`); any explicit value REPLACES it, and an empty value + // is the opt-out. + // + // Panics on a malformed entry rather than warning: unlike metrics, + // a startup job that silently fails to parse leaves a migration + // that never runs, and the symptom ("the tier never drained") + // surfaces months later with nothing pointing back at the config + // line. + if let Ok(raw) = env::var("OXICLOUD_STARTUP_JOBS") { + config.startup_jobs = parse_startup_jobs(&raw); + } + // Database configuration if let Ok(connection_string) = env::var("OXICLOUD_DB_CONNECTION_STRING") { config.database.connection_string = connection_string; @@ -3783,6 +3952,89 @@ pub fn default_config() -> AppConfig { mod tests { use super::*; + #[test] + fn startup_job_parses_name_and_flags() { + let jobs = parse_startup_jobs( + "thumb_derived_import?repair=true, thumb_attached_import ,blobs_consistency?deep=true&force=false", + ); + assert_eq!(jobs.len(), 3); + + assert_eq!(jobs[0].name, "thumb_derived_import"); + assert!(jobs[0].args.repair); + assert!(!jobs[0].args.deep); + + // Bare name → all flags default off, which is the discovery-only + // run. Naming a migration job without `repair` imports and stops. + assert_eq!(jobs[1].name, "thumb_attached_import"); + assert!(!jobs[1].args.repair); + + assert!(jobs[2].args.deep); + assert!(!jobs[2].args.force); + } + + #[test] + fn startup_job_accepts_storage_scope() { + let jobs = parse_startup_jobs("backend_consistency?storage=s3_prod&deep=true"); + assert_eq!(jobs[0].args.storage.as_deref(), Some("s3_prod")); + assert!(jobs[0].args.deep); + } + + #[test] + fn startup_jobs_empty_value_is_the_opt_out() { + assert!(parse_startup_jobs("").is_empty()); + assert!(parse_startup_jobs(" , ,").is_empty()); + } + + /// Both migration jobs drain themselves out of the box, deletion + /// included. Pinned rather than left implicit because this is a + /// destructive default: it deletes on first boot after an upgrade, + /// everywhere, with no operator action. Whoever changes this line + /// should have to change a test that says so. + /// + /// What keeps it safe is the readback in `verify_and_unlink` — import, + /// read the blob back through the normal stack, and only then unlink. + /// That matters most for `thumb_attached_import`, whose bytes are + /// user-uploaded and have no render path to rebuild them. + #[test] + fn default_startup_jobs_drain_both_thumbnail_tiers() { + let jobs = AppConfig::default().startup_jobs; + let names: Vec<&str> = jobs.iter().map(|j| j.name.as_str()).collect(); + assert_eq!(names, ["thumb_derived_import", "thumb_attached_import"]); + assert!(jobs.iter().all(|j| j.args.repair)); + assert!(jobs.iter().all(|j| !j.args.deep && !j.args.force)); + } + + /// A misspelled flag must not parse. Silently ignoring `repare=true` + /// leaves the job in discovery-only mode while the operator believes + /// the tier is draining — a failure that surfaces months later as + /// "the migration never finished", with nothing pointing at the + /// config line. + #[test] + #[should_panic(expected = "unknown flag `repare`")] + fn startup_job_rejects_a_misspelled_flag() { + parse_startup_jobs("thumb_derived_import?repare=true"); + } + + /// Booleans take only true/false — the same rule the HTTP trigger + /// enforces, so a value that works in one place works in the other. + #[test] + #[should_panic(expected = "expected true or false")] + fn startup_job_rejects_a_non_boolean_flag_value() { + parse_startup_jobs("thumb_derived_import?repair=yes"); + } + + #[test] + #[should_panic(expected = "not key=value")] + fn startup_job_rejects_a_valueless_flag() { + parse_startup_jobs("thumb_derived_import?repair"); + } + + #[test] + #[should_panic(expected = "empty job name")] + fn startup_job_rejects_flags_with_no_job() { + parse_startup_jobs("?repair=true"); + } + #[test] fn empty_allowlist_accepts_any_email() { let cfg = MagicLinkConfig::default(); diff --git a/src/common/di.rs b/src/common/di.rs index dd422dc8..0391185c 100644 --- a/src/common/di.rs +++ b/src/common/di.rs @@ -2835,6 +2835,108 @@ impl AppServiceFactory { registered ); + // `OXICLOUD_STARTUP_JOBS` — dispatch each named job once, now. + // + // Exists for the migration jobs. Their scheduled ticks import but + // never delete (`repair` defaults false, per no-silent-auto-repair), + // so a deployment whose operator never opens the admin panel keeps + // importing sidecars it already imported and never drains the + // directory. Naming the job in configuration IS the deliberate + // consent that rule asks for; it is simply given once, at boot, + // rather than per run. + // + // Validated here, dispatched in the background: + // + // * Unknown names **panic**. The registry is fully populated at this + // point, so a name that does not resolve is a typo or a rename, and + // the failure mode of ignoring it is a migration that silently + // never runs. Fail at boot, where the operator is watching. + // * Dispatch is `tokio::spawn` — readiness must never wait on a job + // that walks a filesystem for hours. + // * Sequential within the task, not concurrent: these jobs contend + // for the same directory and DB, and the exclusivity gate would + // turn overlap into a skipped run rather than a queued one. + // * Safe on every boot, including a crash loop: each is idempotent + // and resumable, and once drained a run is a `read_dir` that + // returns nothing. + // + // **Killed mid-run, this resumes from the cursor.** The boot + // recovery sweep runs earlier in this function and flips every row + // the dead process abandoned in `Running` to `Paused`, keeping its + // cursor. `run_or_resume` then picks Resume over a fresh start, so + // a job interrupted by a restart continues where it stopped rather + // than rescanning from the beginning — and a long migration + // completes across however many restarts it takes. + // + // That is a deliberate exception to `boot_recovery_sweep`'s "we do + // not auto-resume; operators trigger the resume explicitly". The + // rule exists so a restart never silently resumes work nobody + // asked for. Here somebody did ask, in configuration, and the whole + // point of the option is not having to ask again. The exception is + // scoped to the named jobs; every other paused run still waits for + // an operator. + // + // The resumed run keeps the flags it started with — `repair` and + // `deep` are persisted to the run's `params` on the fresh open and + // read back on resume — so editing the config mid-migration does + // not retroactively change a run already in flight. + if !self.config.startup_jobs.is_empty() { + let mut planned = Vec::with_capacity(self.config.startup_jobs.len()); + for job in &self.config.startup_jobs { + if app_state.core.job_registry.get(&job.name).await.is_none() { + panic!( + "OXICLOUD_STARTUP_JOBS names `{}`, which is not a registered job. \ + Check the spelling against GET /api/admin/jobs.", + job.name + ); + } + planned.push(job.clone()); + } + + let registry = app_state.core.job_registry.clone(); + tokio::spawn(async move { + for job in planned { + // Audited, not merely logged: a startup job may delete + // files, and "who asked for this" must be answerable + // afterwards. The answer is the configuration, which is + // exactly what this line records. + tracing::info!( + target: "audit", + event = "job.startup_trigger", + job = %job.name, + force = job.args.force, + deep = job.args.deep, + repair = job.args.repair, + storage = ?job.args.storage, + "👮🏻‍♂️ dispatching `{}` from OXICLOUD_STARTUP_JOBS", + job.name, + ); + match registry.trigger(&job.name, &job.args).await { + Some(outcome) => tracing::info!( + target: "oxicloud::scheduler", + event = "job.startup_completed", + job = %job.name, + outcome = outcome.kind(), + "startup job `{}` finished ({})", + job.name, + outcome.kind(), + ), + // Unreachable — the name was resolved above, and + // nothing unregisters. Logged rather than panicking + // because this is a detached task by then. + None => tracing::error!( + target: "oxicloud::scheduler", + event = "job.startup_vanished", + job = %job.name, + "startup job `{}` disappeared from the registry between \ + validation and dispatch", + job.name, + ), + } + } + }); + } + Ok(app_state) } } diff --git a/src/infrastructure/scheduler/mod.rs b/src/infrastructure/scheduler/mod.rs index 3cb9c908..7e16b270 100644 --- a/src/infrastructure/scheduler/mod.rs +++ b/src/infrastructure/scheduler/mod.rs @@ -37,5 +37,7 @@ pub use recoverable::{ RecoverableJobHandler, RunOutcome, RunProgress, RunStatus, RunSummary, derive_progress, record_or_log, run_or_resume, }; -pub use registry::{JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError}; +pub use registry::{ + JobEntry, JobRegistry, JobSummary, PausedRunBrief, RegisterError, StartupTrigger, +}; pub use types::{ErrCause, JobOutcome, JobRunArgs, Mutates}; diff --git a/src/infrastructure/scheduler/registry.rs b/src/infrastructure/scheduler/registry.rs index 942e79a1..e698261c 100644 --- a/src/infrastructure/scheduler/registry.rs +++ b/src/infrastructure/scheduler/registry.rs @@ -236,11 +236,12 @@ impl JobRegistry { last_outcome, running: state.current_run_start.is_some(), recoverable: entry.handler.is_recoverable(), - // Populated in `list_jobs` handler via a single - // DB round-trip — kept out of the registry - // snapshot to avoid pulling a DB dependency into - // the in-memory scheduler state. + // Both populated in the `list_jobs` handler — one + // from a DB round-trip, one from AppConfig. Kept + // out of the registry snapshot so the in-memory + // scheduler state pulls in neither dependency. paused_run: None, + startup: None, } }) .collect() @@ -346,6 +347,32 @@ pub struct JobSummary { /// picks Resume when the latest row is Paused). #[serde(skip_serializing_if = "Option::is_none")] pub paused_run: Option, + /// Populated iff `OXICLOUD_STARTUP_JOBS` names this job — the flags + /// it will be dispatched with at every boot. + /// + /// Surfaced because the panel would otherwise be silently wrong + /// about the most consequential thing on the row: a job configured + /// with `repair=true` deletes files on every restart, and reading + /// the row you would think that only happens when someone clicks. + /// Filled by the `list_jobs` handler, which has the config; the + /// registry deliberately doesn't. + #[serde(skip_serializing_if = "Option::is_none")] + pub startup: Option, +} + +/// The flags a job configured in `OXICLOUD_STARTUP_JOBS` runs with. +/// +/// Mirrors `JobRunArgs` on the wire rather than embedding it, because +/// this is an API shape the admin panel switches on, and `JobRunArgs` +/// is an internal dispatch type free to change without a frontend +/// release. +#[derive(Debug, Clone, Serialize)] +pub struct StartupTrigger { + pub force: bool, + pub deep: bool, + pub repair: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub storage: Option, } /// Enough info about a paused recoverable run for the admin panel to diff --git a/src/infrastructure/services/thumb_derived_import_service.rs b/src/infrastructure/services/thumb_derived_import_service.rs index 596d7f01..dd9a2d43 100644 --- a/src/infrastructure/services/thumb_derived_import_service.rs +++ b/src/infrastructure/services/thumb_derived_import_service.rs @@ -48,6 +48,15 @@ use crate::infrastructure::services::dedup_service::DedupService; pub const THUMB_DERIVED_IMPORT_JOB_NAME: &str = "thumb_derived_import"; +/// Where the legacy tree is moved when it cannot be deleted. +/// +/// Deletion is always attempted first — this is the fallback for the one +/// case `remove_dir` refuses: a file that is not a sidecar sitting in the +/// directory (Finder's `.DS_Store`, most often). What matters to the read +/// path is that `.thumbnails` stops existing, so moving the tree aside +/// achieves the same thing while preserving whatever the stray file was. +pub(crate) const PARKED_DIR_NAME: &str = ".thumbnails.migrated"; + /// Record a sidecar deletion on the audit channel. /// /// Both import jobs delete user-visible files during a one-way migration, so @@ -182,13 +191,36 @@ impl ThumbDerivedImport { stored_hash: &str, path: &std::path::Path, ) -> bool { - let Ok(meta) = fs::metadata(path).await else { + // Compare CONTENT, not length. + // + // This is the only thing standing between a storage bug and + // permanent loss — `thumb_attached_import` deletes user-uploaded + // previews that have no render path to rebuild them, and with the + // startup-job default it does so on first boot after an upgrade, + // in every deployment at once. A guard that load-bearing should + // prove the bytes are the bytes. + // + // Length alone did not. A blob of the right size and the wrong + // content passed: a key-mapping bug handing back another file's + // preview at the same length would have deleted the original and + // kept the impostor, and thumbnails cluster tightly enough in size + // for that to be a real coincidence rather than a theoretical one. + // + // Re-reading the sidecar costs a few KB of I/O, once per file ever + // migrated. The import path already has these bytes in hand, but + // taking them as an argument would leave the already-imported path + // (which has no bytes, only a file) on a weaker check — one code + // path, one guarantee. + let Ok(sidecar) = fs::read(path).await else { return false; }; + // `read_blob_bytes` streams from the backend, reassembling chunks + // if the blob is chunked — no cache sits in front of it, so this + // proves durability and not merely that a write was acknowledged. let Ok(stored) = dedup.read_blob_bytes(stored_hash).await else { return false; }; - if stored.is_empty() || stored.len() as u64 != meta.len() { + if stored.is_empty() || stored.as_ref() != sidecar.as_slice() { return false; } if fs::remove_file(path).await.is_err() { @@ -592,16 +624,47 @@ impl RecoverableJobHandler for ThumbDerivedImport { "🧹 legacy sidecar directory removed — the fallback read path \ is inert from the next restart" ), - Err(e) => tracing::info!( - target: "oxicloud::dedup", - event = "thumb_derived_import.root_kept", - run_id = %store.run_id(), - path = %self.thumbnails_root.display(), - reason = %e, - "legacy sidecar directory not removed; if this says \ - 'directory not empty' with no sidecars left, something \ - else put a file there (a .DS_Store, typically)" - ), + // Something unrelated to thumbnails is in the directory, so + // `remove_dir` refuses. On macOS that is Finder's `.DS_Store`, + // and it would otherwise keep the fallback alive forever on + // every developer machine. + // + // Move the whole tree aside instead. The sidecars are already + // imported and verified, so nothing here is load-bearing; what + // matters is that `.thumbnails` stops existing, because its + // absence is what the read path tests. Renaming preserves the + // stray file for whoever put it there, and keeps the check a + // single `stat` rather than a directory walk. + Err(_) => { + // `with_file_name`, NOT `with_extension`: the directory is + // `.thumbnails`, and a leading-dot name has no extension as + // far as `Path` is concerned — its whole name is the stem. + // `with_extension("thumbnails.migrated")` would have + // produced `.thumbnails.thumbnails.migrated`. + let parked = self.thumbnails_root.with_file_name(PARKED_DIR_NAME); + match fs::rename(&self.thumbnails_root, &parked).await { + Ok(()) => tracing::info!( + target: "oxicloud::dedup", + event = "thumb_derived_import.root_parked", + run_id = %store.run_id(), + from = %self.thumbnails_root.display(), + to = %parked.display(), + "🧹 legacy sidecar directory could not be removed (a \ + non-sidecar file remains) — moved aside instead. The \ + fallback read path is inert from the next restart; \ + the directory is safe to delete by hand." + ), + Err(e) => tracing::warn!( + target: "oxicloud::dedup", + event = "thumb_derived_import.root_kept", + run_id = %store.run_id(), + path = %self.thumbnails_root.display(), + reason = %e, + "legacy sidecar directory could be neither removed nor \ + moved aside — the fallback read path stays live" + ), + } + } } } @@ -646,6 +709,24 @@ pub(crate) mod tests { /// A second hash, for the JPEG sidecar in `legacy_tree`. const H2: &str = "c222222222222222222222222222222222222222222222222222222222222222"; + /// The park path must be a SIBLING of `.thumbnails`, not a suffixed + /// child of its name. + /// + /// `Path::with_extension` looks right and is wrong here: a leading-dot + /// name has no extension as far as `Path` is concerned — `.thumbnails` + /// is entirely stem — so `with_extension("thumbnails.migrated")` + /// yields `.thumbnails.thumbnails.migrated`. The rename would still + /// have "worked", leaving a directory nobody documented and an + /// operator hunting for the name the runbook promised. + #[test] + fn parked_directory_is_a_sibling_named_thumbnails_migrated() { + let root = std::path::Path::new("/srv/storage/.thumbnails"); + assert_eq!( + root.with_file_name(PARKED_DIR_NAME), + std::path::Path::new("/srv/storage/.thumbnails.migrated"), + ); + } + /// BOTH codecs are claimed, and the format comes from the extension. /// /// `.jpg` was previously rejected here, which was correct only while the diff --git a/src/infrastructure/services/thumbnail_service.rs b/src/infrastructure/services/thumbnail_service.rs index 018bfafd..4142d2e1 100644 --- a/src/infrastructure/services/thumbnail_service.rs +++ b/src/infrastructure/services/thumbnail_service.rs @@ -263,27 +263,19 @@ impl ThumbnailService { /// written a sidecar since step 10d2, so there is nothing to create them /// for. /// - /// The probe tests the SIZE directories, not the root. On macOS Finder - /// drops a `.DS_Store` in the root, which blocks `remove_dir` there - /// forever — gating on the root would keep the fallback alive on every - /// developer machine for a reason that has nothing to do with thumbnails. - /// If no size directory exists, no sidecar can exist. + /// One `stat` on the root. The import job guarantees that is enough: it + /// removes the directory once drained, and when something unrelated + /// keeps `remove_dir` from succeeding — Finder's `.DS_Store`, typically + /// — it renames the tree to `.thumbnails.migrated` rather than leaving + /// it in place. So `.thumbnails` existing always means "there may be + /// sidecars under here", and a stray file cannot pin the fallback open. /// /// Result is cached for the process lifetime. It can only be stale in the /// harmless direction: a drain completing mid-life leaves the flag `true` /// until restart, which costs the same failed opens as today. It never /// goes `false` while sidecars remain. pub async fn initialize(&self) -> std::io::Result<()> { - let mut present = false; - for size in ThumbnailSize::all() { - if fs::metadata(self.thumbnails_root.join(size.dir_name())) - .await - .is_ok() - { - present = true; - break; - } - } + let present = fs::metadata(&self.thumbnails_root).await.is_ok(); self.legacy_sidecars.store(present, Ordering::Relaxed); // Asymmetric on purpose. "Present" is actionable and temporary — it diff --git a/src/interfaces/api/handlers/admin_handler.rs b/src/interfaces/api/handlers/admin_handler.rs index 43f75068..4e184929 100644 --- a/src/interfaces/api/handlers/admin_handler.rs +++ b/src/interfaces/api/handlers/admin_handler.rs @@ -2562,6 +2562,28 @@ pub async fn list_jobs(State(state): State>) -> impl IntoResponse } } + // Mark the jobs `OXICLOUD_STARTUP_JOBS` dispatches at boot. Without + // this the panel is silently wrong about the most consequential thing + // on the row: a job configured with `repair=true` deletes files on + // every restart, and the row would suggest that only ever happens + // when someone clicks Run. + for job in summary.iter_mut() { + if let Some(configured) = state + .core + .config + .startup_jobs + .iter() + .find(|s| s.name == job.name) + { + job.startup = Some(crate::infrastructure::scheduler::StartupTrigger { + force: configured.args.force, + deep: configured.args.deep, + repair: configured.args.repair, + storage: configured.args.storage.clone(), + }); + } + } + (StatusCode::OK, Json(summary)).into_response() }