cjw/Oxicloud
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

fix(migration): cancel releases migration_readonly, pause deliberately does not

Browse Source 
Step 4 of docs/plan/jobs-handling-recoverable-error.md — the sharp edge
the plan flagged, and it was already a live trap independent of the
retry work.

`backend_migration` engages `migration_readonly`, which refuses writes
ACROSS THE WHOLE APPLICATION until cutover. Cancelling it cleared
nothing. The flag is persisted, so the state survived restarts — boot
even logs a warning about coming up read-only — and the only escape was
editing `admin_settings` by hand.

Two paths reach a cancel, and only one of them ran any handler code:

  * a RUNNING row re-enters the handler, which now releases the gate at
    its next cancel poll when the intent is terminal;
  * a PAUSED row does NOT. `request_terminal_cancel` flips it straight
    to Cancelled in SQL with no handler in the loop.

The second is the common case and the one that matters: a migration
paused by an outage, holding the freeze, cancelled by an operator
precisely to get writes back. Fixed in the cancel endpoint, which is the
only place that sees it.

Releasing on cancel is safe because cancel ENDS the run with no swap —
the source is still the active backend, so nothing is left to protect,
and a later retry starts fresh and rescans everything.

**Pause deliberately keeps the gate**, per Ed's call: Ops cancels to
release it. That is not conservatism for its own sake. The cursor is a
position in a hash-ordered walk and stays valid only while nothing
writes; release the gate on pause and a blob written afterwards whose
hash sorts BELOW the cursor is never visited, so the run completes,
flips the pointer, and reads for that hash 404 against a target that
never received it. Releasing on pause becomes safe only once resume
rescans from the start or a final catch-up pass runs under the freeze
before the swap — the plan's follow-up, not this commit.

Both release paths are best effort: a run that has already been
cancelled should not become a hard failure because a DB blip prevented
clearing a flag. The in-memory store happens regardless, so writes
resume in this process; a loud warning names the DB copy needing
attention.

The endpoint check is gated on the job name AND on the flag currently
being set, so it is a no-op for every other job — nothing else ever sets
it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Edouard Vanbelle
2026-09-07 20:05:29 +02:00
parent 303a0421c2
commit bed1d807c3
3 changed files with 147 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/interfaces/api/handlers/admin_handler.rs
+67 -10
View File
@@ -2879,16 +2879,73 @@ pub async fn cancel_job(
.request_terminal_cancel(&name)
.await
{
Ok(Some(run_id)) => (
StatusCode::OK,
Json(serde_json::json!({
"cancelled": true,
"run_id": run_id.to_string(),
"note": "Running row → will land in Cancelled at next batch boundary; \
Paused row → flipped to Cancelled immediately.",
})),
)
.into_response(),
Ok(Some(run_id)) => {
// Cancelling a PAUSED migration has to give writes back
// here, because nothing else will.
//
// A Running row re-enters the handler, which releases the
// gate itself at its next cancel poll. A Paused row does
// not: `request_terminal_cancel` flips it straight to
// Cancelled in SQL with no handler in the loop. That is the
// common case — a migration paused by an outage, holding
// `migration_readonly`, which an operator cancels precisely
// TO get writes back. Without this the app stayed read-only
// forever: the flag is persisted, so even a restart reloaded
// it, and the only escape was editing admin_settings by
// hand.
//
// Safe because cancel ends the run with no swap — the source
// is still the active backend, so there is nothing left for
// the freeze to protect. Releasing on PAUSE would not be
// safe; see `release_readonly_on_terminal_cancel`.
//
// Idempotent and harmless for every other job: the flag is
// only ever set by backend_migration, so clearing it when it
// is already false is a no-op.
if name == crate::infrastructure::services::backend_migration_service::BACKEND_MIGRATION_JOB_NAME
&& state
.migration_readonly
.load(std::sync::atomic::Ordering::Relaxed)
{
if let Some(pool) = state.db_pool.as_ref()
&& let Err(e) =
crate::infrastructure::services::entry_backend::persist_migration_readonly(
pool.as_ref(),
false,
)
.await
{
tracing::warn!(
target: "oxicloud::migration",
event = "storage.migration_readonly.release_persist_failed",
run_id = %run_id,
error = %e,
"could not persist migration_readonly=false after cancelling a paused \
migration; writes resume now but a restart will come up read-only"
);
}
state
.migration_readonly
.store(false, std::sync::atomic::Ordering::Relaxed);
tracing::info!(
target: "audit",
event = "storage.migration_readonly.released",
reason = "paused_migration_cancelled",
run_id = %run_id,
"🚧 migration_readonly released — writes resume, active backend unchanged"
);
}
(
StatusCode::OK,
Json(serde_json::json!({
"cancelled": true,
"run_id": run_id.to_string(),
"note": "Running row → will land in Cancelled at next batch boundary; \
Paused row → flipped to Cancelled immediately.",
})),
)
.into_response()
}
Ok(None) => (
StatusCode::OK,
Json(serde_json::json!({