From 04e0df0c8945a366807c8770db5ff0e8be69957a Mon Sep 17 00:00:00 2001 From: Edouard Vanbelle Date: Wed, 2 Sep 2026 19:24:46 +0200 Subject: [PATCH] test(consistency): exercise the Azure backend against Azurite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an Azurite service and a scenario that audits the Azure backend through `?storage=azurite`. It is the only coverage of that code path in the tree: `AzureBlobBackend` has unit tests for its name parser and ordering, but nothing else speaks the protocol, and a paid account is not an option for CI. Azurite implements the real Blob REST API, so this exercises SharedKey signing, prefix/marker paging, and the 256-way shard walk with its termination. ## Harness `docker-compose.test.yml` gains an azurite service on 10000 (tmpfs, so it dies with the stack). `spawn-db.sh` provisions the container itself, because `AzureBlobBackend::initialize` verifies rather than creates — signed by hand with curl + openssl rather than pulling a ~700 MB `az` image for one PUT. Two traps are commented there: the account key is base64 but HMAC wants raw bytes, and the canonicalized resource repeats the account name (`/{acc}/{acc}/{container}`) because the emulator puts in the path what real Azure puts in the host. Getting that wrong yields 403, not a hint. The `azurite` entry is declared in `server.env` but never activated, so the suite's active backend stays local and only this file reaches Azure. ## What it asserts, and what it cannot A failure surfaces as `ok: false`, because an enumeration error now fails the run rather than degrading to a per-row probe. It deliberately asserts no finding count. The container starts empty and the job's grace window is an hour, so a freshly-uploaded blob is skipped in both directions by design — an audit here can only report zero, and "zero findings" would pass whether enumeration worked or returned nothing. The one positive assert, `scanned_count != 0`, therefore sits on the local control, which does hold blobs; `scanned_count` accumulates via `checkpoint`, which the empty-page early return skips. ## No cutover, deliberately Putting real bytes in the container means `backend_migration ?storage=azurite`, which hangs on the first blob: `head_check` issues a ~40-byte ranged GET, `azure_core` 0.21 attaches `x-ms-range-get-content-crc64` to anything under 4 MiB, Azurite 500s, and the deterministic error is retried forever while `migration_readonly` refuses writes app-wide. The full chain and the rejected workaround are in the file header. The scenario is still ordered last in `run.sh` — it is the only one needing a second service, and the cutover comes back there once the official SDK lands. Co-Authored-By: Claude Opus 5 (1M context) --- tests/api/backend_consistency_azure.hurl | 261 +++++++++++++++++++++++ tests/api/run.sh | 10 +- tests/api/storage_multi_entry.hurl | 5 +- tests/common/docker-compose.test.yml | 29 +++ tests/common/server.env | 20 +- tests/common/spawn-db.sh | 83 ++++++- 6 files changed, 404 insertions(+), 4 deletions(-) create mode 100644 tests/api/backend_consistency_azure.hurl diff --git a/tests/api/backend_consistency_azure.hurl b/tests/api/backend_consistency_azure.hurl new file mode 100644 index 00000000..5338c2bb --- /dev/null +++ b/tests/api/backend_consistency_azure.hurl @@ -0,0 +1,261 @@ +# ============================================================= +# OxiCloud – backend_consistency against the Azure backend +# +# Runs against **Azurite**, the Azure Blob emulator started by +# `tests/common/spawn-db.sh`. It speaks the real Blob REST API, so this +# is the only way to exercise the Azure path without an account. The +# `azurite` storage entry is declared in `server.env` but never +# activated — the suite's active backend stays local, and this file +# reaches Azure explicitly through `?storage=azurite`. +# +# ## What it pins +# +# That Azure enumeration works against a real Blob REST implementation — +# SharedKey signing, prefix/marker paging, the 256-way shard walk and its +# termination. `AzureBlobBackend::list_blob_hashes` has unit tests for +# its name parser and ordering, but nothing else in the tree speaks the +# protocol. +# +# A failure surfaces as `ok: false`, because an enumeration error now +# fails the run. It used to degrade to a per-row probe — walk +# `storage.blobs`, ask "are these bytes there" — which found only the +# DB→backend direction and left orphans undetectable, since bytes no row +# claims are invisible to anything starting from the database. That +# fallback is gone; the module docs on `backend_consistency_service.rs` +# say why. +# +# ## What it deliberately does NOT assert +# +# Any finding count. The Azurite container starts empty and the run's +# grace window is one hour, so a freshly-uploaded blob is skipped in +# both directions by design — an audit here can only ever report zero. +# Asserting "zero findings" would pass whether enumeration worked or +# returned nothing at all. +# +# Real orphan/missing coverage needs blobs on the backend older than the +# grace window, which needs either a cutover into Azurite or a way to +# backdate `last_modified`. See "Why there is no cutover here" below. +# +# Prerequisites: setup.hurl must have run (admin user exists), and +# Azurite must be listening on 10000. +# +# Run: +# hurl --variables-file tests/api/test.env --file-root tests \ +# --test tests/api/backend_consistency_azure.hurl +# ============================================================= + + +# ───────────────────────────────────────────────────────────── +# Step 1 – Login +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/auth/login +Content-Type: application/json +{ + "username": "{{username}}", + "password": "{{password}}" +} + +HTTP 200 +[Captures] +token: jsonpath "$.access_token" + + +# ───────────────────────────────────────────────────────────── +# Step 2 – Upload a file so the registry is not empty. +# +# The bytes land on the ACTIVE (local) backend and never reach Azurite, +# so this does not feed step 3. It feeds step 4: a local sweep over an +# empty `storage.blobs` would satisfy every assertion there while +# comparing nothing, and the control is only a control if it had +# something to compare. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/folders +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "name": "hurl-azure-consistency" +} + +HTTP 201 +[Captures] +folder_id: jsonpath "$.id" + + +POST {{base_url}}/api/files/upload +Authorization: Bearer {{token}} +[MultipartFormData] +folder_id: {{folder_id}} +file: file,fixtures/red-image.png; image/png + +HTTP 201 +[Captures] +file_id: jsonpath "$.id" + + +# ───────────────────────────────────────────────────────────── +# Step 2b – Pre-flight: does the Azure backend work at all? +# +# Cheap, synchronous, and it isolates the failure. It separates "Azurite +# is missing, wedged, or misconfigured" from "enumeration is broken", +# which step 3 alone cannot: a dead container and a broken +# `list_blob_hashes` both surface there as the per-row fallback. +# +# `entry_name` resolves against OXICLOUD_STORAGE_ENTRIES, so this +# exercises the same entry step 3 audits rather than an ad-hoc config. +# It does a health check AND a write/read round-trip, and +# `phase_reached` names how far it got, so a failure points at a step +# rather than at the suite. +# +# Commonest cause of a failure here: the container does not exist. +# `AzureBlobBackend::initialize` verifies rather than creates, so +# `spawn-db.sh` provisions it with a hand-signed PUT. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/settings/storage/test +Authorization: Bearer {{token}} +Content-Type: application/json +{ + "entry_name": "azurite" +} + +HTTP 200 +[Asserts] +jsonpath "$.connected" == true +jsonpath "$.roundtrip_passed" == true +jsonpath "$.bytes_written" != 0 +jsonpath "$.bytes_read" != 0 + + +# ───────────────────────────────────────────────────────────── +# Step 3 – Audit the Azure entry. +# +# `?storage=azurite` builds a backend for that named entry directly, +# bypassing the active-backend pointer — which is also the answer for +# auditing either side mid-migration. +# +# ## Why there is no cutover here — DECIDED, do not retry casually +# +# An earlier version ran `backend_migration ?storage=azurite` first, to +# put real bytes in the container. It hangs, on the first blob, and the +# cause is in `azure_core` 0.21 rather than in anything OxiCloud does. +# +# The chain, all verified in source: +# +# backend_migration_service.rs target.head_check(hash) +# → EncryptedBlobBackend::head_check +# → get_blob_range_stream(hash, 0, HEADER_SIZE) ~40 bytes +# → azure_core Range::as_headers adds x-ms-range-get-content-crc64 +# to ANY range under 4 MiB +# → Azurite answers 500 +# → azure_core classifies 500 retryable, response is deterministic, +# so it retries forever — while migration_readonly refuses writes +# application-wide. +# +# `head_check` is a pre-write format probe on the TARGET, so it fires +# before the first byte is copied. Nothing about the migration job is +# wrong; it works against real S3. +# +# **A workaround exists and was rejected** (2026-09-02): issue an +# unranged `get()` for small requests — its 16 MiB initial range clears +# the 4 MiB threshold, so the header is never sent — and truncate +# client-side. Correct against real Azure, but it pays for an emulator +# with production cost (a 40-byte probe becomes a whole-blob transfer) +# and puts new offset arithmetic on the read path, where a mistake +# serves wrong bytes silently instead of failing. See the note on +# `AzureBlobBackend::get_blob_range_stream`. +# +# So the cutover comes back with the official `azure_storage_blob` 1.x, +# where `range_get_content_crc64` is an explicit field to leave unset — +# and with it the orphan/missing assertions this file cannot make today. +# `docs/plan/jobs-handling-recoverable-error.md` covers the other half: +# the run should have paused with a reason instead of hanging, whatever +# the SDK does. +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=azurite +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] + +# THIS is the assertion that matters, and it is a real one. Enumeration +# failure is no longer degraded into a completed run — any Err from +# `list_blob_hashes` now fails the run — so a broken Azure enumeration +# surfaces right here as `ok: false`, whatever went wrong: signing, +# paging, the shard walk, the cursor. +# +# Before the fallback was deleted this needed a proxy assert on +# `extra_stats.mode`, because a broken enumeration completed "cleanly" +# with half its coverage silently gone. +jsonpath "$.ok" == true +jsonpath "$.outcome.outcome" == "ok" + +# No unexpected notices. `backend_unenumerable` is gone with the +# fallback, so what this now guards against is a stray +# `unknown_backend_file` — a non-canonical name in the blob namespace, +# which on a container we provision ourselves means something wrote +# where it should not have. +jsonpath "$.outcome.extra.severity_counts.anomaly" not exists + +# ── Known weakness, stated rather than hidden ──────────────────────── +# +# A run that enumerates SUCCESSFULLY but returns nothing still passes. +# `scanned_count` would catch that, and it is deliberately not asserted +# here: the container is empty, so the job early-returns before its +# first checkpoint and 0 is the correct answer. Step 4 carries that +# assert instead, on the one entry that does hold blobs. +# +# Making it positive HERE needs the corpus on Azure, which needs the +# cutover — see the header. Planting blobs by hand is not a substitute: +# the DB side would then be walked against a backend that does not hold +# the corpus, and every local blob would report `blob_missing_from_backend`. + + +# ───────────────────────────────────────────────────────────── +# Step 4 – The same job against the LOCAL entry behaves identically. +# +# Azure and local now take the same code path, so this is no longer a +# contrast — it is the control. If a future change re-degrades Azure, +# this passing while step 3 fails localises the break to the Azure +# backend rather than to the job. +# +# It also carries the one POSITIVE assert this file can make. Local +# holds the blob step 2 uploaded, so the sweep gets past the empty-page +# early return and checkpoints — which is the only thing here that +# distinguishes "the merge-join compared something" from "the merge-join +# was handed nothing and completed". +# ───────────────────────────────────────────────────────────── +POST {{base_url}}/api/admin/jobs/backend_consistency/trigger?storage=local_main +Authorization: Bearer {{token}} + +HTTP 200 +[Asserts] +jsonpath "$.ok" == true +jsonpath "$.outcome.outcome" == "ok" + +# `scanned_count` accumulates via `checkpoint`, which the early return +# skips — so a non-zero value means blobs were enumerated AND paired +# against `storage.blobs`, not merely that the run ended cleanly. +jsonpath "$.outcome.extra.scanned_count" != 0 + + +# ───────────────────────────────────────────────────────────── +# Step 5 – Teardown. Hurl files share one database, so a folder left +# behind changes what later scenarios see. +# ───────────────────────────────────────────────────────────── +DELETE {{base_url}}/api/folders/{{folder_id}} +Authorization: Bearer {{token}} + +HTTP 204 + + +GET {{base_url}}/api/trash/resources +Authorization: Bearer {{token}} + +HTTP 200 +[Captures] +trash_id: jsonpath "$.items[?(@.resource.id == '{{folder_id}}')].resource.id" + + +DELETE {{base_url}}/api/trash/{{trash_id}} +Authorization: Bearer {{token}} + +HTTP 200 diff --git a/tests/api/run.sh b/tests/api/run.sh index 1c483ea1..9afd51bf 100755 --- a/tests/api/run.sh +++ b/tests/api/run.sh @@ -225,7 +225,15 @@ hurl --variables-file "$API_DIR/test.env" --file-root "$REPO_ROOT/tests" --test "$API_DIR/webdav_permissions.hurl" \ "$API_DIR/webdav_nested_move_cascade.hurl" \ "$API_DIR/wopi_authz.hurl" \ - "$API_DIR/wopi_shared_drive.hurl" + "$API_DIR/wopi_shared_drive.hurl" \ + `# LAST, deliberately — and kept last even though it no longer cuts` \ + `# the storage pointer over. It is the only scenario that depends on a` \ + `# second service (Azurite on 10000), so if that container is missing` \ + `# or wedged the failure lands after everything else has reported,` \ + `# rather than in the middle of an otherwise-green run. It is also` \ + `# where a cutover comes back once the official Azure SDK lands (see` \ + `# the file header), and that WILL need to be last.` \ + "$API_DIR/backend_consistency_azure.hurl" #bash "$API_DIR/dedup_bulk_upload.sh" diff --git a/tests/api/storage_multi_entry.hurl b/tests/api/storage_multi_entry.hurl index 49bfe561..847e93fb 100644 --- a/tests/api/storage_multi_entry.hurl +++ b/tests/api/storage_multi_entry.hurl @@ -44,19 +44,22 @@ Authorization: Bearer {{admin_token}} HTTP 200 [Asserts] # Two entries declared, in _ENTRIES order. -jsonpath "$.entries" count == 2 +jsonpath "$.entries" count == 3 jsonpath "$.entries[0].name" == "local_main" jsonpath "$.entries[1].name" == "s3_stub" +jsonpath "$.entries[2].name" == "azurite" # Backend types match the declarations. jsonpath "$.entries[0].backend" == "local" jsonpath "$.entries[1].backend" == "s3" +jsonpath "$.entries[2].backend" == "azure" # Active pointer: fresh DB has no `active_backend_name` row, so the # boot fallback picks the FIRST entry in _ENTRIES. jsonpath "$.active_entry_name" == "local_main" jsonpath "$.entries[0].is_active" == true jsonpath "$.entries[1].is_active" == false +jsonpath "$.entries[2].is_active" == false # Read-only mode off on a fresh boot (no in-flight migration, no # stale flag in DB). diff --git a/tests/common/docker-compose.test.yml b/tests/common/docker-compose.test.yml index cd6e5fb3..4412eaeb 100644 --- a/tests/common/docker-compose.test.yml +++ b/tests/common/docker-compose.test.yml @@ -14,3 +14,32 @@ services: interval: 2s timeout: 5s retries: 10 + + # Azure Blob Storage emulator. Speaks the real Blob REST API, so it is + # the only way to exercise the Azure backend without an account — and + # the account it does ship is the fixed, publicly-documented dev pair + # (`devstoreaccount1` + the well-known key), so nothing here is a + # credential worth protecting. + # + # Blob service only: `--blobHost` with no queue/table, since the backend + # uses blobs exclusively and the other two would just be open ports. + # + # tmpfs for the same reason postgres uses it — a run must start from an + # empty container, or a previous run's blobs read as orphans in the very + # audit this exists to test. + azurite-test: + image: mcr.microsoft.com/azure-storage/azurite:3.36.0 + command: azurite-blob --blobHost 0.0.0.0 --blobPort 10000 --location /data --skipApiVersionCheck + ports: + - "10000:10000" + tmpfs: + - /data + healthcheck: + # Azurite answers 400 to an unauthenticated root GET, which is proof + # enough that the HTTP listener is up — the SDK handshake is what + # validates credentials, and that happens later in the backend's + # own `initialize`. + test: ["CMD-SHELL", "nc -z 127.0.0.1 10000 || exit 1"] + interval: 2s + timeout: 5s + retries: 15 diff --git a/tests/common/server.env b/tests/common/server.env index 5c2b7018..a24a4127 100644 --- a/tests/common/server.env +++ b/tests/common/server.env @@ -42,7 +42,7 @@ OXICLOUD_NEXTCLOUD_ENABLED=true # `s3_stub` is declared but never activated — it lets storage_config.hurl # assert the entries table has more than one row without needing a # real S3 backend. -OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub +OXICLOUD_STORAGE_ENTRIES=local_main,s3_stub,azurite OXICLOUD_STORAGE_local_main_BACKEND=local OXICLOUD_STORAGE_s3_stub_BACKEND=s3 OXICLOUD_STORAGE_s3_stub_S3_BUCKET=oxicloud-test-stub @@ -51,6 +51,24 @@ OXICLOUD_STORAGE_s3_stub_S3_ENDPOINT_URL=http://127.0.0.1:9999 OXICLOUD_STORAGE_s3_stub_S3_ACCESS_KEY=stub OXICLOUD_STORAGE_s3_stub_S3_SECRET_KEY=stub +# `azurite` — a REAL, reachable Azure backend, unlike `s3_stub` above. +# It points at the Azurite emulator started by spawn-db.sh, which speaks +# the actual Blob REST API, so this is the only way to exercise the Azure +# path without an account. +# +# Declared but NOT activated: the active backend stays local_main, so the +# rest of the suite is untouched. Tests reach it explicitly through +# `?storage=azurite` on the jobs that accept an entry name — which is +# exactly what that parameter was added for. +# +# The credentials are Azurite's fixed, publicly-documented development +# pair. They are in every Microsoft quickstart; nothing here is secret. +OXICLOUD_STORAGE_azurite_BACKEND=azure +OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_NAME=devstoreaccount1 +OXICLOUD_STORAGE_azurite_AZURE_ACCOUNT_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== +OXICLOUD_STORAGE_azurite_AZURE_CONTAINER=oxicloud-test +OXICLOUD_STORAGE_azurite_AZURE_ENDPOINT_URL=http://127.0.0.1:10000/devstoreaccount1 + RUST_LOG="warn,audit=info,sqlx::migrate=info" #RUST_LOG="warn,audit=info,oxicloud::quota=debug" #RUST_LOG=debug diff --git a/tests/common/spawn-db.sh b/tests/common/spawn-db.sh index 0db7cff3..ec67f59c 100755 --- a/tests/common/spawn-db.sh +++ b/tests/common/spawn-db.sh @@ -52,7 +52,57 @@ wait_for_postgres_ready() { exit 1 } -echo "[setup] Starting test postgres..." +# Create the blob container Azurite serves. +# +# `AzureBlobBackend::initialize` VERIFIES the container exists and fails +# with a 404 if it does not — it deliberately does not create one, since +# auto-creating would turn a typo'd container name into a silently +# working empty container. So the harness provisions it, exactly as an +# operator would in production. +# +# Signed by hand rather than shelling out to the Azure CLI: `az` would +# mean pulling a ~700 MB image into every CI run to issue one PUT. This +# needs only curl and openssl, both already required. +# +# Two things make this fiddly enough to be worth commenting: +# * The key is base64 and HMAC needs raw bytes, so it is decoded and +# re-encoded as hex for `-macopt hexkey:`. +# * The canonicalized resource repeats the account name — +# `/{account}/{account}/{container}` — because the emulator puts the +# account in the URL path where real Azure puts it in the host. This +# is the classic Azurite signing trap; getting it wrong yields 403, +# not a hint. +# +# Idempotent by result: a second run gets 409 ContainerAlreadyExists, +# which is success for our purposes. +create_azurite_container() { + local acc=devstoreaccount1 + local cont=oxicloud-test + # Azurite's fixed development key — hardcoded in the image, published + # by Microsoft, and valid against nothing but a local emulator. + local key='Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==' + local ver=2021-08-06 + local date_hdr hexkey sts sig code + date_hdr=$(LC_ALL=C TZ=GMT date '+%a, %d %b %Y %H:%M:%S GMT') + hexkey=$(printf '%s' "$key" | base64 -d | od -An -tx1 | tr -d ' \n') + # Twelve leading empty lines are the unused standard headers + # (Content-*, Date, If-*, Range) the SharedKey scheme requires in + # fixed positions. + sts="PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:${date_hdr}\nx-ms-version:${ver}\n/${acc}/${acc}/${cont}\nrestype:container" + sig=$(printf '%b' "$sts" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:${hexkey}" -binary | base64) + code=$(curl -s -o /dev/null -w '%{http_code}' -X PUT \ + -H "x-ms-date: ${date_hdr}" \ + -H "x-ms-version: ${ver}" \ + -H "Authorization: SharedKey ${acc}:${sig}" \ + "http://127.0.0.1:10000/${acc}/${cont}?restype=container" 2>/dev/null) + case "$code" in + 201) echo "[setup] Azurite container '${cont}' created." ;; + 409) echo "[setup] Azurite container '${cont}' already exists." ;; + *) echo "[setup] WARNING: could not create Azurite container '${cont}' (HTTP ${code}) — Azure-backed tests will fail." >&2 ;; + esac +} + +echo "[setup] Starting test postgres + azurite..." docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true docker compose -f "$COMPOSE_FILE" up -d echo "[setup] Waiting for postgres on port 5433..." @@ -60,3 +110,34 @@ wait_for_port 127.0.0.1 5433 echo "[setup] Waiting for postgres to accept queries..." wait_for_postgres_ready echo "[setup] Postgres is ready." + +# Azurite backs the `azurite` storage entry, which `backend_consistency` +# audits via `?storage=azurite`. Only a port wait: the entry is declared +# but never activated, so nothing in the suite touches it until a test +# names it, and by then the listener has had the whole server boot to +# settle. A failure here should not take down a run that mostly does not +# use it — so this warns rather than exits, and the Azure test fails on +# its own terms with a clearer message than "setup timed out". +# +# Its own loop rather than `wait_for_port`: that helper calls `exit 1` on +# timeout, and `exit` inside a function ends the script whatever context +# it was called from — so wrapping it in an `if` would not degrade, it +# would just fail later and less clearly. +echo "[setup] Waiting for azurite on port 10000..." +azurite_deadline=$(( $(date +%s) + 30 )) +until nc -z 127.0.0.1 10000 2>/dev/null; do + if [[ $(date +%s) -ge $azurite_deadline ]]; then + echo "[setup] WARNING: azurite did not come up — Azure-backed tests will fail." >&2 + break + fi + sleep 0.5 +done +if nc -z 127.0.0.1 10000 2>/dev/null; then + # Port open only means the listener is up — NOT that the tests can use + # it. Saying "ready" here was misleading: the container was still + # missing, so the first Azure job failed with a 404 while setup had + # already reported success. `create_azurite_container` is what makes + # the claim true, so it is the one that gets to announce it. + echo "[setup] Azurite listening; provisioning container..." + create_azurite_container +fi