fix(name duplicate): fixed via NFC normalisation

TL;DR:

    fix duplicate filename via:

    ```
    docker exec <container> migrate-nfc-filenames --dry-run    # preview
    docker exec <container> migrate-nfc-filenames              # execute
    ```

 == issue ==

  Last week I uploaded Capture d'écran 2026-06-03 à 20.04.24.png from the web. It synced down to Nextcloud on my Mac. Two minutes later, the Web UI was showing the file twice.

  Both rows had:
  - the same name
  - the same size
  - the same content hash

  So why two rows? Because to PostgreSQL, the names weren't the same.

  Web upload (browser → Postgres):
    "é" stored as 1 codepoint  (U+00E9)        bytes: c3 a9     ← NFC

  NiextCloud client (macOS → Postgres):
    "é" stored as 2 codepoints (e + U+0301)    bytes: 65 cc 81  ← NFD

  macOS's APFS keeps filenames in NFD (decomposed); browsers send NFC (composed). Visually é and é are identical. To WHERE name = $1 they're two different keys. Our UNIQUE index on (folder_id, name, user_id) never fired — and the row count quietly drifted every time a Mac user touched an accented
  filename.

 == The fix is two halves ==

  1. No new duplicates — every name-receiving boundary (file upload, NC PUT, rename, MOVE, path lookup) now NFC-normalizes before touching the database. The storage invariant becomes "every stored name is NFC".
  2. Clean up existing data — one-shot migrate-nfc-filenames binary walks storage.files, NFC-normalizes any non-NFC row, and resolves the collisions we've accumulated. Same-content duplicates go to trash (recoverable); different-content collisions get renamed with a .duplicate suffix.

 == use of the clean up ==

    example of use (do not forget to define env **DATABASE_URL**)

    either
        `cargo run --bin migrate-nfc-filenames -- --dry-run`
    or
        `cargo build --bin migrate-nfc-filenames`
        `./target/debug/migrate-nfc-filenames --dry-run`

    example:
```
    % ./target/debug/migrate-nfc-filenames --dry-run
    === NFC filename migration (DRY RUN — no writes) ===

    Loaded 543 non-trashed file rows

    NORMALIZE  163451b5-5e6c-404b-9b1e-f4b01a2b7269  user=42433185-4717-416d-9a15-4580fff171ec  'Capture d’écran 2026-03-20 à 14.44.50.png' → 'Capture d’écran 2026-03-20 à 14.44.50.png'
    NORMALIZE  827dddec-4dd5-48c2-a120-dec5289f7d29  user=969deca6-7935-4f12-a430-4d636b62fa3e  'Capture d’écran 2026-04-03 à 15.43.38.png' → 'Capture d’écran 2026-04-03 à 15.43.38.png'
    NORMALIZE  09559934-a620-472d-9ba8-fc3cfeb6dc6f  user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5  'Capture d’écran 2026-06-03 à 20.05.38.png' → 'Capture d’écran 2026-06-03 à 20.05.38.png'
    NORMALIZE  5ce6dbf9-0562-4758-8783-671aa9069590  user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5  'Capture d’écran 2026-06-05 à 11.07.25.png' → 'Capture d’écran 2026-06-05 à 11.07.25.png'
    DEDUP      newer=26bcf82b-99cc-45c8-9d69-dd7e5c4484ff (trash, same blob)  older=df3adc67-a778-424d-a817-b930c75f3b06  user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5  hash=0d2cc7b0ffce2850

    === Summary ===
      scanned                            : 543
      already in NFC                     : 538
      normalized in place (no collision) : 4
      dedup-trashed (same content)       : 1
      renamed to .duplicate              : 0

    DRY RUN — no rows were written. Re-run without --dry-run to apply.
```

    once valid remove --dry-run
This commit is contained in:
Edouard Vanbelle
2026-06-06 17:47:59 +02:00
parent 5a18ec4bac
commit 7db547a669
8 changed files with 433 additions and 8 deletions
+7
View File
@@ -14,6 +14,7 @@ COPY static static
RUN mkdir -p src/bin && \
echo 'fn main() { println!("Dummy build for caching dependencies"); }' > src/main.rs && \
echo 'fn main() {}' > src/bin/generate-openapi.rs && \
echo 'fn main() {}' > src/bin/migrate-nfc-filenames.rs && \
cargo build --release && \
rm -rf src static-dist target/release/deps/oxicloud* target/release/build/oxicloud-*
@@ -56,6 +57,12 @@ RUN apk --no-cache upgrade && \
# Copy the compiled binary and entrypoint (--chmod avoids extra RUN chmod layers)
COPY --from=builder --chmod=755 /app/target/release/oxicloud /usr/local/bin/
# Ship the NFC filename migration binary alongside the server so
# operators can run it inside the container without a separate Rust
# toolchain — `docker exec <container> migrate-nfc-filenames --dry-run`
# to preview, drop `--dry-run` to execute. One-shot tool, safe to
# ship; it only mutates `storage.files` rows whose name ≠ NFC(name).
COPY --from=builder --chmod=755 /app/target/release/migrate-nfc-filenames /usr/local/bin/
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN sed -i 's/\r//' /usr/local/bin/entrypoint.sh && \
chmod 755 /usr/local/bin/entrypoint.sh