GET /api/photos sent only X-Next-Cursor — no ETag — so every gallery
re-mount rebuilt up to 500 PhotoDtos, serde-serialized the whole vector,
and shipped the full body even when nothing changed.
The handler now emits a lightweight content-derived ETag
(hash of before + limit + max(modified_at) + row count) and honours
If-None-Match, with Cache-Control: private, no-cache so the SPA's default
fetch cache mode always revalidates. An unchanged "navigate away and back"
becomes an empty 304 instead of a full rebuild + reserialize + transfer.
The DB query still runs (the cheap part); the win is skipping the DTO
build, serialization, and body bytes.
Proven end-to-end (throwaway Postgres + server, 7 images):
1st GET (no If-None-Match) -> 200 4586 bytes + ETag
2nd GET (If-None-Match matches) -> 304 0 bytes
3rd GET (If-None-Match stale) -> 200 4586 bytes (correctly invalidated)
~655 B/photo, so a full 500-row first page saves ~320 KB + a 500-DTO
build/serialize per unchanged revalidation. Unlike a cold load this is the
common gallery-navigation path, so it hits real user-facing latency.
Regression test: tests/api/photos_etag.hurl (added to the api-test suite).
Methodology in benches/PHOTOS-ETAG.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
read_blob_bytes read the same storage.chunk_manifests PK row twice per
full-blob read — blob_size (SELECT total_size) then read_blob_stream
(SELECT chunk_hashes) — even though both columns live in one row. Fold
them into a single `SELECT chunk_hashes, total_size` and share the chunk
stream builder via a new stream_chunks helper. The legacy (no-manifest)
path is unchanged. Output is identical; the read just costs one fewer DB
round-trip.
Benchmark (examples/bench_blob_manifest.rs, isolates the manifest lookup
against the real Postgres): ~1.9x throughput and p50/p99 roughly halved on
that sub-step; the win is the removed round-trip under pool pressure during
upload bursts. Note this is the manifest sub-step only — end-to-end
read_blob_bytes is dominated by the actual chunk reads, and it is a
background path (thumbnail generation / EXIF / indexing), not normal gallery
serving. Methodology + honest framing in benches/BLOB-MANIFEST.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).
- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
(infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
extract one representative frame as PNG, bounded by its own semaphore + a
per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
to a size- and time-bounded temp file on the data volume, extract a frame, and
reuse the shared render_and_persist_all_webp helper — so video thumbnails are
WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
path; the server is now the source of truth.
Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.
Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a successful OIDC callback the backend redirected the browser to
`{frontend_url}/?oidc_code=…` (the site root). But the SvelteKit SPA only
reads `oidc_code` on the `/login` route: the root route immediately
`goto`s `/files`, and the layout's auth guard bounces an unauthenticated
visitor to `/login?redirect=…` — both of which drop the `oidc_code` query
param. The exchange step (`POST /api/auth/oidc/exchange`) therefore never
runs, so the user lands back on the login form with no session even though
the IdP round-trip and callback succeeded.
Redirect to `{frontend_url}/login?oidc_code=…` instead — the route that
actually performs the exchange. `/login` is public, so the guard doesn't
interfere; after a successful exchange the page navigates on to the app.
This was masked until now by #510 (the duplicate-callback 403 always fired
first); with that fixed, the callback reaches the frontend and this second
bug surfaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thumbnails are now generated eagerly as lossy WebP (the primary codec) and
served to clients that advertise `Accept: image/webp`; JPEG is kept as a lazy
fallback for older clients and NextCloud, generated on first request and then
cached like WebP.
- ThumbnailFormat{Webp,Jpeg} enum threaded through encode/render/generate, the
on-disk path ({hash}.webp / {hash}.jpg), the moka cache key
(file_id, size, format), and cleanup (both formats removed).
- file_handler: parse Accept -> format, format-keyed ETag, `Vary: Accept` on
every response (incl. 304) so shared caches never serve the wrong codec;
Content-Type is byte-sniffed (infer) so it always matches the bytes.
- preview_handler (NextCloud) pins JPEG.
- webp = "0.3" (vendored libwebp via cc, no system dependency).
WEBP_QUALITY=82, chosen via a quality sweep (bench Table E1): SSIM within
~0.005 of JPEG q80 (imperceptible at thumbnail scale) for ~62% fewer bytes. On
the photo-realistic bench corpus the full set (3 sizes x 3 photos) drops 65.6%
(213->73 KB); real photos with edges/text land nearer ~25-40%. Encode is +5ms,
paid once in the eager background generator (off the request path).
The bench corpus is now photo-realistic (per-channel sums of low-frequency
sinusoids) instead of white noise, which had distorted codec byte ratios.
Methodology + numbers in benches/WEBP.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OIDC SSO login intermittently ended on a 403 "Invalid or expired OIDC state
— possible CSRF attack" even though the login had already succeeded
server-side.
Root cause: the (now-removed) legacy vanilla-JS frontend registered a
`/sw.js` service worker that, with navigation preload enabled, double-fetched
the top-level navigation to `/api/auth/oidc/callback`. The OIDC `state` is
single-use, so the first callback consumed it and logged the user in while
the duplicate (~0.4s later) found the state gone and returned the 403 the
browser rendered.
Backend — idempotent callback: after a successful web login, remember
`state -> exchange_code` in a short-lived (120s) cache. A duplicate callback
whose state was already consumed now replays that same redirect instead of
403-ing, returning the cached result directly without re-running the IdP code
exchange (the authorization `code` is single-use too). Keyed by the
unguessable 32-byte state, so it adds no new attack surface and fixes the 403
for everyone — including browsers still running a stale legacy service worker.
Frontend — evict the stale worker: the current SvelteKit app registers no
service worker, so fresh clients can't double-fire. But a browser that
previously loaded the legacy frontend still has `/sw.js` registered and
controlling pages (and `/sw.js` now 404s, so vendor self-cleanup is
inconsistent). killLegacyServiceWorker() runs first in the root layout's
onMount: it surgically unregisters only `/sw.js` workers, drops only the
legacy `oxicloud-cache-*` caches, and reloads once (guarded).
Fixes#510.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The owner short-circuit in PgAclEngine::check ran a PK query
(SELECT user_id FROM storage.folders/files WHERE id=$1) on every authorization
check of a folder/file — the common case, since users mostly act on their own
resources. Memoise it in an owner_cache (moka, TTL 300s, 100k cap). The owner
column is immutable, so this is safe: the cache maps resource -> real owner and
can never grant a non-owner access (a different caller's owner==uid test fails
against the cached owner and falls through to grants); a hard-deleted resource
that briefly resolves to its former owner simply fails later at execution with
NotFound. The per-check sql_queries counter now increments only on a miss.
Removes 1 DB query + 1 pool-connection acquisition per owner check. Magnitude is
deployment-specific (query latency x whether the pool is contended); see
benches/ACL-OWNER-CACHE.md.
Also adds two DB perf-investigation harnesses, gated behind the `bench` feature
(need the dev Postgres; zero prod impact):
- examples/bench_db_pool.rs + benches/DB-POOL.md — pool size vs tail latency
- examples/bench_owner_cache.rs + benches/ACL-OWNER-CACHE.md — owner query vs cache
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the image crate's scalar resampler with fast_image_resize
(AVX2/SSE4.1/NEON) in a shared encode_thumbnail() helper. render_all now
converts to RGB8 once and SIMD-resizes the shared buffer per size. Lanczos3 for
downscaling, CatmullRom when upscaling (Lanczos rings on enlargement).
Also folds the duplicated path-variant generate_all_sizes_background into the
shared render path -- it had missed BOTH shrink-on-load and SIMD resizing -- so
every thumbnail path now goes through one optimised routine (no duplication).
Measured on 14 cores vs the post-1.5 state (benches/BASELINE.md):
- PNG 2.60x faster (33.6->12.9ms), GIF/WebP 1.25-1.6x: full-resolution decode
paths where the resize dominates, so SIMD helps most
- JPEG only ~7% (shrink-on-load already shrank the bitmap) but peak heap fell
another ~2.5x (17.6->7.1MB): tight RGB buffers, RGB conversion once
- quality SSIM 0.986-0.994 at identical dims (>=0.98 gate)
Thumbnails are now exactly max_dim on the long side (e.g. 400x266) vs the old
fit-within 399x266 -- a <=1px change, invisible under object-fit: cover.
Bench example gains an exact-dims quality reference + semaphore throughput table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shrink-on-load decoupled peak heap from source resolution (~18-25 MB per decode
regardless of MP), so the RAM ceiling that justified halving decode concurrency
is gone. max_concurrent_decodes() now defaults to all cores, with an
OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY override for ops tuning.
Measured on the real service path (Semaphore + spawn_blocking), 14 cores:
- 12MP: 92.7 -> 133.7 photos/s (1.44x)
- 24MP: 49.5 -> 69.9 photos/s (1.41x)
peak heap unchanged; cpus*2 yields nothing, confirming cpus is the right ceiling
for CPU-bound work (the gap to 2x is rayon oversubscription -- Task 1.7).
Adds a semaphore-bounded throughput harness (Table D) to the bench example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decode JPEGs at the smallest DCT scale (1/8·1/4·1/2·1/1) whose long axis is
still ≥ the largest needed thumbnail (800px), via jpeg-decoder, instead of a
full-resolution decode through the image crate. The full-res bitmap — the
dominant time and RAM cost — is never materialised. PNG/GIF/WebP and unusual
JPEG colour spaces (CMYK / 16-bit grey) fall back to a full decode.
Extracts the shared decode + EXIF-orientation logic into decode_oriented(),
removing the duplication that existed between render_thumbnail_from_data and
render_all_thumbnails_from_data.
Measured on 14 cores (see benches/BASELINE.md):
- render_all 1.8-2.0× faster (12MP 111->61ms, 48MP 398->203ms)
- peak heap 5.5-14.8× lower, now decoupled from source MP (~18-25MB regardless)
- saturated throughput 3-3.6× (parallel efficiency 4.9×->8.5×)
- quality SSIM 0.987-0.999 (>=0.98 gate), PSNR 47-55dB
Also adds the Phase 0 benchmark harness (gated behind the `bench` feature, zero
prod impact): deterministic image corpus (src/bench_support.rs), criterion
latency bench (benches/thumbnails.rs), and a peak-RAM/throughput/SSIM harness
(examples/bench_thumbnails_mem.rs). Baseline + before/after in benches/BASELINE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An audit (backend /api routes vs SvelteKit frontend usage, adversarially
verified across the whole repo) found these 5 routes have ZERO callers — no
frontend, no test, no protocol layer, no internal caller — and are superseded:
- GET /api/folders/paginated no-op duplicate of GET /api/folders
(discards the page arg); superseded by
the cursor-paginated /{id}/resources.
- POST /api/dedup/upload superseded by /api/files/upload, which
does the identical CDC dedup ingest.
- POST /api/people/{id}/hide the hide-person toggle was never built
into the UI (is_hidden never read).
- GET /api/admin/settings/general never called anywhere.
- GET /api/admin/settings/registration only the PUT is used; the GET had no
caller (PUT kept).
Removes each route, its handler + _impl, the now-orphaned DedupUploadResponse
DTO + its two serialization tests, the set_hidden service method (only caller
was hide_person), and the OpenAPI path/schema registrations.
Also cleans 4 stale markers: two #[allow(deprecated)] that no longer suppress
anything (zero #[deprecated] remain), the "Legacy folder endpoints (contents,
listing)" comment (both already removed), and a "Re-export AppError for backward
compatibility" comment describing a re-export that doesn't exist.
Net -323 lines. The 31 other unused-by-frontend routes (device-code auth,
CardDAV contact-groups, people/photos & music WIP, dedup/admin debug, i18n,
openapi.json) are intentional surface and were left untouched.
cargo clippy --all-features --all-targets -D warnings: clean. cargo test: 446 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).
Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
to frontend/static/ so they ship with the SPA. This also fixes the
favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
to the SvelteKit /nextcloud/error route.
Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
fallback so `just dev` works without a prior build.
Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).
Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
and drop check-contrast/check-headings (coupled to the old token
taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.
Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
(superseded by /resources).
Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of folder uploads "freezing at ~95%": the global Content-Security-
Policy `script-src` was `'self'` + inline-script hashes with NO
`'wasm-unsafe-eval'`. Chromium therefore blocked `WebAssembly.instantiate`
("Wasm code generation disallowed by embedder"), so the vendored BLAKE3/FastCDC
WASM threw on instantiation — both on the main thread (instant by-hash uploads
and the batch dedup check) and inside the delta-upload worker. Every file then
fell back to a plain byte upload, and the backend logs showed 0 check-batch /
0 negotiate calls. Large files (32 MB service logs) compounded it and the
session token expired mid-upload, so the last handful failed.
- web/mod.rs: add `'wasm-unsafe-eval'` to `script-src`. WASM-only, safe variant
— does NOT enable `eval()`/`new Function()`. Restores instant uploads, delta
(sub-file dedup), and the client hashing the idempotent re-upload relies on.
- deltaUpload.ts: liveness watchdog on the delta worker. A healthy worker posts
progress sub-second; if it goes silent for 20 s it is wedged (WASM init or
chunking hung without throwing) — disable delta for this file AND every later
one so they fall straight through to a plain upload instead of each burning
the full 120 s+ delta timeout. Defense-in-depth so a broken WASM path can
never again freeze an upload for minutes.
cargo test: pass. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-uploading a partially-uploaded folder used to surface hundreds of spurious
"already exists" failures, and a file the watchdog aborted (or one the server
committed just before the client gave up) was lost.
Backend — save_file_with_blob_impl (the shared write path for both plain and
by-hash uploads): on a name conflict (23505), if the existing non-trashed file
holds byte-identical content (same folder, same name, same blob hash), return
that file as success instead of erroring. A different-content clash still
conflicts. Re-upload / re-sync becomes a clean no-op for everything already
stored — only the genuinely missing files transfer.
Frontend — uploadWithRetry: each file gets one automatic retry on a transient
failure (quota is never retried). With backend idempotency, retrying an
already-stored file is an instant no-op and a stalled/aborted file gets a real
second chance, so a folder upload self-completes instead of leaving gaps.
cargo test: 448 passed. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundles the backend+frontend delta-upload (content-dedup) feature with a
batch of frontend fixes from this session.
Upload / dedup:
- Client-hashed delta & instant upload (deltaUpload, hashWasm vendor shim)
- Backend dedup batch endpoint (dedup_service, dedup_handler, routes)
- session store owned-hash helpers; unit tests + upload-strategy bench
Frontend UI/UX:
- Colour file-type icons in grid/list (per-type tinted tiles + glyph hue)
- Robust thumbnail fallback; PDFs now show their type icon (backend
generates no PDF thumbnails) instead of a blank tile
- Fix PDF preview: load via a same-origin blob: iframe — the API URL is
blocked by the global X-Frame-Options: DENY in the browser's framed
PDF viewer, matching the existing CSP `frame-src blob:` design
- Groups: localized virtual-group description (no DB schema-note leak),
add nav.groups to the 15 missing locales, fix primary-button contrast
- Repoint --color-text-light → --color-on-accent (was faint grey on accent)
- Nudge the admin role badge off the user-menu header divider
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The normalized cursor-paginated /resources API is live and the bundled
frontend already uses it for favorites, recent, trash and grants. These five
deprecated old-format endpoints had no remaining frontend or protocol
consumers (the Nextcloud handlers call the service layer directly, not these
HTTP routes), so remove them for a uniform API and less duplicate listing
logic:
- GET /api/folders/{id}/contents → use /api/folders/{id}/resources
- GET /api/folders/{id}/contents/paginated → use /api/folders/{id}/resources
- GET /api/favorites → use /api/favorites/resources
- GET /api/recent → use /api/recent/resources
- GET /api/trash → use /api/trash/resources
Removes the HTTP handlers, their routes, OpenAPI path registrations, and the
now-dead list_folder_contents{,_paginated}_impl helpers + unused imports. The
underlying service methods (favorites_service.get_favorites,
trash_service.get_trash_items, etc.) are KEPT — the Nextcloud OCS/trashbin
handlers depend on them.
Deliberately NOT removed: GET /api/folders/{id}/listing. It is still the Files
view's primary data path and offers ETag/304 conditional caching plus
one-shot favorite/share badge sets that /resources does not yet provide;
migrating it needs a separate parity pass on /resources first.
Updates the OpenAPI structure test and the two docs that referenced the
removed paths. `cargo clippy -D warnings` clean; 443 lib tests pass; OpenAPI
regenerates with the 5 paths gone and the /resources replacements present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geotagged photos never appeared in Photos → Places because their GPS was
silently dropped. Capture-metadata extraction read GPS/camera/dimensions
only from kamadak-exif, which rejects many real-world EXIF blocks (phones,
photo editors, anything with a non-standard trailing IFD) with
`InvalidFormat("Unexpected next IFD")` and returns nothing. nom-exif —
already used to upgrade the capture date and far more lenient — parses those
same files fine, so only the date survived and latitude/longitude stayed
NULL.
nom-exif now also reads GPS (and the date) and `merge_image_metadata`
combines both sources:
- nom-exif's tz-correct date wins when present (unchanged behaviour);
- nom-exif's GPS only fills gaps kamadak left, so kamadak still wins when it
parsed the file (no regression);
- when kamadak fails outright, a record is still produced from nom-exif's
date/GPS instead of being discarded.
Existing photos can be backfilled via POST /api/admin/photos/metadata/reextract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two issues kept the Photos → Places map blank once the SPA could boot:
- CSP `worker-src 'self'` blocked MapLibre GL, which spawns its web worker
from a blob: URL, so the map never constructed. Allow `worker-src
'self' blob:` ('self' still covers same-origin workers like delta-upload).
- `checkBasemap()` trusted `res.ok`, but the SPA fallback serves index.html
(HTTP 200, text/html) for any missing path — so a missing basemap.pmtiles
read as "present", and pmtiles.js then choked on HTML ("Wrong magic
number for PMTiles archive"). Reject text/html responses so an absent
basemap falls back cleanly to the themed blank style.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global `script-src 'self'` CSP header blocked every inline <script>,
so the SvelteKit hydration bootstrap never ran and the SPA never mounted
(blank page behind the splash spinner). The anti-FOUC theme init in
app.html was blocked too.
Instead of weakening the policy with 'unsafe-inline', the backend now
builds `script-src` at startup from 'self' plus a SHA-256 hash of every
inline <script> in the served HTML shells. The policy stays strict (no
'unsafe-inline' for scripts) and the hashes are recomputed from the built
assets on each startup, so a frontend rebuild needs no header edit — even
though SvelteKit's bootstrap hash changes every build.
- web::content_security_policy builds the header; web::resolve_static_path
is extracted so serving and hashing read the exact same bytes.
- byte-exact inline-script extraction (skips src= externals), unit-tested
against a known SHA-256 vector and extraction edge cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit migration gated build.rs's static-dist/OUT_DIR generation
behind OXICLOUD_RUST_ASSETS=1 (early return), but login_v2_handler.rs
still embedded the page with
`include_str!(concat!(env!("OUT_DIR"), "/nextcloud-login.html"))`. With
OXICLOUD_RUST_ASSETS unset (the default), that file is never written to
OUT_DIR, so a clean `cargo build` failed to compile. (#489)
Migrate the page off include_str! to an askama template
(templates/nextcloud/login.html), mirroring the existing
DrivePickerTemplate in the same handler. This drops the only
compile-time dependency on the legacy build.rs pipeline, so the
OXICLOUD_RUST_ASSETS=1 CI workaround is no longer needed and is removed
from ci.yml, load-smoke.yml and load-nightly.yml.
Also fix the second failure on #489: with OXICLOUD_RUST_ASSETS=1 the
release pipeline panicked in copy_dir_recursive because `static/locales`
is now a symlink to frontend/static/locales. entry.file_type() reports
the link itself (not its target), so the symlinked directory was routed
to fs::copy and failed with "the source path is neither a regular file
nor a symlink to a regular file". Classify entries with fs::metadata,
which follows symlinks, so symlinked directories are traversed.
static/nextcloud-login.html is removed (its content moved into the
template; no other consumer) and dropped from build.rs HTML_INCLUDE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNCEMN6fC2xSmxqCVbstkd
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.
Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
`PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
which already covers public links as 'token' grants — same membership the
/grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).
Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
badge sets straight from the listing and updates them optimistically on
favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
the single, authoritative, fetch-free source.
Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
Bring chroot to nextcloud capability: login on Nextcloud via username="{user}~{folder-uuid}
Doing a such login will chroot the folder folder-uuid
If user has several folder as root (parent=None), the login flow will request which
folder user want to chroot
now Drive is purely a metadata
each drive has always a root folder
this model minimize Oxicloud changes, and simplify
the Drive name is simply the folder's root's name
note: owner of Drive has more permission that an owner of the root folder
Bring the feature-rich main branch into the frontend Svelte rewrite
(PR #478, base bcn/frontend-svelte-rewrite). main moved well ahead of the
PR's branch point (b8a0018): it added the Places photo-map and People
(faces) backends, photos enhancements, the ReBAC→role-grants migration,
load tests, and more.
Conflicts resolved (4 files):
- Dockerfile: combine the explicit --bin allowlist (defence-in-depth from
main) with the SPA copy from the frontend build stage (PR).
- .github/workflows/ci.yml: keep the PR's Svelte frontend job
(svelte-check + eslint + stylelint + prettier + vitest); the legacy
static/-targeted tsc/locale/icon advisory steps don't fit the new
working-directory: frontend job and svelte-check supersedes them.
- justfile: keep both the new fe-* / dev recipes (PR) and the load-* k6
recipes (main).
- static/locales: keep the PR's symlink (-> ../frontend/static/locales);
main's new photos/people locale keys are folded into the Svelte locale
files alongside the ported views.
Backend (people/places/faces handlers, routes, DI, migrations) merged
cleanly. `cargo check --bins` passes. The new Places/People UI is not yet
in the Svelte app; that is ported in follow-up commits.
Implements the last Phase 2 piece: a working face detector/embedder behind
the new `faces-onnx` cargo feature (mirrors how `plugins` gates wasmtime).
Inert by default — the default build is unchanged and ships the no-op
analyzer.
Pipeline (InsightFace/immich pattern): SCRFD detection with 5-point
landmarks → least-squares similarity alignment to the canonical 112×112
template → ArcFace embedding → L2-normalized 512-d vector.
- face_geometry.rs (always compiled, unit-tested): SCRFD anchor/distance
decode, NMS, the closed-form (complex-number) similarity transform,
bilinear affine warp, NCHW normalization, L2-norm, Laplacian sharpness.
11 unit tests cover the error-prone math with no model needed.
- onnx_face_analyzer.rs (feature `faces-onnx`): wires the geometry to ONNX
Runtime via `ort` (load-dynamic, so libonnxruntime is dlopen'd at runtime
and the crate builds without it). Inference runs on spawn_blocking; each
session is serialized behind a Mutex. Loads via `ort::init_from` (fallible)
not ORT's lazy loader, which would panic under `panic = "abort"`.
- config: FacesConfig + OXICLOUD_FACES_{ORT_DYLIB,DETECTOR_MODEL,
EMBEDDER_MODEL,DET_SIZE,DET_THRESHOLD,NMS_THRESHOLD,INTRA_THREADS}.
- di: build_face_analyzer() loads the real analyzer when the feature is
compiled in and runtime+models are configured; any missing piece or load
failure degrades to the no-op analyzer (logged) so startup never fails.
- ort/ndarray added as optional deps; example.env documents the setup.
Models and the ONNX Runtime dylib are operator-provided at runtime and are
never committed. Cannot be exercised in CI (no models/dylib); the geometry
is unit-tested and the ONNX seam is isolated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 6 — the People HTTP API (mounted only when
OXICLOUD_ENABLE_FACES is on; every handler is caller-scoped):
- GET /api/people list identity clusters
- GET /api/people/{id}/photos a person's photo file ids
- PATCH /api/people/{id} name / rename a person
- POST /api/people/{id}/hide hide / unhide
- POST /api/people/merge merge two clusters
- POST /api/people/recluster re-run clustering
- DELETE /api/people/data erase all face data (opt-out)
- GET /api/people/faces/{file_id} face boxes for lightbox tagging
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 5:
- FaceIndexingService: a FileLifecycleHook that, on image upload, detects +
embeds faces in a background task and stores them. Dedup-aware (clones an
identical blob's faces instead of re-running inference), reindexes on
overwrite, and relies on the DB cascade for deletes. Completely inert when
no model is ready.
- DI: registers the hook in the FileLifecycleService chain and exposes
PeopleService in AppState — both gated on OXICLOUD_ENABLE_FACES, both
using the default no-op analyzer until the operator wires a real ONNX
model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 4:
- recluster(user): union-find connected-components over the user's face
embeddings (cosine threshold); groups of >= min_faces become persons, and
an existing person's name is preserved across reclusters. O(n^2) — fine
for moderate libraries; an ANN index is the documented scale-up.
- caller_id-scoped use cases for the HTTP layer: list_people (non-empty
clusters, cover thumbnail, most-photographed first), person_photos,
faces_for_file, rename, hide, merge, and delete_all (right to erasure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 3: FacePgRepository implements FaceRepository.
Embeddings stored/read as BYTEA (512 little-endian f32), bbox as REAL[].
Every query is user-scoped. Covers face CRUD, person CRUD (create / rename
/ cover / hide), files-for-person, and a transactional delete-all-for-user
(right to erasure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Phase 2 increment 2:
- domain: Face, Person, BoundingBox, DetectedFace (512-d embeddings).
- ports: FaceAnalyzerPort (detect + embed from raw bytes; decodes internally
so the application layer stays image/ML-crate agnostic) and FaceRepository
(user-scoped face/person persistence).
- DTOs: PersonDto, FaceBoxDto.
- NoopFaceAnalyzer — reports is_ready()==false and returns no faces, so the
whole People pipeline compiles and runs inert until the operator wires a
real ONNX-backed analyzer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
First slice of Phase 2 (People / faces):
- migration: `faces` schema with `faces.persons` and `faces.faces`.
Embeddings are stored as BYTEA (512 x f32) — no pgvector extension
dependency; similarity is computed in-app (pgvector/VectorChord is the
documented scale-up). Cascade deletes (by user and by source file)
satisfy the right to erasure.
- OXICLOUD_ENABLE_FACES feature flag, OFF by default (biometric data,
opt-in per deployment).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Three list endpoints resolved each resource with one query per id:
- GET /api/grants/incoming and /api/grants/outgoing used
join_all(ids.map(get_file)) + join_all(ids.map(get_folder)), so a single
page (limit ≤ 200) could demand ~200 concurrent connections from the
20-connection primary pool, causing acquire-timeouts and head-of-line
blocking under load.
- The NextCloud favorites REPORT (oc:filter-files) fetched get_file/
get_folder once per favorite — up to N serial round-trips per sync.
Add by-ids batch reads that mirror the existing get_file/get_folder column
mapping and NOT is_trashed filter:
- FileBlobReadRepository::get_files_by_ids / FolderDbRepository::get_folders_by_ids
(one SELECT ... WHERE id = ANY($1)), exposed as FileRetrievalService::
get_files_by_ids / FolderService::get_folders_by_ids returning DTOs.
- Both grant handlers and the favorites REPORT now issue two batch queries
total and look results up by id, preserving original order. Missing ids
(stale grants whose resource was deleted, or trashed/removed favorites)
drop out exactly as before. No auth-semantics change: these paths already
resolved ids vetted by the authorization engine / favorites table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAzLEQDaLak3dnrEN3YT35
Adds a Places tab to the Photos section: a self-hosted MapLibre GL map of
the user's geotagged photos.
- places.js: lazy-loads vendored MapLibre + pmtiles.js on first open; draws
the server-aggregated clusters (GET /api/photos/geo) as HTML thumbnail
markers (no glyphs/sprites, no client-side clustering); pan/zoom refetches
for the new viewport; click a cluster to zoom in, or a single photo to
open it in the lightbox.
- Optional vector basemap read directly from /basemaps/basemap.pmtiles over
HTTP Range (label-light Protomaps style, light/dark aware); falls back to
a themed background when no basemap is present. ODbL attribution shown.
- "Moments | Places" sub-nav wired into the Photos section.
- enable_places now defaults on, since the route + UI are ready.
No new backend serving code: tower-http ServeDir already serves static/
with Range, so the .pmtiles basemap is just a static file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
JWT access tokens freeze role/identity at login (access 1h, refresh 7d)
and validated tokens are cached for 30s. The Bearer and cookie auth paths
trusted claims.role and never re-checked the account, so demoting an admin,
or disabling/deleting an account, did not revoke access until the token
expired. The app-password path already re-read role/active from the DB;
only the JWT/cookie path had the gap.
Re-validate the caller against the live user record on the token path via
the already-cached get_user_flags (role / is_external / active), bounded by
USER_FLAGS_CACHE_TTL and invalidated eagerly on set_user_active /
change_user_role / delete_user_admin:
- middleware/user.rs: new resolve_live_role helper (+ pure decide_live_role
core) — returns the *current* role, rejects deleted (NotFound) and
deactivated accounts, and fails open on transient lookup errors (mirrors
require_internal_user). Login/refresh remain the canonical active gate.
- middleware/auth.rs: auth_middleware (Bearer + cookie) now populates
CurrentUser with the live role and rejects revoked accounts (Bearer ->
401 AccountInactive; cookie -> fall through to 401/login redirect).
require_admin emits an audit line on denial.
- middleware/admin.rs: require_admin / require_authenticated re-check the
live record, return the live role, and audit admin denials.
Downstream admin gates (dedup_handler, subject_group_handler, OCS) inherit
the live role automatically via CurrentUser / require_authenticated.
Tests: decide_live_role policy (active / demoted / deactivated / deleted /
transient fail-open) and AccountInactive -> 401.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAzLEQDaLak3dnrEN3YT35
Phase 1 server side, gated on OXICLOUD_ENABLE_PLACES (off by default):
- migration: partial index on storage.file_metadata(longitude, latitude).
- FileBlobReadRepository::list_geo_clusters — plain-SQL grid aggregation
(no PostGIS) scoped to the caller's own non-trashed photos, returning a
centroid, count and a representative file id per non-empty cell.
- PlacesService (caller_id-scoped; user-scoped data needs no authz check,
mirroring RecentService) with a zoom→cell-size mapping.
- GET /api/photos/geo?bbox=w,s,e,n&zoom=N returning GeoCluster[]. The route
is mounted only when the Places service is present, and is registered in
the OpenAPI path list.
The map frontend (PMTiles serving + MapLibre module) is deferred pending
the basemap-sourcing and vendoring decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
list_media_files now LEFT JOINs storage.file_metadata and returns each
photo's pixel dimensions next to the sort date. The endpoint wraps FileDto
in a flattened PhotoDto carrying width/height, so the gallery can lay tiles
out at their true aspect ratio (justified layout) without a second per-file
metadata round-trip and without layout shift. FileItem gains optional
width/height. No change to FileDto or its other construction sites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
remove_manifest_reference unlinked a chunk's backing file right after the
row-delete committed — the same TOCTOU the GC grace window was added to
close: a concurrent upload of identical content can re-reference (pin) the
chunk in the gap between commit and unlink, after which the deferred
unlink strands a referenced chunk with no bytes.
Route physical chunk reclamation through the single grace-protected path:
on last reference, delete the manifest and decrement its chunks (stamping
orphaned_at on the ones that reach 0), but leave the chunk rows and files
for garbage_collect() to reclaim once orphaned past the grace window. The
manifest deletion and its blob-keyed thumbnail hook stay eager.
remove_legacy_reference and cleanup_if_orphaned's legacy path are left as
eager deletes on purpose: a legacy whole-file hash can never be re-created
by an ingest (uploads are always CDC now), so there is no writer to race —
the existing "row gone ⇒ no resurrection" reasoning holds for them.
Adds an integration test asserting a CDC manifest dereference leaves
chunks orphaned-but-present, then reclaimed by a post-grace GC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
Deduplication GC (garbage_collect, Phase 2):
- Add an orphan grace period before a ref_count=0 blob's backing file is
physically deleted, mirroring git's gc.pruneExpire. New
storage.blobs.orphaned_at records when a blob last reached ref_count 0;
the delete trigger and every decrement / 0-ref insert path stamp it,
every re-reference clears it.
- Cross-check that no manifest lists the chunk and no file points at the
blob before deleting it (mirrors Phase 1's file check), so a stale
ref_count can only delay collection, never delete live content.
- Unlink the backing files with bounded parallel fan-out.
Together these close a TOCTOU where a concurrent upload of identical
content could re-reference a chunk in the window between the GC row
delete committing and the backing file being unlinked. Individual file
deletes still reclaim eagerly; only bulk empty-trash and the periodic
sweep observe the grace window.
Trash: match ErrorKind::NotFound instead of substring-matching the error
message when treating an already-deleted item as success.
Content-index worker: supervise the drain loop and restart it with
backoff after a panic, instead of letting a panic silently freeze the
search index while the dirty queue grows unbounded.
Adds migration 20260802000000_blob_gc_grace.sql and an integration test
covering the grace window and reference cross-checks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0172rsVwzTwD216R9HXT2aU4
DioCrafts