Adds full CRUD on mount contents, mirroring the P1 read pattern (handlers/
services classify; authorization stays in the service via the mount-root
folder grant; the provider does the I/O).
- mkdir / rename / delete / move-within branch inside FolderService and
FileManagementService (router injected into both)
- streaming upload via a new ExternalUploadService: the upload handler detects a
mount destination BEFORE the CAS ingest and streams the multipart body
straight to the provider (no BLAKE3/dedup). `write_stream` now takes a
lifetime-bound boxed stream so the borrowing multipart field can be passed
without buffering.
- deletes on mounts are permanent (no trash): the trash-first folder handler
routes `ext:` ids straight to the provider delete; file delete goes through the
branched delete_and_cleanup
- cross-backend move/copy (mount ↔ native, or between mounts) is forbidden
(UnsupportedOperation); the mount root itself cannot be renamed/moved/deleted
- every mutation emits a `target:"audit" event="external_mount.write"` line
- shared mount_dto builders synthesize FolderDto/FileDto from a provider MountStat
Tests: 529 unit + integration tests for mkdir/rename/delete, file rename/delete,
streaming upload, cross-boundary forbid, and stranger-denied — all against real
Postgres + a real provider (testcontainers).
Adds the foundation for external file mounts: admin-configured backends
(raw host filesystem in v1; sftp/webdav/… as future provider kinds) surfaced
as a folder inside a user's drive. Mount contents are virtual/live-passthrough
— read straight from the backend, never stored in storage.files — and are a
deliberately separate, limited storage type (no dedup/sharing/trash/search).
The feature is dark by default (OXICLOUD_ENABLE_EXTERNAL_MOUNTS=false).
P1 scope (this PR): data model, the pluggable provider abstraction, and the
read-only REST surface (mount listing + download). Read-write (P2),
WebDAV/NextCloud path resolution (P3), and the admin UI (P4) follow.
Core model
- Mount root = a real storage.folders row; authorization for everything inside
collapses onto that folder UUID (ltree-ancestry grant cascade).
- Children are virtual, addressed by ext:<mount_id>:<base64url(node_id)> where
node_id is provider-owned and opaque to the rest of the system.
- A lock-free (arc-swap) MountRegistry maps mount-root UUID -> provider; a thin
MountRouter::classify() is the single cheap hook handlers call before parsing
an id as a UUID. With no mounts configured it always returns Regular, so
existing code paths are unchanged.
Added
- migrations/20260805000000_external_mounts.sql (storage.external_mounts, kind + config JSONB)
- domain/services/external_mount_id (id envelope + virtual etags)
- application/ports/external_mount_ports (ExternalMountProvider, MountProviderFactory, repo port)
- infrastructure local_fs_mount_provider (tokio::fs, symlink-escape-safe) + factory
- application MountRegistry + MountRouter, pg ExternalMountRepository
- DI wiring (AppState.mount_router), FeaturesConfig.enable_external_mounts
- listing branch (FolderService::list_mount_dir_with_perms + folder_handler) and
download branch (FileRetrievalService stat/open mount methods + file_handler)
Authorization stays in the service layer (authz.require(Resource::Folder(mount_id)));
handlers only classify. Cross-backend operations are out of scope for P1.
Tests: 529 unit tests + 5 testcontainers integration tests (real Postgres 17),
including end-to-end authorization (owner allowed, stranger denied). Line
coverage of the new modules is 84–100% (cargo-llvm-cov). Known gap:
file_handler::download_mount_file (HTTP glue) needs a full-app test (P4).
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
This reverts ee51b32. The create_new change showed no measurable throughput
benefit — three 9-rep interleaved runs on the same ext4 device swung −12%..+21%
at the 256 KiB CDC size (a negative stat on a warm dentry cache is ~µs, below
the shared-disk noise floor). Applying the same "no change without a measured
win" bar used for the pool-sizing revert: the idiom/TOCTOU angle is real but the
race is already prevented upstream by the PG pin-or-classify serialisation, so
it's defence-in-depth only — not enough to keep an unmeasured change. Reverts
the production change, the bench, and its doc together.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the try_exists(stat) + File::create(O_CREAT|O_TRUNC) pair with a single
OpenOptions::create_new (O_CREAT|O_EXCL), treating AlreadyExists as the existing
idempotent skip. One metadata syscall per new chunk instead of two (each a
spawn_blocking round-trip), and O_EXCL closes the check-then-create TOCTOU the
old pair left open (a racing writer could be truncated).
Honest measurement caveat (benches/BLOB-WRITE.md): the wall-clock throughput
effect is BELOW the noise floor of the test environment — three 9-rep
interleaved runs on the same ext4 device swing −12%..+21% at the 256 KiB CDC
size, because a negative stat on a warm dentry cache is ~µs, dwarfed by the
chunk's create+write+flush. So this is justified as a code-quality / correctness
change (canonical idiom, strictly fewer syscalls, closes a TOCTOU, zero
downside), NOT as a benchmarked perf win.
The sibling idea — reusing the File handle for the fsync sweep — is deliberately
NOT done: sync_blobs is a single end-of-stream sweep over all the upload's new
hashes, so retaining handles would hold thousands of FDs open (>ulimit) on a
large upload. The re-open sweep is a deliberate FD-frugal design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
This reverts the image-pool migration (commit 5629ba6). The bench
(bench_pool_concurrency / POOL-CONCURRENCY.md) measured the one pool it could
isolate — the thumbnail decode semaphore — and found flat throughput, p99 AND
peak RSS (137 MiB) from K=1..16: shrink-on-load already makes each decode
RAM-cheap, so sizing it to the CFS quota gains nothing measurable. Adding code
without a measured benefit isn't worth it.
Kept: the effective_parallelism() helper (it has a *measured* win in the Tokio
runtime — benches/RUNTIME.md) and the benchmark itself (reusable). The ffmpeg
video fan-out has a plausible a-priori case (one OS process per permit) but is
left as a future, deliberately-measured change rather than shipped on
speculation. Doc updated to record the decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The thumbnail decode semaphore, the transcode rayon pool, and the ffmpeg
video-thumbnail fan-out all sized from std::thread::available_parallelism(),
which honours CPU affinity but ignores the CFS bandwidth quota (--cpus /
cgroup cpu.max). Under a container quota they therefore permit one CPU-heavy
task per *host* core onto cores the scheduler can't grant — the same
over-subscription the runtime worker pool had.
Switch all three to common::runtime::effective_parallelism() (= min(affinity,
CFS quota)), and fix the doc comments that wrongly claimed available_parallelism
respects cgroup quotas. No change off-quota (effective == available there); the
env overrides (OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY,
OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY) are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the bare #[tokio::main] with an explicit runtime::Builder so both
pools are sized, logged at startup, and operator-tunable.
#[tokio::main] hides two defaults that misbehave under container limits:
- worker_threads = available_parallelism(), which honours CPU affinity
(sched_getaffinity: cpuset, taskset) but IGNORES the CFS bandwidth quota
(docker --cpus, cgroup v2 cpu.max, v1 cpu.cfs_quota_us). On a 2-core-quota
container on a 64-core host it spawns 64 workers time-slicing 2 cores.
- max_blocking_threads = 512, a multi-GB RSS blast radius for this heavy
spawn_blocking user (thumbnails, transcode, zip, PDF/text extraction,
Argon2 ~19 MB/hash).
New common::runtime module folds the CFS quota back in: effective_parallelism()
= min(available_parallelism, cgroup quota). runtime_pool_sizes() defaults
workers to that and caps the blocking pool at max(32, 8*workers), both
overridable via OXICLOUD_WORKER_THREADS (or TOKIO_WORKER_THREADS) and
OXICLOUD_MAX_BLOCKING_THREADS. The cgroup v1/v2 parsers are pure + unit-tested.
Note: this corrects the premise that "rayon respects the quota, tokio doesn't"
— the image/rayon pools also use available_parallelism(), so they over-spawn
under a CFS quota too; effective_parallelism() is the reusable fix. Unset env
on an uncontended host reproduces the previous worker count exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
OptimizedFileContent::Mmap was constructed nowhere — the documented "Tier 2:
memory-mapped I/O (10-100 MB)" path was never wired, so optimized_inner only
ever returns Bytes (<10 MB) or Stream (>=10 MB). The variant survived only as
an enum case plus two dead match arms in the file and share download handlers.
Remove the variant and its arms, and fix the now-misleading retrieval-service
tier docs (everything >=10 MB streams via CDC chunk reassembly with the
backend read-ahead; there is no mmap tier). Behaviour is unchanged — the
deleted arms were unreachable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The local backend inherited the trait's conservative read_prefetch() = 1
(strictly sequential chunk reassembly), while S3/Azure already use 8. The
prior rationale was that concurrent opens over scattered content-addressed
chunk files turn one sequential read into competing random I/O ("slower
cold"). Benchmarked that assumption with examples/bench_blob_prefetch
(sweeps the buffered(N) depth over a real LocalBlobBackend under disk-bound
vs network-bound consumers and warm vs cold page cache).
Result on SSD-class storage (median MB/s vs N=1):
warm disk-bound N=2 +11.8% N=8 +3.9% N=16 -4.4%
cold disk-bound N=2 +7.2% (no cold regression on SSD)
network-bound (throttled) ~0% at any N — the socket, not the disk, caps it
So N=8 is wrong for local (leaves gain on the table, risks HDD seek thrash)
and the network-bound win the analysis assumed doesn't materialize: buffered()
here overlaps the per-chunk File::open (cheap on local disk), not the data
read. N=2 captures most of the disk-bound gain — which covers localhost/LAN
downloads AND the internal blob reads that drain as fast as the disk delivers
(thumbnail render, transcode, ZIP export, content extraction) — at the lowest
fan-out. Env-tunable via OXICLOUD_LOCAL_READ_PREFETCH (set 1 on seek-bound
HDDs to restore the old behaviour; raise on fast NVMe). Signature unchanged,
so all ~16 LocalBlobBackend::new call sites are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Add an end-to-end and unit test suite for the SvelteKit frontend:
- Playwright e2e specs (tests/e2e/spa) with a throwaway container stack,
codegen scenarios, and an Istanbul-based coverage report pipeline.
- Vitest unit tests across API endpoints, components, stores and composables.
- `data-testid` hooks on interactive elements (AppShell, FileViewer,
ShareDialog, search, photos, files breadcrumbs, login/Nextcloud flows,
public share pages) so the e2e suite can target them deterministically.
- Serve the SPA app-shell CSP from a <meta> policy (svelte.config.js) plus a
middleware that skips the CSP header on HTML; move the Nextcloud Login Flow
v2 grant page to the SvelteKit /nextcloud/login route.
- `just front-codegen` recipe and start-server-spa.sh harness.
Make the test environment robust and consistent:
- Install a deterministic in-memory localStorage/sessionStorage in the Vitest
setup so storage behaves identically across Node versions (Node 26 ships a
native Web Storage global that otherwise shadows jsdom's).
- Pin devenv to Node 26 + PostgreSQL 18 and pin every CI job to Node 26.3.0
so the dev shell and CI run the same toolchain versions.
Repair the API/WebDAV (hurl) suite, which had drifted from the backend:
- Migrate the removed `/api/folders/{id}/listing` endpoint to `/resources`
(cursor-paginated `{items:[{resource_type,resource}]}` shape) across the
batch-copy, grants, nested-group, and WebDAV NC tests + the dav_helpers
wipe routine.
- Stop photos_etag from uploading the dedup-tracked fixture so the dedup
blob-lifecycle test can own its content-addressed blob exclusively.
- dedup_create now asserts the idempotent same-content re-upload (201 +
existing file id) instead of the stale 409 expectation.
Generated coverage reports, nyc output and the e2e server runtime data dir
are gitignored rather than committed.
GET /api/photos sent only X-Next-Cursor — no ETag — so every gallery
re-mount rebuilt up to 500 PhotoDtos, serde-serialized the whole vector,
and shipped the full body even when nothing changed.
The handler now emits a lightweight content-derived ETag
(hash of before + limit + max(modified_at) + row count) and honours
If-None-Match, with Cache-Control: private, no-cache so the SPA's default
fetch cache mode always revalidates. An unchanged "navigate away and back"
becomes an empty 304 instead of a full rebuild + reserialize + transfer.
The DB query still runs (the cheap part); the win is skipping the DTO
build, serialization, and body bytes.
Proven end-to-end (throwaway Postgres + server, 7 images):
1st GET (no If-None-Match) -> 200 4586 bytes + ETag
2nd GET (If-None-Match matches) -> 304 0 bytes
3rd GET (If-None-Match stale) -> 200 4586 bytes (correctly invalidated)
~655 B/photo, so a full 500-row first page saves ~320 KB + a 500-DTO
build/serialize per unchanged revalidation. Unlike a cold load this is the
common gallery-navigation path, so it hits real user-facing latency.
Regression test: tests/api/photos_etag.hurl (added to the api-test suite).
Methodology in benches/PHOTOS-ETAG.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
read_blob_bytes read the same storage.chunk_manifests PK row twice per
full-blob read — blob_size (SELECT total_size) then read_blob_stream
(SELECT chunk_hashes) — even though both columns live in one row. Fold
them into a single `SELECT chunk_hashes, total_size` and share the chunk
stream builder via a new stream_chunks helper. The legacy (no-manifest)
path is unchanged. Output is identical; the read just costs one fewer DB
round-trip.
Benchmark (examples/bench_blob_manifest.rs, isolates the manifest lookup
against the real Postgres): ~1.9x throughput and p50/p99 roughly halved on
that sub-step; the win is the removed round-trip under pool pressure during
upload bursts. Note this is the manifest sub-step only — end-to-end
read_blob_bytes is dominated by the actual chunk reads, and it is a
background path (thumbnail generation / EXIF / indexing), not normal gallery
serving. Methodology + honest framing in benches/BLOB-MANIFEST.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).
- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
(infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
extract one representative frame as PNG, bounded by its own semaphore + a
per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
to a size- and time-bounded temp file on the data volume, extract a frame, and
reuse the shared render_and_persist_all_webp helper — so video thumbnails are
WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
path; the server is now the source of truth.
Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.
Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a successful OIDC callback the backend redirected the browser to
`{frontend_url}/?oidc_code=…` (the site root). But the SvelteKit SPA only
reads `oidc_code` on the `/login` route: the root route immediately
`goto`s `/files`, and the layout's auth guard bounces an unauthenticated
visitor to `/login?redirect=…` — both of which drop the `oidc_code` query
param. The exchange step (`POST /api/auth/oidc/exchange`) therefore never
runs, so the user lands back on the login form with no session even though
the IdP round-trip and callback succeeded.
Redirect to `{frontend_url}/login?oidc_code=…` instead — the route that
actually performs the exchange. `/login` is public, so the guard doesn't
interfere; after a successful exchange the page navigates on to the app.
This was masked until now by #510 (the duplicate-callback 403 always fired
first); with that fixed, the callback reaches the frontend and this second
bug surfaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thumbnails are now generated eagerly as lossy WebP (the primary codec) and
served to clients that advertise `Accept: image/webp`; JPEG is kept as a lazy
fallback for older clients and NextCloud, generated on first request and then
cached like WebP.
- ThumbnailFormat{Webp,Jpeg} enum threaded through encode/render/generate, the
on-disk path ({hash}.webp / {hash}.jpg), the moka cache key
(file_id, size, format), and cleanup (both formats removed).
- file_handler: parse Accept -> format, format-keyed ETag, `Vary: Accept` on
every response (incl. 304) so shared caches never serve the wrong codec;
Content-Type is byte-sniffed (infer) so it always matches the bytes.
- preview_handler (NextCloud) pins JPEG.
- webp = "0.3" (vendored libwebp via cc, no system dependency).
WEBP_QUALITY=82, chosen via a quality sweep (bench Table E1): SSIM within
~0.005 of JPEG q80 (imperceptible at thumbnail scale) for ~62% fewer bytes. On
the photo-realistic bench corpus the full set (3 sizes x 3 photos) drops 65.6%
(213->73 KB); real photos with edges/text land nearer ~25-40%. Encode is +5ms,
paid once in the eager background generator (off the request path).
The bench corpus is now photo-realistic (per-channel sums of low-frequency
sinusoids) instead of white noise, which had distorted codec byte ratios.
Methodology + numbers in benches/WEBP.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OIDC SSO login intermittently ended on a 403 "Invalid or expired OIDC state
— possible CSRF attack" even though the login had already succeeded
server-side.
Root cause: the (now-removed) legacy vanilla-JS frontend registered a
`/sw.js` service worker that, with navigation preload enabled, double-fetched
the top-level navigation to `/api/auth/oidc/callback`. The OIDC `state` is
single-use, so the first callback consumed it and logged the user in while
the duplicate (~0.4s later) found the state gone and returned the 403 the
browser rendered.
Backend — idempotent callback: after a successful web login, remember
`state -> exchange_code` in a short-lived (120s) cache. A duplicate callback
whose state was already consumed now replays that same redirect instead of
403-ing, returning the cached result directly without re-running the IdP code
exchange (the authorization `code` is single-use too). Keyed by the
unguessable 32-byte state, so it adds no new attack surface and fixes the 403
for everyone — including browsers still running a stale legacy service worker.
Frontend — evict the stale worker: the current SvelteKit app registers no
service worker, so fresh clients can't double-fire. But a browser that
previously loaded the legacy frontend still has `/sw.js` registered and
controlling pages (and `/sw.js` now 404s, so vendor self-cleanup is
inconsistent). killLegacyServiceWorker() runs first in the root layout's
onMount: it surgically unregisters only `/sw.js` workers, drops only the
legacy `oxicloud-cache-*` caches, and reloads once (guarded).
Fixes#510.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The owner short-circuit in PgAclEngine::check ran a PK query
(SELECT user_id FROM storage.folders/files WHERE id=$1) on every authorization
check of a folder/file — the common case, since users mostly act on their own
resources. Memoise it in an owner_cache (moka, TTL 300s, 100k cap). The owner
column is immutable, so this is safe: the cache maps resource -> real owner and
can never grant a non-owner access (a different caller's owner==uid test fails
against the cached owner and falls through to grants); a hard-deleted resource
that briefly resolves to its former owner simply fails later at execution with
NotFound. The per-check sql_queries counter now increments only on a miss.
Removes 1 DB query + 1 pool-connection acquisition per owner check. Magnitude is
deployment-specific (query latency x whether the pool is contended); see
benches/ACL-OWNER-CACHE.md.
Also adds two DB perf-investigation harnesses, gated behind the `bench` feature
(need the dev Postgres; zero prod impact):
- examples/bench_db_pool.rs + benches/DB-POOL.md — pool size vs tail latency
- examples/bench_owner_cache.rs + benches/ACL-OWNER-CACHE.md — owner query vs cache
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the image crate's scalar resampler with fast_image_resize
(AVX2/SSE4.1/NEON) in a shared encode_thumbnail() helper. render_all now
converts to RGB8 once and SIMD-resizes the shared buffer per size. Lanczos3 for
downscaling, CatmullRom when upscaling (Lanczos rings on enlargement).
Also folds the duplicated path-variant generate_all_sizes_background into the
shared render path -- it had missed BOTH shrink-on-load and SIMD resizing -- so
every thumbnail path now goes through one optimised routine (no duplication).
Measured on 14 cores vs the post-1.5 state (benches/BASELINE.md):
- PNG 2.60x faster (33.6->12.9ms), GIF/WebP 1.25-1.6x: full-resolution decode
paths where the resize dominates, so SIMD helps most
- JPEG only ~7% (shrink-on-load already shrank the bitmap) but peak heap fell
another ~2.5x (17.6->7.1MB): tight RGB buffers, RGB conversion once
- quality SSIM 0.986-0.994 at identical dims (>=0.98 gate)
Thumbnails are now exactly max_dim on the long side (e.g. 400x266) vs the old
fit-within 399x266 -- a <=1px change, invisible under object-fit: cover.
Bench example gains an exact-dims quality reference + semaphore throughput table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shrink-on-load decoupled peak heap from source resolution (~18-25 MB per decode
regardless of MP), so the RAM ceiling that justified halving decode concurrency
is gone. max_concurrent_decodes() now defaults to all cores, with an
OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY override for ops tuning.
Measured on the real service path (Semaphore + spawn_blocking), 14 cores:
- 12MP: 92.7 -> 133.7 photos/s (1.44x)
- 24MP: 49.5 -> 69.9 photos/s (1.41x)
peak heap unchanged; cpus*2 yields nothing, confirming cpus is the right ceiling
for CPU-bound work (the gap to 2x is rayon oversubscription -- Task 1.7).
Adds a semaphore-bounded throughput harness (Table D) to the bench example.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Decode JPEGs at the smallest DCT scale (1/8·1/4·1/2·1/1) whose long axis is
still ≥ the largest needed thumbnail (800px), via jpeg-decoder, instead of a
full-resolution decode through the image crate. The full-res bitmap — the
dominant time and RAM cost — is never materialised. PNG/GIF/WebP and unusual
JPEG colour spaces (CMYK / 16-bit grey) fall back to a full decode.
Extracts the shared decode + EXIF-orientation logic into decode_oriented(),
removing the duplication that existed between render_thumbnail_from_data and
render_all_thumbnails_from_data.
Measured on 14 cores (see benches/BASELINE.md):
- render_all 1.8-2.0× faster (12MP 111->61ms, 48MP 398->203ms)
- peak heap 5.5-14.8× lower, now decoupled from source MP (~18-25MB regardless)
- saturated throughput 3-3.6× (parallel efficiency 4.9×->8.5×)
- quality SSIM 0.987-0.999 (>=0.98 gate), PSNR 47-55dB
Also adds the Phase 0 benchmark harness (gated behind the `bench` feature, zero
prod impact): deterministic image corpus (src/bench_support.rs), criterion
latency bench (benches/thumbnails.rs), and a peak-RAM/throughput/SSIM harness
(examples/bench_thumbnails_mem.rs). Baseline + before/after in benches/BASELINE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An audit (backend /api routes vs SvelteKit frontend usage, adversarially
verified across the whole repo) found these 5 routes have ZERO callers — no
frontend, no test, no protocol layer, no internal caller — and are superseded:
- GET /api/folders/paginated no-op duplicate of GET /api/folders
(discards the page arg); superseded by
the cursor-paginated /{id}/resources.
- POST /api/dedup/upload superseded by /api/files/upload, which
does the identical CDC dedup ingest.
- POST /api/people/{id}/hide the hide-person toggle was never built
into the UI (is_hidden never read).
- GET /api/admin/settings/general never called anywhere.
- GET /api/admin/settings/registration only the PUT is used; the GET had no
caller (PUT kept).
Removes each route, its handler + _impl, the now-orphaned DedupUploadResponse
DTO + its two serialization tests, the set_hidden service method (only caller
was hide_person), and the OpenAPI path/schema registrations.
Also cleans 4 stale markers: two #[allow(deprecated)] that no longer suppress
anything (zero #[deprecated] remain), the "Legacy folder endpoints (contents,
listing)" comment (both already removed), and a "Re-export AppError for backward
compatibility" comment describing a re-export that doesn't exist.
Net -323 lines. The 31 other unused-by-frontend routes (device-code auth,
CardDAV contact-groups, people/photos & music WIP, dedup/admin debug, i18n,
openapi.json) are intentional surface and were left untouched.
cargo clippy --all-features --all-targets -D warnings: clean. cargo test: 446 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).
Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
to frontend/static/ so they ship with the SPA. This also fixes the
favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
to the SvelteKit /nextcloud/error route.
Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
fallback so `just dev` works without a prior build.
Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).
Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
and drop check-contrast/check-headings (coupled to the old token
taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.
Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
(superseded by /resources).
Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of folder uploads "freezing at ~95%": the global Content-Security-
Policy `script-src` was `'self'` + inline-script hashes with NO
`'wasm-unsafe-eval'`. Chromium therefore blocked `WebAssembly.instantiate`
("Wasm code generation disallowed by embedder"), so the vendored BLAKE3/FastCDC
WASM threw on instantiation — both on the main thread (instant by-hash uploads
and the batch dedup check) and inside the delta-upload worker. Every file then
fell back to a plain byte upload, and the backend logs showed 0 check-batch /
0 negotiate calls. Large files (32 MB service logs) compounded it and the
session token expired mid-upload, so the last handful failed.
- web/mod.rs: add `'wasm-unsafe-eval'` to `script-src`. WASM-only, safe variant
— does NOT enable `eval()`/`new Function()`. Restores instant uploads, delta
(sub-file dedup), and the client hashing the idempotent re-upload relies on.
- deltaUpload.ts: liveness watchdog on the delta worker. A healthy worker posts
progress sub-second; if it goes silent for 20 s it is wedged (WASM init or
chunking hung without throwing) — disable delta for this file AND every later
one so they fall straight through to a plain upload instead of each burning
the full 120 s+ delta timeout. Defense-in-depth so a broken WASM path can
never again freeze an upload for minutes.
cargo test: pass. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-uploading a partially-uploaded folder used to surface hundreds of spurious
"already exists" failures, and a file the watchdog aborted (or one the server
committed just before the client gave up) was lost.
Backend — save_file_with_blob_impl (the shared write path for both plain and
by-hash uploads): on a name conflict (23505), if the existing non-trashed file
holds byte-identical content (same folder, same name, same blob hash), return
that file as success instead of erroring. A different-content clash still
conflicts. Re-upload / re-sync becomes a clean no-op for everything already
stored — only the genuinely missing files transfer.
Frontend — uploadWithRetry: each file gets one automatic retry on a transient
failure (quota is never retried). With backend idempotency, retrying an
already-stored file is an instant no-op and a stalled/aborted file gets a real
second chance, so a folder upload self-completes instead of leaving gaps.
cargo test: 448 passed. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundles the backend+frontend delta-upload (content-dedup) feature with a
batch of frontend fixes from this session.
Upload / dedup:
- Client-hashed delta & instant upload (deltaUpload, hashWasm vendor shim)
- Backend dedup batch endpoint (dedup_service, dedup_handler, routes)
- session store owned-hash helpers; unit tests + upload-strategy bench
Frontend UI/UX:
- Colour file-type icons in grid/list (per-type tinted tiles + glyph hue)
- Robust thumbnail fallback; PDFs now show their type icon (backend
generates no PDF thumbnails) instead of a blank tile
- Fix PDF preview: load via a same-origin blob: iframe — the API URL is
blocked by the global X-Frame-Options: DENY in the browser's framed
PDF viewer, matching the existing CSP `frame-src blob:` design
- Groups: localized virtual-group description (no DB schema-note leak),
add nav.groups to the 15 missing locales, fix primary-button contrast
- Repoint --color-text-light → --color-on-accent (was faint grey on accent)
- Nudge the admin role badge off the user-menu header divider
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The normalized cursor-paginated /resources API is live and the bundled
frontend already uses it for favorites, recent, trash and grants. These five
deprecated old-format endpoints had no remaining frontend or protocol
consumers (the Nextcloud handlers call the service layer directly, not these
HTTP routes), so remove them for a uniform API and less duplicate listing
logic:
- GET /api/folders/{id}/contents → use /api/folders/{id}/resources
- GET /api/folders/{id}/contents/paginated → use /api/folders/{id}/resources
- GET /api/favorites → use /api/favorites/resources
- GET /api/recent → use /api/recent/resources
- GET /api/trash → use /api/trash/resources
Removes the HTTP handlers, their routes, OpenAPI path registrations, and the
now-dead list_folder_contents{,_paginated}_impl helpers + unused imports. The
underlying service methods (favorites_service.get_favorites,
trash_service.get_trash_items, etc.) are KEPT — the Nextcloud OCS/trashbin
handlers depend on them.
Deliberately NOT removed: GET /api/folders/{id}/listing. It is still the Files
view's primary data path and offers ETag/304 conditional caching plus
one-shot favorite/share badge sets that /resources does not yet provide;
migrating it needs a separate parity pass on /resources first.
Updates the OpenAPI structure test and the two docs that referenced the
removed paths. `cargo clippy -D warnings` clean; 443 lib tests pass; OpenAPI
regenerates with the 5 paths gone and the /resources replacements present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Geotagged photos never appeared in Photos → Places because their GPS was
silently dropped. Capture-metadata extraction read GPS/camera/dimensions
only from kamadak-exif, which rejects many real-world EXIF blocks (phones,
photo editors, anything with a non-standard trailing IFD) with
`InvalidFormat("Unexpected next IFD")` and returns nothing. nom-exif —
already used to upgrade the capture date and far more lenient — parses those
same files fine, so only the date survived and latitude/longitude stayed
NULL.
nom-exif now also reads GPS (and the date) and `merge_image_metadata`
combines both sources:
- nom-exif's tz-correct date wins when present (unchanged behaviour);
- nom-exif's GPS only fills gaps kamadak left, so kamadak still wins when it
parsed the file (no regression);
- when kamadak fails outright, a record is still produced from nom-exif's
date/GPS instead of being discarded.
Existing photos can be backfilled via POST /api/admin/photos/metadata/reextract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two issues kept the Photos → Places map blank once the SPA could boot:
- CSP `worker-src 'self'` blocked MapLibre GL, which spawns its web worker
from a blob: URL, so the map never constructed. Allow `worker-src
'self' blob:` ('self' still covers same-origin workers like delta-upload).
- `checkBasemap()` trusted `res.ok`, but the SPA fallback serves index.html
(HTTP 200, text/html) for any missing path — so a missing basemap.pmtiles
read as "present", and pmtiles.js then choked on HTML ("Wrong magic
number for PMTiles archive"). Reject text/html responses so an absent
basemap falls back cleanly to the themed blank style.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global `script-src 'self'` CSP header blocked every inline <script>,
so the SvelteKit hydration bootstrap never ran and the SPA never mounted
(blank page behind the splash spinner). The anti-FOUC theme init in
app.html was blocked too.
Instead of weakening the policy with 'unsafe-inline', the backend now
builds `script-src` at startup from 'self' plus a SHA-256 hash of every
inline <script> in the served HTML shells. The policy stays strict (no
'unsafe-inline' for scripts) and the hashes are recomputed from the built
assets on each startup, so a frontend rebuild needs no header edit — even
though SvelteKit's bootstrap hash changes every build.
- web::content_security_policy builds the header; web::resolve_static_path
is extracted so serving and hashing read the exact same bytes.
- byte-exact inline-script extraction (skips src= externals), unit-tested
against a known SHA-256 vector and extraction edge cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SvelteKit migration gated build.rs's static-dist/OUT_DIR generation
behind OXICLOUD_RUST_ASSETS=1 (early return), but login_v2_handler.rs
still embedded the page with
`include_str!(concat!(env!("OUT_DIR"), "/nextcloud-login.html"))`. With
OXICLOUD_RUST_ASSETS unset (the default), that file is never written to
OUT_DIR, so a clean `cargo build` failed to compile. (#489)
Migrate the page off include_str! to an askama template
(templates/nextcloud/login.html), mirroring the existing
DrivePickerTemplate in the same handler. This drops the only
compile-time dependency on the legacy build.rs pipeline, so the
OXICLOUD_RUST_ASSETS=1 CI workaround is no longer needed and is removed
from ci.yml, load-smoke.yml and load-nightly.yml.
Also fix the second failure on #489: with OXICLOUD_RUST_ASSETS=1 the
release pipeline panicked in copy_dir_recursive because `static/locales`
is now a symlink to frontend/static/locales. entry.file_type() reports
the link itself (not its target), so the symlinked directory was routed
to fs::copy and failed with "the source path is neither a regular file
nor a symlink to a regular file". Classify entries with fs::metadata,
which follows symlinks, so symlinked directories are traversed.
static/nextcloud-login.html is removed (its content moved into the
template; no other consumer) and dropped from build.rs HTML_INCLUDE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNCEMN6fC2xSmxqCVbstkd
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.
Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
`PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
which already covers public links as 'token' grants — same membership the
/grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).
Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
badge sets straight from the listing and updates them optimistically on
favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
the single, authoritative, fetch-free source.
Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
Bradley Nelson