- ensure Oxicloud s correctly functional with OIDC login
- refresh token works
- givenname, familyname, picture are imported in users' informations
- test relogin (server is not using same path)
- test email verified
- test admin group provided from IdP
lopdf <0.42.0 has an unbounded-recursion stack overflow on deeply
nested PDF objects (advisory 2026-06-21). The vector through OxiCloud
is the search-index text extractor — anyone who can upload a file can
ship a malicious PDF, and the existing catch_unwind in
text_extractor::extract_pdf does not save us: a stack overflow aborts
the process, it is not a panic.
pdf-extract 0.12.0 requires lopdf ^0.42 which adds the depth bound;
the only consumer call (`extract_text_from_mem`) is API-compatible,
no source changes needed.
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
Stamp `orphaned_at = now()` on the ref-0 row so it sits inside the
GC grace window for the duration of this test. Without it,
`orphaned_at IS NULL` is treated by `garbage_collect` as
"pre-migration, immediately reapable" — and any sibling test in
the shared pool that calls `garbage_collect()` (e.g.
`garbage_collect_respects_grace_and_cross_checks`) would race
with the pin below and delete the row first.
call fire_blob_hooks to respect lifecycle
dedup_service::garbage_collect_with_grac must call fire_blob_hooks() once blob are dropped
so other services like thumbnail can proceed to their cleanup
renable thumbnail test, ensure that blob lifecycle correctly
trigger thumbnail cleanup on blob deletion
need to call `/api/admin/internal/trigger-gc?force=true`
- quota per drive
- add 2 internal API endpoints to test purpose
disabled by default, enable it via `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
this enable:
/api/admin/internal/trigger-sweep
to sweep the trash and recalculated quota
/api/admin/internal/trigger-gc
to garbage orphan blobs
use full for end to end tests and validate lifecycles
normalize environment to avoid such issues during tests
on non EN local machine:
```
AssertionError: expected 'il y a 2 ans' to match /year/
❯ src/lib/utils/time.test.ts:24:60
22|
23| it('formats past times in the largest matching unit', () => {
24| expect(relativeTimeAgo(Date.now() - 2 * 31_536_000_000)).toMatch(/year/);
```
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
These example targets landed unformatted on main and fail the Rustfmt CI
check (`cargo fmt --all --check`); reformat them so this PR's checks pass.
No logic changes.
The e2e CI job ran the legacy `scenarios/*` specs against the vanilla `static/`
frontend that upstream has since removed, so it could never pass. Point CI at
this repo's SvelteKit SPA suite (tests/e2e/spa) and wire up what it needs:
- CI: build the release binary with `--features plugins` (the admin Plugins-tab
specs exercise the WASM runtime) and run `npm run test:coverage`, building the
instrumented SPA with COVERAGE=1 VITE_E2E=1 so the server serves the
data-testid-instrumented build the specs drive.
- Coverage harness: target 127.0.0.1 instead of `localhost` (which resolves to
::1 first on CI runners while the server binds IPv4, so readiness never
connected) and poll `/ready` for webServer readiness; tee start-server-spa.sh
output to a log surfaced by an always-run CI step for diagnostics.
- Files page: restore a persistent breadcrumb home link (buildCrumbs returns
only the path folders, so there was no "go home" affordance), and fix the
`?file=` deep-link race where the viewer→URL effect stripped the param before
the listing loaded — a bookmarked preview link now opens the viewer.
All 101 spa specs pass locally.
This reverts ee51b32. The create_new change showed no measurable throughput
benefit — three 9-rep interleaved runs on the same ext4 device swung −12%..+21%
at the 256 KiB CDC size (a negative stat on a warm dentry cache is ~µs, below
the shared-disk noise floor). Applying the same "no change without a measured
win" bar used for the pool-sizing revert: the idiom/TOCTOU angle is real but the
race is already prevented upstream by the PG pin-or-classify serialisation, so
it's defence-in-depth only — not enough to keep an unmeasured change. Reverts
the production change, the bench, and its doc together.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the try_exists(stat) + File::create(O_CREAT|O_TRUNC) pair with a single
OpenOptions::create_new (O_CREAT|O_EXCL), treating AlreadyExists as the existing
idempotent skip. One metadata syscall per new chunk instead of two (each a
spawn_blocking round-trip), and O_EXCL closes the check-then-create TOCTOU the
old pair left open (a racing writer could be truncated).
Honest measurement caveat (benches/BLOB-WRITE.md): the wall-clock throughput
effect is BELOW the noise floor of the test environment — three 9-rep
interleaved runs on the same ext4 device swing −12%..+21% at the 256 KiB CDC
size, because a negative stat on a warm dentry cache is ~µs, dwarfed by the
chunk's create+write+flush. So this is justified as a code-quality / correctness
change (canonical idiom, strictly fewer syscalls, closes a TOCTOU, zero
downside), NOT as a benchmarked perf win.
The sibling idea — reusing the File handle for the fsync sweep — is deliberately
NOT done: sync_blobs is a single end-of-stream sweep over all the upload's new
hashes, so retaining handles would hold thousands of FDs open (>ulimit) on a
large upload. The re-open sweep is a deliberate FD-frugal design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
This reverts the image-pool migration (commit 5629ba6). The bench
(bench_pool_concurrency / POOL-CONCURRENCY.md) measured the one pool it could
isolate — the thumbnail decode semaphore — and found flat throughput, p99 AND
peak RSS (137 MiB) from K=1..16: shrink-on-load already makes each decode
RAM-cheap, so sizing it to the CFS quota gains nothing measurable. Adding code
without a measured benefit isn't worth it.
Kept: the effective_parallelism() helper (it has a *measured* win in the Tokio
runtime — benches/RUNTIME.md) and the benchmark itself (reusable). The ffmpeg
video fan-out has a plausible a-priori case (one OS process per permit) but is
left as a future, deliberately-measured change rather than shipped on
speculation. Doc updated to record the decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Adds Part B (peak RSS for K concurrent decodes) to bench_pool_concurrency and
records the findings in benches/POOL-CONCURRENCY.md.
Honest result: under a 2-core quota the thumbnail decode pool shows flat
throughput, p99, AND peak RSS (137 MiB) from K=1..16 — shrink-on-load already
made each decode RAM-cheap, so over-permitting costs nothing measurable here.
The effective_parallelism() migration is therefore a correctness/consistency
change with no downside, mainly protecting the transcode + ffmpeg pools (and
extreme host-core/quota ratios) this box can't reproduce — not a throughput win.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
bench_pool_concurrency drives the real service path (Semaphore(K) gating
spawn_blocking(bench_render_all)) with a gallery of concurrent callers and
sweeps the decode-permit count K, reporting throughput + p50/p99. Run under
taskset to model a CPU quota: it shows the effect of sizing the image pools to
effective_parallelism (K=cores) vs the host count (over-subscribed).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The thumbnail decode semaphore, the transcode rayon pool, and the ffmpeg
video-thumbnail fan-out all sized from std::thread::available_parallelism(),
which honours CPU affinity but ignores the CFS bandwidth quota (--cpus /
cgroup cpu.max). Under a container quota they therefore permit one CPU-heavy
task per *host* core onto cores the scheduler can't grant — the same
over-subscription the runtime worker pool had.
Switch all three to common::runtime::effective_parallelism() (= min(affinity,
CFS quota)), and fix the doc comments that wrongly claimed available_parallelism
respects cgroup quotas. No change off-quota (effective == available there); the
env overrides (OXICLOUD_THUMBNAIL_DECODE_CONCURRENCY,
OXICLOUD_VIDEO_THUMBNAIL_CONCURRENCY) are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Companion benches/*.md (matching the repo convention) capturing the
before/after numbers and the honest interpretation behind the read_prefetch
1->2 tuning and the runtime pool sizing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
bench_tokio_runtime (behind the bench feature) measures the two things
build_runtime changes vs the #[tokio::main] defaults:
A) worker over-subscription — throughput + p50/p99 of an async+CPU workload
with many workers vs core-sized, run under taskset to model a CPU quota.
B) blocking-pool RSS blast radius — peak RSS flooding the blocking pool with
memory-heavy spawn_blocking tasks, default 512 vs bounded.
No Postgres needed. Lets us verify the runtime change empirically before/after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Replace the bare #[tokio::main] with an explicit runtime::Builder so both
pools are sized, logged at startup, and operator-tunable.
#[tokio::main] hides two defaults that misbehave under container limits:
- worker_threads = available_parallelism(), which honours CPU affinity
(sched_getaffinity: cpuset, taskset) but IGNORES the CFS bandwidth quota
(docker --cpus, cgroup v2 cpu.max, v1 cpu.cfs_quota_us). On a 2-core-quota
container on a 64-core host it spawns 64 workers time-slicing 2 cores.
- max_blocking_threads = 512, a multi-GB RSS blast radius for this heavy
spawn_blocking user (thumbnails, transcode, zip, PDF/text extraction,
Argon2 ~19 MB/hash).
New common::runtime module folds the CFS quota back in: effective_parallelism()
= min(available_parallelism, cgroup quota). runtime_pool_sizes() defaults
workers to that and caps the blocking pool at max(32, 8*workers), both
overridable via OXICLOUD_WORKER_THREADS (or TOKIO_WORKER_THREADS) and
OXICLOUD_MAX_BLOCKING_THREADS. The cgroup v1/v2 parsers are pure + unit-tested.
Note: this corrects the premise that "rayon respects the quota, tokio doesn't"
— the image/rayon pools also use available_parallelism(), so they over-spawn
under a CFS quota too; effective_parallelism() is the reusable fix. Unset env
on an uncontended host reproduces the previous worker count exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Edouard Vanbelle