The admin panel had no repair toggle wired to anything but a hardcoded
name list naming the two refcount tenants, so `thumb_derived_import` and
`thumb_attached_import` could not be run in repair mode from the UI at
all despite supporting it. And nothing in the job list said what any
given job does or whether clicking Run on production writes anything.
Three defaulted methods on `JobHandler` and `RecoverableJobHandler`:
fn description(&self) -> &'static str
fn mutates(&self) -> Mutates // Never | Always | OnRepairOnly
fn repair_description(&self) -> Option<&'static str>
`RecoverableAdapter` forwards them — the registry only holds
`dyn JobHandler`, so a tenant's metadata is invisible otherwise, and
falling back to the defaults would report every recoverable job as
read-only, including the ones that delete files.
Three values rather than a boolean because a job can be read-only by
default and destructive under `?repair=true`; a boolean answers wrongly
for one of its two modes, and `false` on something that unlinks files is
the dangerous direction to be wrong in. `repair_description` returning
`Option` collapses "does it repair" and "what does repair do" into one
method: presence gates the toggle, content is the confirmation text —
which the frontend cannot invent, since correcting a counter and
deleting sidecars are not the same warning.
`OnRepairOnly` with no `repair_description` is rejected at registration:
it claims to mutate only under a flag it does not support.
All 17 registered jobs declare all three. The panel now renders the
description under each name, badges read-only jobs, confirms before a
plain run of a mutating one, and offers the repair variant off the
backend flag instead of the name list.
Descriptions are English in the trait, next to the behaviour: one in
`locales/*.json` rots invisibly the moment a job changes, and a
translator cannot know what `manifests_consistency` reconciles. i18n can
layer on later keyed by job name with these as the fallback.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Benchmark-gated round (benches/ROUND12.md; every change ships with a
BEFORE/AFTER harness + equivalence gates, one candidate rejected by its
own bench):
DB / query shapes (bench_round12_queries):
- NC sharee search: username-only projection instead of the 21-column row
(incl. the <=512 KiB avatar) per match, + gin_trgm_ops indexes on
auth.users for the leading-wildcard ILIKE (4.98x; 54.7x with index).
- Password login: delete the redundant full-row update_user — create_session
already stamps last_login_at in its own txn (4.45x per login).
- Email-verified stamp: narrow conditional UPDATE (8.9x); OIDC repeat login
now compares profile state in memory and issues ZERO queries when nothing
changed (was: full 17-column rewrite per login).
- Refresh rotation: revoke+insert+stamp fused into one transaction via new
rotate_session port method (1.18x).
- WOPI CheckFileInfo / authorize_wopi_access: require(Read) + get_file +
check(Update) overlapped with tokio::join!, original result precedence
(cold 1.34x).
- Upload quota gate: user-envelope + drive-cap checks fused into ONE
round-trip (check_upload_quotas) — the NC chunked PUT pays this per
chunk (1.81x, 2 -> 1 queries/chunk); shared verdict evaluators keep
error shapes byte-identical.
CPU / allocs (bench_round12_micro):
- sized_json: pre-sized listing serialization replacing axum Json's 128 B
seed + doubling-realloc chain on files/folder-resources/photos/search
responses (1.40x, 13 -> 2 allocs per 500-row page; byte-identical).
- Security headers: 4 SetResponseHeaderLayer folded into the CSP middleware
pass (5 layers -> 1; 1.43x per request, -26 allocs; header set gated
byte-identical incl. 304s).
- Media capture-metadata: single-read extraction — nom-exif now parses the
buffer kamadak already read (zero-copy Bytes) and videos open once with a
kind() dispatch; per-image opens 2-3 -> 1 (1.44x warm geomean, 1.6-3.2x
cold cache; extraction outputs gated identical incl. the MIME-mislabel
track fallback).
- Chunked-upload session ops: owner gate folded into the operation's own
DashMap lookup + stack-encoded uuid compare (5 -> 3 lookups, -2 allocs,
1.28x per chunk).
Blob cache (bench_blob_cache_index + round-3 regression guard):
- CachedBlobBackend index: tokio::sync::Mutex<LruCache> -> moka::sync::Cache
with byte weigher. The mutex serialized every cached chunk read and scaled
NEGATIVELY (2.08 -> 1.07 Mops/s from 1 -> 2 readers); moka probes are
lock-free (2.17x at K=2). Byte budget now enforced by moka (manual
current_size + collect_evictions machinery deleted); eviction listener
unlinks size-evicted files only (Replaced entries keep their file —
gated). Single-flight miss gate unchanged (16 concurrent misses -> 1
fetch re-verified via the round-3 harness).
- put_blob now populates the cache BEFORE the inner backend consumes the
source file (the old order failed 100% of the time — local renames,
S3/Azure delete the source — so the first read after a whole-file put
re-downloaded from the remote); inner-put failure invalidates the entry.
Frontend (vitest gates):
- List-view thumbnails request the 150px icon rendition instead of 400px
preview into a 40px slot (~7.1x fewer pixels, ~4-5x fewer bytes per
thumbnail across list views); grid keeps preview.
Rejected by its own bench (kept as evidence in bench_round12_micro §2):
- Single-pass compression predicate: the monomorphized And-chain already
costs ~4.6 ns / 0 allocs total; the fused node measured within noise.
New migration: 20260719000000_users_search_trgm.sql (trgm indexes).
Deferred with prepared design: grouped file/grid view virtualization
(single-VirtualRows flatten, the photos pattern) — next round's headline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BfidAJD5AHw23jtvBUNamB
this fix https://github.com/AtalayaLabs/OxiCloud/issues/607
which was introduced by commit 12dc648cff
when a user does activity in a drive, admin can invalidate cache via the internal
call /api/admin/internal/trigger-sweep
this permit end 2 end test to validte immediately that used_bytes corresponds to the expected result
Every change is benchmark-verified (harness + before/after numbers in
benches/, measured on this branch; reproduction commands in each doc):
DAV / sync-client hot paths
- PROPFIND dead-properties: one = ANY($1) query per 500-child page instead
of one sequential query per child, and indexable `=` predicates instead
of IS NOT DISTINCT FROM (seq scans). 2,000-child folder: 1.07-4.54 s of
DB chatter -> 4-6 ms (258-773x). Applied to native + NC PROPFIND and
both NC REPORT handlers. [benches/DEAD-PROPS.md]
- Folder paging: keyset cursor (name > $last) + new partial index
(folder_id, name) replaces LIMIT/OFFSET full-folder rescan per page.
Full 20k-file walk: 1266 ms -> 77 ms (16.5x). New migration
20260917000000. [benches/PROPFIND-PAGING.md]
- NC chroot / default-drive resolution: moka caches (30 s TTL, explicit
invalidation on drive mutations) for find_default_for_user and the
markerless chroot FolderDto. 2 uncached queries + 2 pool checkouts per
NC/WebDAV/WOPI request -> sub-us moka hit (p50 0.7-3.6 ms -> ~1 us).
[benches/CHROOT-CACHE.md]
- Quota: PROPFINDs whose prop list never names a quota prop skip the
2-query resolution entirely (wants_quota()); the remaining lookups read
2 columns instead of the full auth.users row with its <=512 KiB avatar
(11-16x, p50 3.4 ms -> 0.29 ms). Same narrow read now gates every
upload quota check. [benches/QUOTA-PATH.md]
CPU on the request path
- ZIP exports (folder download, share ZIP, batch download): entries whose
MIME says already-compressed (JPEG/MP4/zip/pdf/...) are Stored instead
of Deflate - deflate ran inline on the tokio writer task at ~41 MB/s
for ~0% size gain. Mixed media corpus: 4.31x wall and CPU, archive size
unchanged. Shared predicate in common::mime_detect. [benches/ZIP-MEDIA.md]
- Compression layers: tower-http's default maps to Brotli QUALITY 11
(verified in brotli-8.0.2 source and empirically: 90 ms per 64 KiB JSON
response, 1.3 s per 700 KiB bundle). Both layers pinned to Precise(4):
99x less CPU for ~15% more bytes. SPA assets are now precompressed at
build time (scripts/precompress.mjs, 77% smaller) and served via
ServeDir::precompressed_br/gzip: 2016x less per-request work, and
clients get the better q11 bytes. [benches/STATIC-PRECOMPRESSED.md]
Batched / cached backend paths [benches/NPLUS1-AND-CACHES.md]
- Content-search ReBAC re-verification: new
AuthorizationEngine::check_files_read_batch (default = old loop;
PgAclEngine override batches drive resolution + reuses role cache).
200 sequential point SELECTs per search -> 1-2 queries.
- Batch-ZIP subtree downloads: drop per-file re-authz + per-file Recent
recording (2 writes/file) for subtree entries already authorized at the
root - mirrors the native folder-download path. ~6,000 statements
removed from a 2,000-file archive.
- CDC chunk manifests: immutable by content address, now moka-cached
(weight-bounded 32 MiB, 60 s TTL, positive-only, invalidated on delete)
- removes one manifest query (p50 0.44-4.4 ms) from every stream,
range and full blob read.
- People tab: grouped COUNT + batched cover lookup instead of dragging
every face row with its 2 KiB embedding (10k faces: 30.4 ms & 21 MB ->
3.8 ms & 1.3 KB, 8.1x); merge() is one set-based UPDATE.
[benches/PEOPLE-LIST.md]
- Photos timeline cursor: raw timestamptz comparison instead of
EXTRACT(EPOCH ...) wrapper + IS NULL OR disjunction - cursor is an
index boundary again, deep scroll stops re-scanning skipped rows.
- Public share landing: one atomic UPDATE ... access_count + 1 (was
SELECT + full-row write-back: racy, lost updates, clobbered concurrent
owner edits) - 3 round-trips -> 2 per visit.
- move_to_trash: dead full-entity SELECT feeding a documented no-op
removed from both branches; dead fields dropped from TrashService.
- NFC normalization: is_nfc_quick fast path skips the decompose/recompose
state machine for the ~100% already-NFC case (every row loaded from PG).
Frontend
- Large folders paint after page one (~200 items) via fetchFolderListing's
new onPage hook instead of waiting for every sequential page.
- Tested-and-reverted (kept for the record): cached Intl.Collator for name
sorts - vitest showed it 2x SLOWER than V8's argument-less localeCompare
fast path (5.6 ms vs 12.1 ms / 5k names). Sort order untouched.
New bench harnesses under examples/ (bench feature): zip_media,
dead_props, chroot_cache, quota_path, people_list, propfind_paging,
static_precompress.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CBK1RdtzyP6759Muqe1K1w
- quota per drive
- add 2 internal API endpoints to test purpose
disabled by default, enable it via `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
this enable:
/api/admin/internal/trigger-sweep
to sweep the trash and recalculated quota
/api/admin/internal/trigger-gc
to garbage orphan blobs
use full for end to end tests and validate lifecycles
After every upload, maybe_update_storage_usage spawned a full
`SUM(size) OVER all the user's non-trashed files` to refresh
auth.users.storage_used_bytes — O(N) in the user's file count per upload,
i.e. O(N²) for a bulk upload. (The covering index makes it index-only but
still scans N rows.)
Replace it with an O(1) incremental `storage_used_bytes += size`, keyed by the
file's owner_id (dropping the brittle "My Folder - <user>" path-parsing hack).
Deletes/trash never decremented this value — they already rely on the periodic
reconciliation sweep — so the model is unchanged: the sweep remains the
correctness backstop for every mutation, and the counter is clamped at 0.
Both stay fire-and-forget on a background task, off the upload's latency path.
Benchmarked (per-call, vs the user's existing file count N):
N=1k: full-SUM 202us vs incremental 123us
N=10k: full-SUM 1185us vs incremental 113us (10x)
N=50k: full-SUM 5397us vs incremental 114us (47x — incremental is flat O(1))
Bulk upload of 10k files (insert + usage update each):
full-SUM (O(N²)) 10.37s -> incremental (O(N)) 4.89s (>2x, diverges with scale)
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
GET /api/auth/me ran a synchronous O(N) SUM(size) over all the user's
files plus an unconditional UPDATE of auth.users on every call — one of
the most frequently hit endpoints — adding per-request latency, DB write
load, dead tuples and WAL even when nothing changed.
- /api/auth/me now serves the cached storage_used_bytes column instead of
recomputing it inline.
- New StorageUsageService::start_reconciliation_job runs a periodic sweep
on the maintenance pool that keeps the cached value current for every
mutation (uploads, deletes, trash), so freshness no longer depends on
hitting /me. Interval via OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default
600s, floored at 30s; first sweep deferred one interval to avoid boot load).
- update_storage_usage only writes when the value actually changes
(IS DISTINCT FROM), so the sweep produces no dead tuple / WAL on no-ops.
- New covering partial index idx_files_user_size_active makes the usage
SUM an index-only scan instead of a heap scan over all the user's files.
Also collapse the same pre-existing clippy collapsible_else_if in
carddav_handler that blocks the -D warnings gate on this base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
reflect OIDC
schema migration, User entity additions, and three defense-in-depth gaps closed, with all 280 unit tests and 13 Hurl files green
infrastructure (lettre + EmailSender port).
- Migration migrations/20260612000003_users_username_email_login.sql — adds nullable given_name/family_name columns to auth.users.
- User entity (src/domain/entities/user.rs) — has_login_credential() placeholder-check encapsulation, set_username revalidating setter, given/family-name fields + getters/setters, validate_username widened 32→254 and now accepts email shape.
from_data_full extended with two new params; all 7 callsites in user_pg_repository.rs updated.
- Schema-side legacy guards (src/application/services/auth_application_service.rs) — bumped the duplicated 32-char check in setup_create_admin and admin_create_user to 254 to match.
- Gap #1 (subject_group_service.rs) — add_member now rejects external candidates with an audit-logged AccessDenied. Service gained an Arc<UserPgRepository> field, wired through DI. New integration test test_external_user_cannot_be_added_as_member.
- Gap #2 (user_repository.rs + auth_ports.rs + user_pg_repository.rs) — list_users/search_users gained an include_external: bool param defaulting effectively to false everywhere internal-user-facing. auth_application_service exposes a new
list_users_including_external for the admin surface.
- Gap #3 (pg_acl_engine.rs) — expand_user now SELECTs is_external and skips INTERNAL_GROUP_ID for externals; defaults to is_external=true on missing user to fail closed.
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
- Add type aliases (FileRow, FolderRow, FolderRowPaginated, FolderRowOptUser) to reduce type complexity
- Simplify redundant closures in app_password_handler and webdav_handler
- Remove needless borrow in auth_handler
- Collapse nested if/let chains in login_lockout, webdav_lock, auth, rate_limit
- Box LockEntry in acquire() Err variant to fix large enum variant warning
- Rename DeviceCodeStatus::from_str to parse to avoid should_implement_trait lint
- Add #[allow(clippy::too_many_arguments)] and #[allow(clippy::result_unit_err)] where appropriate
- Convert integration_tests from cargo feature to custom cfg attribute
- Add check-cfg lint config in Cargo.toml for integration_tests cfg
Remove async-trait dependency and use native Rust async fn in traits.
Replace Arc<dyn Trait> with Arc<ConcreteType> throughout the codebase
to enable monomorphization and eliminate dynamic dispatch overhead.
Key changes:
- Remove write-behind cache (no implementation existed)
- Fix should_transcode static method call
- Use ContactStorageAdapter directly instead of dyn AddressBookUseCase
- Clean up unused trait imports across services and DI
https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
The original implementation tried to find user's home folder via
file_repository.list_files(None), but this only returns files (not folders)
and folders are in a separate table. This resulted in storage always
showing 0 bytes.
Fixed by using a direct SQL query to sum file sizes by user_id from
storage.files table, which is more efficient (O(1) vs recursive) and
correctly calculates storage for all files owned by the user.
Standardize code formatting across all 173 Rust source files
using rustfmt. No functional changes - purely cosmetic.
This establishes a consistent code style baseline for the
project going forward.
- Auto-apply schema.sql when database tables don't exist (embedded in binary)
- Handle missing tables gracefully during admin registration (treat as fresh install)
- Fix docker-compose depends_on to wait for postgres healthcheck
- Rename personal folder from 'Mi Carpeta' to 'My Folder' with backward compat
- Translate remaining Spanish messages to English
- Remove Serialize/Deserialize from File, Folder, Session, User, Contact entities
- Create contact_persistence_dto.rs for JSONB persistence in infrastructure layer
- Update contact_pg_repository to use persistence DTOs
- Fix dependency on zip crate (downgrade from 7.2.0 to 2.1.0)
- Fix unused variable warnings in main.rs
- Move PathService import from domain to infrastructure
- Add missing fields to CoreServices and RepositoryServices
- Create proper service initialization in main.rs
Clean Architecture improvements:
- Domain layer no longer depends on serde framework
- Persistence concerns isolated to infrastructure layer
- TokenClaims in auth_service.rs is only exception (required for JWT)
Edouard Vanbelle