Commit Graph

24 Commits

Author SHA1 Message Date
Diocrafts 761d159a92 feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking
  (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing
- Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[])
- Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends)
- 3-phase store_chunks pipeline:
  Phase 0: batch-check existing chunks (single PG query)
  Phase 1: selective disk read (skip existing chunks entirely)
  Phase 2: parallel upload with buffer_unordered(8)
- CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback
- Transactional manifest + chunk ref-count cascade on remove_reference
- 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.)
- Update deduplication.md to reflect new architecture
2026-04-14 23:17:39 +02:00
Mathieu Boutolleau 3e23d9541b Change PostgreSQL data directory to /var/lib/postgresql/
Starting with PostgreSQL 18, the data directory (PGDATA)
of the Docker Image has changed to be version specific.
For PostgreSQL 18 it is /var/lib/postgresql/18/docker/.

This means that to persist data, mounts and volumes
should target the new location: /var/lib/postgresql/.

Fixes DioCrafts/OxiCloud#281
2026-04-11 21:39:11 +02:00
Edouard Vanbelle badf35f08f chore: remove all executable attributes on non bash files 2026-04-01 23:14:42 +02:00
Nico Kahlert 080b87a277 Add documentation for the helm chart 2026-03-12 13:07:19 +01:00
Diocrafts 06ed0455ce perf: migrate all user/session/auth IDs from VARCHAR(36) to native UUID
- Schema: all ~15 VARCHAR(36) columns → UUID with DEFAULT gen_random_uuid()
- Domain entities: User, Session, DeviceCode, AppPassword, Share → id: Uuid
- DTOs: CurrentUser.id → Uuid (API boundary DTOs keep String for JSON)
- Auth middleware: parse JWT claims.sub (String) → Uuid at boundary
- All repository traits, port traits, service impls updated end-to-end
- Handlers: pass Uuid by value (Copy, 16 bytes) instead of String refs
- Settings chain: updated_by column → Uuid (was text, caused setup crash)
- Removed ~650 lines of String↔Uuid conversion boilerplate
- Eliminates per-request heap allocations for ID cloning
- 16-byte binary comparison vs 36-byte string comparison in all queries
- Native UUID indexing in PostgreSQL (btree on 16 bytes vs 36-char text)

85 files changed, 1090 insertions(+), 1739 deletions(-)
2026-03-07 14:59:32 +01:00
Diocrafts 9f08460027 perf: OnceLock for env var, Arc<CurrentUser> in auth, pre-compute query lowercase
- rate_limit: cache OXICLOUD_TRUST_PROXY_HEADERS in OnceLock<bool> to avoid
  syscall on every request (~500ns → ~1ns)
- auth middleware: insert Arc<CurrentUser> instead of bare CurrentUser;
  all 5 extractors now clone Arc (~1ns) instead of 4 Strings (~60-100ns)
- search_service: pre-compute query.to_lowercase() once before loops,
  eliminating N redundant heap allocations per search
2026-03-07 11:23:56 +01:00
Dionisio f2d35ca792 feat: auto-persist JWT secret, remove setup token requirement
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
2026-03-05 22:12:53 +01:00
Dionisio 2c3dde2d75 chore: remove redundant migrations directory and migrate binary
All content from 003_add_device_codes.sql and 004_add_trigram_indexes.sql
was already absorbed into db/schema.sql (the single source of truth).

The migrations had additional problems:
- Broken numbering (started at 003, missing 001/002)
- 004 used CREATE INDEX CONCURRENTLY which fails inside sqlx transactions
- No production flow ever invoked the migrate binary

Removed: db/migrations/, src/bin/migrate.rs, migrations Cargo feature,
[[bin]] migrate target, and doc/database-migrations.md.
2026-03-02 23:52:37 +01:00
George Wu 2c428678b6 docs: add example.env and document all environment variables
- Create example.env with all 31 environment variables documented
- Update docker-compose.yml to use env_file directive
- Update doc/deployment.md with Quick Start and missing variables
- Add OXICLOUD_BASE_URL and WOPI configuration docs
- Update README.md Docker section with .env setup
- Remove duplicate port mapping (incorporates 4577e56)
- Fix server port default from 8085 to 8086
2026-02-21 18:44:38 -08:00
zjean 1cf69d439d fix dutch language selection 2026-02-21 20:44:05 +00:00
Dionisio Pozo 828da554c6 Rename ADMIN-SETTINGS.md to ADMIN-SETTINGS1.md 2026-02-17 18:24:30 +01:00
Dionisio e071841ec2 docs: update documentation to reflect 100% blob storage model
Rewrite documentation to match the new architecture where all
file metadata lives in PostgreSQL and content is stored as
content-addressed blobs via DedupService.

Updated files:
- internal-architecture.md: complete rewrite — new DB schema,
  blob repos (FolderDb, FileBlobRead/Write, TrashDb), updated
  DI container, service groups, architecture diagram, data flows
- file-system-safety.md: repurposed as storage-safety.md —
  covers PostgreSQL ACID guarantees + DedupService atomic writes
- caching-architecture.md: updated repo references to blob repos,
  removed write-behind cache section, updated upload/download flows
- trash-feature-summary.md: rewritten for soft-delete model
  (is_trashed flag, trash_items VIEW, TrashDbRepository)
- share-integration.md: clarified ShareFsRepository scope,
  updated DI snippet, added blob storage context note
- deduplication.md: updated DI snippet (dedup injected into repos)
- deployment.md: updated feature matrix (file storage requires DB)
- important-delta-sync-implementation.md: updated DI references

Removed legacy references: IdMappingPort, StorageMediator,
WriteBehindCache, FsFileRepository, FsFolderRepository,
TrashFsRepository, folder_ids.json, file_ids.json.
2026-02-14 18:27:30 +01:00
Dionisio 11b7e15319 next todo 2026-02-13 13:22:58 +01:00
Dionisio fb65d1976b fix: resolve file management operations not working (#83)
- Fix rename: context menu was nullifying target reference before rename dialog could use it
- Fix delete files/folders: auth extractors were mandatory, causing 401 when auth not configured
- Fix view-file: async fetch race condition with context menu cleanup
- Fix orphaned ID mappings on file deletion
- Fix Authorization: Bearer null headers sent without token
- Add OptionalUserId and OptionalAuthUser infallible extractors
2026-02-13 08:54:51 +01:00
Dionisio 8123406ab9 fixing db error and upgrading technical documentation 2026-02-12 22:29:35 +01:00
Diocrafts 5bd505ccd7 modernizing frontend 2026-02-08 22:44:42 +01:00
Dionisio 8f2b0a354c big refactoring 2026-02-03 17:59:04 +01:00
DioCrafts 52d8250d51 adding card dav and cald dav 2025-04-13 01:04:04 +02:00
DioCrafts 5d67bc4d84 fix several bugs 2025-04-10 01:43:25 +02:00
DioCrafts 8f1d213526 improve postgresql performance 2025-04-09 00:21:20 +02:00
DioCrafts ba3ef2a530 adding webdav features 2025-04-04 21:31:41 +02:00
DioCrafts 01c81bfb1a adding features 2025-04-02 23:14:12 +02:00
dionidev e79ca8304b adding comments, moving technical documentation, improve Dockerfile, delete unuseful files 2025-03-30 14:17:09 +00:00
DioCrafts 1c9dca662a adding README screenshot 2025-03-23 23:13:02 +01:00