feat(OXICLOUD_DIRECT_PUT_MAX_BYTES): add a security limit on direct PUT
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
This commit is contained in:
+3
-2
@@ -11,8 +11,9 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
|
||||
| `OXICLOUD_SERVER_PORT` | `8086` | Server port |
|
||||
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
||||
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Maximum upload size in bytes (10 GB on 64-bit, 1 GB on 32-bit) |
|
||||
| `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Distinct from `OXICLOUD_MAX_UPLOAD_SIZE`, which is the whole-file cap. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Whole-file size ceiling, in bytes (10 GB on 64-bit, 1 GB on 32-bit). Applies to BOTH direct PUTs (per-request body) and chunked uploads (declared `total_size`, checked upfront at session creation). |
|
||||
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | `1073741824` | Per-request cap for non-chunked PUT bodies, in bytes (1 GiB). Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol (resumable on failure). See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||
| `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Per-chunk cap, independent of `OXICLOUD_MAX_UPLOAD_SIZE` (whole-file cap). See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||
| `OXICLOUD_UPLOAD_TMPDIR` | (OS temp dir) | Spool directory for non-chunked PUT bodies (`/api/files/upload`, WebDAV PUT). Point at a real-disk path on the same FS as `.blobs/` to avoid tmpfs OOMKill and make blob promotion an atomic rename. See [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Same-FS / NVMe placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
|
||||
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
|
||||
|
||||
@@ -50,60 +50,85 @@ Two practical consequences:
|
||||
|
||||
Three independent caps control how large an upload OxiCloud will
|
||||
accept. Pick them with disk and tmpfs sizing in mind: the spool/chunk
|
||||
directories must be able to hold the worst case (max cap × concurrent
|
||||
directories must be able to hold the worst case (cap × concurrent
|
||||
uploads).
|
||||
|
||||
| Variable | Default | What it caps |
|
||||
|---|---|---|
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | Total file size — whether uploaded as one `PUT` or assembled from many chunks. The hard ceiling on any single file. |
|
||||
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | A single chunked-PUT request body (`PATCH /api/uploads/{id}` or `PUT /dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file uploaded in 100 MB chunks needs ~50 PUTs, each bounded by this. |
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE` (again, for the PUT path) | 10 GB | A single non-chunked PUT body. Same env var as the whole-file cap because for non-chunked uploads they're equivalent. |
|
||||
| Variable | Default | What it caps | When it fires |
|
||||
|---|---|---|---|
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. |
|
||||
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `POST /api/files/upload`, `PUT /webdav/...`, and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
|
||||
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. |
|
||||
|
||||
### Why these matter for tmpfs sizing
|
||||
### Recommendation: prefer chunked uploads for large files
|
||||
|
||||
The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below
|
||||
`OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric.
|
||||
Files between those two caps can only succeed via the chunked
|
||||
protocol. Three reasons to keep them that way:
|
||||
|
||||
- **Resilience.** A direct PUT at 95 % of 5 GB that drops loses
|
||||
everything. The same drop on a chunked upload loses one ~5 MB
|
||||
chunk; the client retries that chunk and continues.
|
||||
- **Memory + disk pressure.** Direct PUT spools the full body to
|
||||
disk per request. Ten concurrent 5 GB direct PUTs use up to 50 GB
|
||||
of transient spool disk. Chunked spreads each upload across many
|
||||
small PATCHes; per-request resource use stays bounded by
|
||||
`OXICLOUD_CHUNK_MAX_BYTES`.
|
||||
- **Convention.** NextCloud desktop and the OxiCloud web UI already
|
||||
switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
|
||||
|
||||
### Why caps matter for tmpfs sizing
|
||||
|
||||
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded to one
|
||||
HTTP frame (~64 KB) per request regardless of the caps. **Disk space**,
|
||||
however, scales with the caps:
|
||||
|
||||
- **Direct PUT** (single-file): each in-flight upload spools the full
|
||||
body to disk under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst
|
||||
case disk = `OXICLOUD_MAX_UPLOAD_SIZE × concurrent_PUTs`.
|
||||
- **Chunked upload**: each in-flight session accumulates chunks under
|
||||
`OXICLOUD_CHUNK_DIR`, then assembles them into a single temp file
|
||||
before promotion. Worst case disk per session = **2 × file_size**
|
||||
(chunks + assembled file); total disk =
|
||||
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_sessions`.
|
||||
- **Direct PUT**: each in-flight upload spools the full body to disk
|
||||
under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst case disk =
|
||||
`OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs`.
|
||||
- **Chunked upload**: each in-flight session accumulates chunks
|
||||
under `OXICLOUD_CHUNK_DIR`, then assembles them into a single temp
|
||||
file before promotion. Worst case disk per session = **2 ×
|
||||
file_size** (chunks + assembled file); total disk =
|
||||
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
|
||||
|
||||
The chunked formula uses `OXICLOUD_MAX_UPLOAD_SIZE` because that's
|
||||
what bounds the declared `total_size` at session creation. The
|
||||
direct-PUT formula uses the smaller `OXICLOUD_DIRECT_PUT_MAX_BYTES`
|
||||
since that's what bounds each direct PUT body.
|
||||
|
||||
### Sizing examples
|
||||
|
||||
A 4 GB tmpfs serving a small team (~5 concurrent uploads):
|
||||
A 4 GB tmpfs serving a small team (5 concurrent direct PUTs OR 5
|
||||
concurrent chunked sessions):
|
||||
|
||||
| Setting | Disk worst case | Safe on 4 GB tmpfs? |
|
||||
|---|---|---|
|
||||
| `MAX_UPLOAD=10 GB`, no `CHUNK_MAX` tweak | 50 GB direct, 100 GB chunked | ❌ no — single upload OOMs the tmpfs |
|
||||
| `MAX_UPLOAD=500 MB`, `CHUNK_MAX=50 MB` | 2.5 GB direct, 5 GB chunked | ⚠ direct fits, chunked overflows |
|
||||
| `MAX_UPLOAD=300 MB`, `CHUNK_MAX=30 MB` | 1.5 GB direct, 3 GB chunked | ✅ both fit |
|
||||
| Settings | Direct-PUT worst case | Chunked worst case | Safe on 4 GB tmpfs? |
|
||||
|---|---|---|---|
|
||||
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=100 MB | 5 GiB (5 × 1 GiB) | 100 GB (5 × 2 × 10 GB) | ❌ chunked overflows |
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=100 MB, `OXICLOUD_CHUNK_MAX_BYTES`=20 MB | 500 MB | 5 GB | ⚠ direct PUT fits, chunked still overflows |
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE`=300 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=50 MB, `OXICLOUD_CHUNK_MAX_BYTES`=10 MB | 250 MB | 3 GB | ✅ both fit |
|
||||
|
||||
A real-disk volume (cheap, large):
|
||||
|
||||
| Setting | Disk worst case | Comment |
|
||||
|---|---|---|
|
||||
| `MAX_UPLOAD=10 GB`, `CHUNK_MAX=100 MB` (defaults) | 100 GB chunked worst case | Fine on a 200+ GB volume; almost any real-disk setup |
|
||||
| `MAX_UPLOAD=100 GB`, `CHUNK_MAX=500 MB` | 1 TB chunked worst case | Plausible for video archives; needs a dedicated upload volume |
|
||||
| Settings | Direct-PUT worst case | Chunked worst case | Comment |
|
||||
|---|---|---|---|
|
||||
| Defaults (see row above) | 5 GiB | 100 GB | Fine on a 200+ GB volume; almost any real-disk setup |
|
||||
| `OXICLOUD_MAX_UPLOAD_SIZE`=100 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=5 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=500 MB | 25 GiB | 1 TB | Plausible for video archives; needs a dedicated upload volume |
|
||||
|
||||
### Choosing tmpfs vs real disk
|
||||
|
||||
| Constraint | Choice |
|
||||
|---|---|
|
||||
| Upload caps × concurrent users ≤ free RAM × 0.5 | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
|
||||
| Upload caps × concurrent users > free RAM × 0.5 | **real disk** — same FS as `.blobs/` ideal |
|
||||
| `OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs + 2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions ≤ free RAM × 0.5` | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
|
||||
| Worst case exceeds half free RAM | **real disk** — same filesystem as `.blobs/` ideal |
|
||||
| Container with cgroup memory limit | **real disk** — tmpfs spool counts against the cgroup limit and triggers OOMKill |
|
||||
| Multi-GB uploads expected | **real disk** — even small concurrency on tmpfs runs out of space |
|
||||
| Small-file workload only (≤ 50 MB), high concurrency | tmpfs gives a noticeable intake speedup |
|
||||
|
||||
The defaults (`MAX_UPLOAD=10 GB`, `CHUNK_MAX=100 MB`) assume **real
|
||||
disk**. Don't run the defaults against tmpfs unless you've sized it
|
||||
for the worst case.
|
||||
The defaults (`OXICLOUD_MAX_UPLOAD_SIZE`=10 GB,
|
||||
`OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB,
|
||||
`OXICLOUD_CHUNK_MAX_BYTES`=100 MB) assume **real disk**. Don't run
|
||||
the defaults against tmpfs unless you've sized it for the worst case.
|
||||
|
||||
## TL;DR
|
||||
|
||||
|
||||
+38
-8
@@ -31,18 +31,48 @@ OXICLOUD_SERVER_HOST=127.0.0.1
|
||||
# Example: https://cloud.example.com
|
||||
#OXICLOUD_BASE_URL=https://cloud.example.com
|
||||
|
||||
# Maximum upload size in bytes (default: 10 GB on 64-bit)
|
||||
# ── Upload size caps ──────────────────────────────────────────────────
|
||||
# See docs/config/storage-fine-tuning.md for sizing guidance.
|
||||
|
||||
# Whole-file size ceiling. Applies to BOTH direct PUTs (per-request body)
|
||||
# and chunked uploads (declared `total_size` at session creation — checked
|
||||
# upfront so oversized requests never accumulate chunks on disk).
|
||||
# Default: 10 GB on 64-bit, 1 GB on 32-bit.
|
||||
#OXICLOUD_MAX_UPLOAD_SIZE=10737418240
|
||||
|
||||
# Directory for upload spool temp files. Uploads are streamed to a temp file
|
||||
# before deduplication. By default this uses the OS temp dir ($TMPDIR / /tmp),
|
||||
# which in many containers is tmpfs (RAM) — writing a large upload there fills
|
||||
# page-cache that counts against the cgroup memory limit and can OOMKill the
|
||||
# process. Point this at a real-disk path (same filesystem as the storage
|
||||
# backend is ideal) to keep the upload footprint off RAM. Leave unset to use
|
||||
# the OS default.
|
||||
# Per-request cap for non-chunked PUT bodies (`POST /api/files/upload`,
|
||||
# `PUT /webdav/...`, `PUT /remote.php/dav/files/...`). Set below
|
||||
# MAX_UPLOAD_SIZE so files larger than this are pushed onto the chunked
|
||||
# protocol (resumable on failure, bounded per-request by CHUNK_MAX_BYTES).
|
||||
# Default: 1 GiB.
|
||||
#OXICLOUD_DIRECT_PUT_MAX_BYTES=1073741824
|
||||
|
||||
# Per-chunk cap for a single chunked-upload PUT (PATCH /api/uploads/{id}
|
||||
# or PUT /dav/uploads/.../chunk). NC desktop and the OxiCloud frontend
|
||||
# split large files into chunks of this size or smaller, so this knob
|
||||
# tightly bounds the worst-case per-request memory/disk footprint
|
||||
# independently of the whole-file cap. Default: 100 MB.
|
||||
#OXICLOUD_CHUNK_MAX_BYTES=104857600
|
||||
|
||||
# ── Upload spool directories ─────────────────────────────────────────
|
||||
# Where in-flight uploads land BEFORE being promoted into final blob
|
||||
# storage. Same-filesystem placement (with `.blobs/`) makes the
|
||||
# promotion an atomic rename(2); NVMe placement speeds up intake.
|
||||
# See docs/config/storage-fine-tuning.md for layout examples.
|
||||
|
||||
# Directory for non-chunked PUT spool tempfiles. Default: OS temp dir
|
||||
# ($TMPDIR / /tmp), often tmpfs (RAM) in containers — writing a large
|
||||
# upload there fills page-cache that counts against the cgroup memory
|
||||
# limit and can OOMKill the process. Point this at a real-disk path
|
||||
# (same FS as the storage backend is ideal).
|
||||
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
|
||||
|
||||
# Root directory for chunked-upload sessions (REST + NextCloud chunked
|
||||
# share this root). Default: {STORAGE_PATH}/.uploads. Pointing this at
|
||||
# NVMe accelerates the chunk-write + assembly loop; pointing it at the
|
||||
# same FS as `.blobs/` makes blob promotion atomic.
|
||||
#OXICLOUD_CHUNK_DIR=/var/lib/oxicloud/.uploads
|
||||
|
||||
# How often (seconds) the background sweep reconciles each user's cached
|
||||
# storage usage with the real sum of their files (default: 600 = 10 min).
|
||||
# GET /api/auth/me serves the cached value instead of recomputing per request;
|
||||
|
||||
@@ -218,6 +218,16 @@ pub struct StorageConfig {
|
||||
/// be far tighter than the whole-file cap and prevents one HTTP request
|
||||
/// from monopolising server memory or disk. Env: `OXICLOUD_CHUNK_MAX_BYTES`.
|
||||
pub chunk_max_bytes: usize,
|
||||
/// Maximum size of a single non-chunked PUT body, in bytes (default:
|
||||
/// 1 GiB). Set below `max_upload_size` so files larger than this are
|
||||
/// pushed onto the chunked-upload protocol (`/api/uploads/…` or
|
||||
/// `/dav/uploads/…`) — which is resilient to mid-transfer failures,
|
||||
/// resumable, and bounded per-request by `chunk_max_bytes`. Without
|
||||
/// this cap a 10 GB direct PUT spools 10 GB to disk in a single
|
||||
/// request; a connection drop at 95 % loses everything. The server
|
||||
/// returns 413 with a "use chunked upload" hint when a direct PUT
|
||||
/// exceeds this cap. Env: `OXICLOUD_DIRECT_PUT_MAX_BYTES`.
|
||||
pub direct_put_max_bytes: usize,
|
||||
/// Directory for upload spool temp files. When `Some`, large uploads are
|
||||
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
|
||||
/// containers, where the spool's page-cache counts against the cgroup
|
||||
@@ -376,6 +386,7 @@ impl Default for StorageConfig {
|
||||
trash_retention_days: 30, // 30 days
|
||||
max_upload_size: MAX_UPLOAD_SIZE,
|
||||
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
|
||||
direct_put_max_bytes: 1024 * 1024 * 1024, // 1 GiB — pushes larger uploads onto the chunked protocol
|
||||
upload_temp_dir: None,
|
||||
chunk_dir: None,
|
||||
usage_reconcile_secs: 600, // 10 minutes
|
||||
@@ -1247,6 +1258,12 @@ impl AppConfig {
|
||||
{
|
||||
config.storage.chunk_max_bytes = val;
|
||||
}
|
||||
if let Ok(direct_max) =
|
||||
env::var("OXICLOUD_DIRECT_PUT_MAX_BYTES").map(|v| v.parse::<usize>())
|
||||
&& let Ok(val) = direct_max
|
||||
{
|
||||
config.storage.direct_put_max_bytes = val;
|
||||
}
|
||||
|
||||
// Upload spool directory — keep large upload temp files off tmpfs/RAM
|
||||
// (otherwise their page-cache counts against the cgroup memory limit).
|
||||
|
||||
@@ -162,6 +162,33 @@ impl ChunkedUploadHandler {
|
||||
.into_response();
|
||||
}
|
||||
|
||||
// ── Whole-file cap ──────────────────────────────────────────
|
||||
// Reject upfront, before any chunk is uploaded — wasting
|
||||
// bandwidth + server disk on an upload that's going to be
|
||||
// rejected at /complete is the worst-of-both-worlds outcome.
|
||||
// `max_upload_size` is the same ceiling that bounds direct
|
||||
// PUTs (per-byte during streaming there; declared per-session
|
||||
// here). When quotas are disabled, this is the only whole-file
|
||||
// limit for chunked uploads — without it a hostile client
|
||||
// could declare `total_size: 1 TB` and accumulate chunks
|
||||
// until disk fills.
|
||||
let max_upload = state.core.config.storage.max_upload_size as u64;
|
||||
if request.total_size > max_upload {
|
||||
tracing::warn!(
|
||||
"⛔ CHUNKED UPLOAD REJECTED (total_size cap): user={}, file={}, declared={}, max={}",
|
||||
auth_user.username,
|
||||
request.filename,
|
||||
request.total_size,
|
||||
max_upload
|
||||
);
|
||||
return AppError::payload_too_large(format!(
|
||||
"Declared total_size {} exceeds the server's `max_upload_size` cap ({} bytes). \
|
||||
Raise OXICLOUD_MAX_UPLOAD_SIZE on the server if larger uploads are expected.",
|
||||
request.total_size, max_upload
|
||||
))
|
||||
.into_response();
|
||||
}
|
||||
|
||||
// ── Permission pre-check: caller must have Create on the target
|
||||
// folder BEFORE we allocate a session and accept chunks. The
|
||||
// upload service re-checks at finalize time, but failing here
|
||||
|
||||
@@ -927,8 +927,10 @@ async fn handle_put(
|
||||
// another user — acceptable risk since PathResolver should always
|
||||
// be enabled in production)
|
||||
|
||||
// Hard upload size limit from config
|
||||
let max_upload = state.core.config.storage.max_upload_size;
|
||||
// Direct PUT cap — see `nextcloud/webdav_handler::handle_put` for
|
||||
// the reasoning. Files above `direct_put_max_bytes` must go through
|
||||
// the chunked-upload protocol (`/api/uploads/…`) which is resumable.
|
||||
let max_upload = state.core.config.storage.direct_put_max_bytes;
|
||||
|
||||
// Extract content type before consuming the request
|
||||
let content_type = req
|
||||
|
||||
@@ -596,7 +596,14 @@ async fn handle_put(
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|v| v.parse::<i64>().ok());
|
||||
|
||||
let max_upload = state.core.config.storage.max_upload_size;
|
||||
// ── Direct PUT cap ───────────────────────────────────────────────
|
||||
// We use `direct_put_max_bytes` (default 1 GiB), not `max_upload_size`
|
||||
// (default 10 GB). Larger files must come through the chunked upload
|
||||
// protocol (`/dav/uploads/...`) which is resumable on failure and
|
||||
// bounded per-request by `chunk_max_bytes`. Trying to stream a
|
||||
// multi-GB body through a single PUT is a footgun: a connection drop
|
||||
// at 95 % loses everything.
|
||||
let max_upload = state.core.config.storage.direct_put_max_bytes;
|
||||
|
||||
// Stream the body to a temp file + incremental hash — never buffer the
|
||||
// full upload in RAM. The old `body::to_bytes` path loaded the entire
|
||||
|
||||
@@ -70,7 +70,10 @@ pub async fn spool_body_to_temp(
|
||||
drop(file);
|
||||
let _ = tokio::fs::remove_file(&temp_path).await;
|
||||
return Err(AppError::payload_too_large(format!(
|
||||
"Upload exceeds maximum size of {max_upload} bytes"
|
||||
"Upload body exceeds the direct-PUT cap ({max_upload} bytes). \
|
||||
Use the chunked-upload protocol (REST: `/api/uploads/...`, \
|
||||
NextCloud: `/remote.php/dav/uploads/...`) for files larger than this. \
|
||||
Chunked uploads are resumable on transient failure."
|
||||
)));
|
||||
}
|
||||
hasher.update(chunk);
|
||||
|
||||
@@ -150,6 +150,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
// hardest to spot from chunked-upload tests.
|
||||
tracing::info!(
|
||||
max_upload_size_mb = config.storage.max_upload_size / (1024 * 1024),
|
||||
direct_put_max_bytes_mb = config.storage.direct_put_max_bytes / (1024 * 1024),
|
||||
chunk_max_bytes_mb = config.storage.chunk_max_bytes / (1024 * 1024),
|
||||
"Upload limits loaded from config"
|
||||
);
|
||||
|
||||
@@ -563,3 +563,120 @@ POST {{base_url}}/api/uploads/{{upload_id_nobody}}/complete
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP 201
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════
|
||||
# Whole-file size cap (OXICLOUD_MAX_UPLOAD_SIZE)
|
||||
# ═════════════════════════════════════════════════════════════
|
||||
# The whole-file cap is checked at session creation against the
|
||||
# JSON-declared `total_size` — no upload body required to trip
|
||||
# it. Test default `OXICLOUD_MAX_UPLOAD_SIZE` is 10 GiB; we
|
||||
# declare 100 GiB to be safely above without bothering with a
|
||||
# Hurl `--variables` override.
|
||||
#
|
||||
# This guards the case where chunks would otherwise accumulate
|
||||
# disk space against an oversized declared upload — the reject
|
||||
# fires BEFORE any chunk is PATCHed.
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 21 — POST /api/uploads with `total_size` above
|
||||
# OXICLOUD_MAX_UPLOAD_SIZE → 413 Payload Too Large.
|
||||
# No body bytes are sent; the check is purely
|
||||
# against the declared JSON field.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/uploads
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/json
|
||||
{
|
||||
"filename": "way-too-big.bin",
|
||||
"folder_id": "{{home_folder_id}}",
|
||||
"content_type": "application/octet-stream",
|
||||
"total_size": 107374182400,
|
||||
"chunk_size": 5242880
|
||||
}
|
||||
|
||||
HTTP 413
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 22 — Sanity check: a session JUST BELOW the cap is
|
||||
# accepted. Uses a sub-cap value (32 bytes — same
|
||||
# pattern as earlier steps so no extra fixture is
|
||||
# needed). Confirms that the reject in step 21 was
|
||||
# the size cap, not an unrelated regression.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
POST {{base_url}}/api/uploads
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/json
|
||||
{
|
||||
"filename": "small-and-fine.txt",
|
||||
"folder_id": "{{home_folder_id}}",
|
||||
"content_type": "text/plain",
|
||||
"total_size": 32,
|
||||
"chunk_size": 1048576
|
||||
}
|
||||
|
||||
HTTP 201
|
||||
[Captures]
|
||||
upload_id_sanity: jsonpath "$.upload_id"
|
||||
|
||||
# Cleanup — cancel without uploading.
|
||||
DELETE {{base_url}}/api/uploads/{{upload_id_sanity}}
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP 204
|
||||
|
||||
|
||||
# ═════════════════════════════════════════════════════════════
|
||||
# Direct-PUT body cap (OXICLOUD_DIRECT_PUT_MAX_BYTES)
|
||||
# ═════════════════════════════════════════════════════════════
|
||||
# `OXICLOUD_DIRECT_PUT_MAX_BYTES` (4 MiB in the test env) bounds
|
||||
# a single non-chunked PUT body. Larger uploads must come through
|
||||
# the chunked protocol — the server returns 413 with a hint
|
||||
# pointing at `/api/uploads/...` / `/dav/uploads/...`. The cap
|
||||
# fires mid-stream via the same accumulator that bounds chunks.
|
||||
#
|
||||
# Tested via the native REST WebDAV PUT endpoint (`/webdav/...`)
|
||||
# because it's JWT-authed (no app-password mint dance) and
|
||||
# straightforwardly path-mapped. The cap is wired into the same
|
||||
# `spool_body_to_temp` helper from the NextCloud single-file PUT
|
||||
# path, so this exercise covers both wirings.
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 23 — Direct PUT of the 5 MiB fixture → 413. The same
|
||||
# fixture used in step 8 for the chunked-PATCH cap;
|
||||
# here it lands against the direct-PUT cap.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PUT {{base_url}}/webdav/test-direct-put-over-cap.bin
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: application/octet-stream
|
||||
file,fixtures/chunk-over-cap-5mb.bin;
|
||||
|
||||
HTTP 413
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
# Step 24 — Sanity: direct PUT WELL under the cap → 201 or 204.
|
||||
# Confirms step 23's reject was the cap, not a route /
|
||||
# auth / handler regression.
|
||||
# ─────────────────────────────────────────────────────────────
|
||||
PUT {{base_url}}/webdav/test-direct-put-under-cap.txt
|
||||
Authorization: Bearer {{token}}
|
||||
Content-Type: text/plain
|
||||
file,fixtures/hello.txt;
|
||||
|
||||
# WebDAV PUT returns 201 on new file, 204 on overwrite. Either
|
||||
# value confirms a successful body acceptance.
|
||||
HTTP *
|
||||
[Asserts]
|
||||
status >= 200
|
||||
status < 300
|
||||
|
||||
# Cleanup so a re-run finds a clean slate (DELETE is idempotent
|
||||
# enough for this — 204 on success, 404 if nothing left).
|
||||
DELETE {{base_url}}/webdav/test-direct-put-under-cap.txt
|
||||
Authorization: Bearer {{token}}
|
||||
|
||||
HTTP *
|
||||
|
||||
@@ -27,6 +27,14 @@ RUST_LOG="warn,audit=info"
|
||||
# under the cap, while the cap test sends a 5 MiB fixture to trigger 413.
|
||||
OXICLOUD_CHUNK_MAX_BYTES=4194304
|
||||
|
||||
# Direct-PUT (non-chunked) cap, exercised by chunked_upload_cap.hurl.
|
||||
# 4 MiB: same threshold as the chunked cap so the existing 5 MiB
|
||||
# fixture (chunk-over-cap-5mb.bin) can prove BOTH caps with one
|
||||
# generated file. All existing direct-PUT tests
|
||||
# (test_dedup_webdav_multichunk.sh = 2.76 MB, _ref_count = ~66 KB,
|
||||
# _nextcloud_put_blake3 = 32 B) stay safely under this cap.
|
||||
OXICLOUD_DIRECT_PUT_MAX_BYTES=4194304
|
||||
|
||||
# grow up limits for tests
|
||||
OXICLOUD_RATE_LIMIT_REFRESH_MAX=360
|
||||
OXICLOUD_RATE_LIMIT_LOGIN_MAX=360
|
||||
|
||||
Reference in New Issue
Block a user