批次4后续专项完成:D11清偿 + OCC方案B实施 + 部署参数 + D2诚实标注 + CI门禁

① D11 HTML 报告 RustFS 单源化(TECH_DEBT P2 清偿):可视化产物写任务临时目录后
   裸传报告键 html/reports/{filename}(文件名寻址),/html StaticFiles 挂载删除,
   新增 html_report_router 根路径代理(报告键→遗留 JSON 包装→本地卷兜底→404,
   防穿越);URL 形状 /html/{filename} 不变,持久化引用零迁移;celery 摘除
   html_data 卷,镜像不再烤入陈旧报告;顺带删除 get_stp_file_with_data 死数据块
② OCC 方案 B(D10 清偿):run_occ(op_name, payload) 契约 + 常驻工作进程池
   (occ_process_pool + occ_worker 操作注册表),超时/崩溃 terminate 换新补位、
   任务级超时 recover 整体重建,残留线程泄漏根治;TopoDS 不跨进程(generate_cavity
   分模 + 方案 STEP 持久化全在子进程内,返回 export_manifest);删除内存形状缓存链、
   CADExporter.export_mold_results、shape_loader(→ stp_materializer)
③ OCC 方案 A 部署参数:CELERY_CONCURRENCY / CELERY_MAX_TASKS_PER_CHILD 进
   Dockerfile.celery + compose + .env.example
④ D2 诚实标注:铝价响应带 source: "simulated",前端按来源渲染标注(原硬编码
   "上海期货交易所"属虚假声明),死代码 getAluminumPrice 删除
⑤ CI 门禁:.gitea/workflows/ci.yml 三 job(pytest / 前端构建含 vue-tsc /
   openapi 漂移检测)

接口变更三件套随批完成(openapi 76→77 paths + gen:api + 前端构建通过;方案 B
接口面零变化)。测试基线 143 passed, 0 skipped(新增 16 项)。文档六处同步。

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
2026-09-18 17:22:01 +08:00
parent 0e6b3b1811
commit e728dcd226
36 changed files with 1474 additions and 565 deletions
+5
View File
@@ -88,3 +88,8 @@ LLM_API_KEY=sk-your-api-key
LLM_MODEL=gpt-4o-mini
LLM_TIMEOUT=60
LLM_MAX_TOKENS=2000
# Celery/OCC 吞吐调优(可选,默认值见 deploy/Dockerfile.celery;
# concurrency 即 OCC 并行分析数,见 docs/topics/performance/OCC_THROUGHPUT.md)
# CELERY_CONCURRENCY=2
# CELERY_MAX_TASKS_PER_CHILD=50
+68
View File
@@ -0,0 +1,68 @@
# CI 门禁(Gitea Actions,语法与 GitHub Actions 兼容)
#
# 三个 job:
# 1. backend-tests —— pytest 全量(sqlite+aiosqlite,无外部服务依赖;
# OCC 契约测试在无 pythonocc 的 pip 环境自动 skip)
# 2. frontend-build —— npm ci + npm run build(含 vue-tsc -b 类型检查,D15 已修复)
# 3. openapi-drift —— 用 OCC 环境重导出 openapi.json 与仓库版本比对,
# 防止接口变更三件套(AGENTS §2)被遗漏导致前后端漂移
#
# 运行前提:Gitea 实例启用 Actions 且注册了 runner;
# `ubuntu-latest` label 需映射到带 node/git 的镜像(gitea runner 默认映射满足)。
# 已验证事实:pytest 与 openapi 导出均不依赖 .env(settings 惰性校验)。
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
backend-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install dependencies
# requirements.txt 自带阿里云 pip 镜像配置
run: pip install -r requirements.txt
- name: Run tests
run: python -m pytest tests/ -q
frontend-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install dependencies
working-directory: frontend
run: npm ci
- name: Type check and build
working-directory: frontend
run: npm run build
openapi-drift:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Miniforge and pythonocc
# pythonocc-core 仅经 conda-forge 提供(与 deploy/Dockerfile.moldinsight 同版本约束)
# 默认走 TUNA 镜像(Gitea 服务器多为国内网络);海外环境可换回官方源
run: |
wget -q https://mirrors.tuna.tsinghua.edu.cn/github-release/conda-forge/miniforge/LatestRelease/Miniforge3-Linux-x86_64.sh -O miniforge.sh
bash miniforge.sh -b -p "$HOME/miniforge"
"$HOME/miniforge/bin/conda" create -n ci -c https://mirrors.tuna.tsinghua.edu.cn/conda-forge -y python=3.12 pythonocc-core=7.9.0
- name: Install python dependencies
run: "$HOME/miniforge/envs/ci/bin/pip" install -r requirements.txt
- name: Export openapi.json and compare with committed version
run: |
"$HOME/miniforge/envs/ci/bin/python" -c "import sys; sys.path.insert(0, 'src'); from entrypoints.unified import app; import json; print(json.dumps(app.openapi(), ensure_ascii=False, indent=2))" > /tmp/openapi.json
git diff --exit-code --no-index /tmp/openapi.json openapi.json || {
echo "::error::openapi.json 与代码不一致——请按 docs/API_CONTRACT.md §4 重导出并执行 npm run gen:api"
exit 1
}
+9 -6
View File
@@ -53,7 +53,8 @@ src/
machining_router.py # 加工类接口:/design-cam /check-collision /optimize-toolpath /design-electrodes /simulate-machining
export_router.py # 导出类接口:/export-mold /export-download /export-recommendations
core_modules.py # 核心计算模块惰性装载器(设计/加工路由共用,装载失败 503)
aluminum_price_routes.py # /api/aluminum-price/* 铝价(当前为模拟/参考数据,见 TECH_DEBT D2)
aluminum_price_routes.py # /api/aluminum-price/* 铝价(模拟数据,响应带 source: "simulated",见 TECH_DEBT D2)
html_report_router.py # GET /html/{filename} 报告代理(根路径挂载:RustFS 报告键 → 遗留 JSON 包装 → 本地卷兜底;include_into 由入口调用)
debug_router.py # /api/debug/tasks 全量任务 dump(仅 DEBUG 模式注册,仍需登录)
core/ # 几何与方案核心算法(OCC 重依赖区)
stp_parser.py # STEP/STP 解析
@@ -71,16 +72,18 @@ src/
cavity_layout_optimizer.py # 型腔布局优化
mold_machining.py / mold_cam.py # 加工与 CAM
mold_quality_inspector.py # 质量检查
cad_exporter.py # CAD 导出
cad_exporter.py # CAD 导出(export_mold_results 随方案 B 已删;export_step 等供 OCC 子进程持久化/转换)
occ_worker.py # OCC 常驻工作进程入口:操作注册表(parse_stp/generate_mesh/generate_cavity/analyze_mold_design/detect_undercuts/convert_component_step/ping/sleep/warmup)+ worker_main 消息循环
services/ # 业务服务层
task_dispatcher.py # 后台任务统一分派(勿绕过它 fire-and-forget)
task_query_service.py # 任务状态查询聚合
processing_service.py # 分析处理编排
processing_service.py # 分析处理编排(run_occ 经常驻 OCC 进程池调度,见 occ_process_pool.py / OCC_THROUGHPUT.md)
occ_process_pool.py # OCC 常驻工作进程池(方案 B):超时/崩溃 terminate 换新补位;任务级超时 recover 整体重建
calculation_service.py # 计算服务
cost_estimate_service.py # 成本估算
cam_bundle_service.py # CAM 结果打包
verification_service.py # FreeCAD 验证(可选)
shape_loader.py # shape 加载(含 OCC 超时后 executor 重建逻辑)
stp_materializer.py # 按 task_id 把 STP 原件落盘临时文件(OCC 解析在子进程内,形状不跨进程)
material_service.py # 物料价格服务
aluminum_price_service.py # 铝价服务(模拟数据)
llm_service.py # LLM 增强分析(可选,OpenAI 兼容)
@@ -98,7 +101,7 @@ src/
models/ # inventory 域 ORM(catalog / warehouse / trading / finance 四文件,共 15 表)
utils.py
shared/ # 【共享平台层:只放真正跨模块复用的基础能力,勿堆业务】
app_factory.py # create_app:request_id 日志中间件 / auth_router / /health / SPA fallback / /html mount
app_factory.py # create_app:request_id 日志中间件 / auth_router / /health / SPA fallback / connect_rustfs 开关(D11 后 /html 由 moldinsight 代理路由提供,不再挂本地 StaticFiles)
config/settings.py # Settings 单例:dotenv + os.getenv;DB_*/SECRET_KEY 惰性校验无默认
database/database.py # async engine / session / get_db_session
database/init_db.py # 建表与管理员种子
@@ -110,7 +113,7 @@ src/
services/redis_task_manager.py # Redis 任务状态(Hash 字段级原子更新,兼容旧 string)
utils/logger.py # 结构化日志(json/text)+ request_id
utils/file_handler.py # 上传文件处理
utils/html_generator.py # /html 静态分析报告生成
utils/html_generator.py # 可视化报告生成(HTML/摘要/数据 JSON;产物写任务临时目录,由 moldinsight 上传 RustFS 报告键)
celery_app.py # Celery app(Redis broker,task_acks_late)
celery_tasks.py # moldinsight 异步分析任务
frontend/ # Vue 3 独立工程:src/modules 按域组织(moldinsight/inventory/users/login/home);src/types/api.ts 为 openapi 生成物,勿手改
+9 -1
View File
@@ -3,4 +3,12 @@
# gemold-backend:latest 不一致,干净环境下 celery 镜像构建必然失败。
FROM gemold-backend:latest
CMD ["celery", "-A", "celery_app", "worker", "--workdir=/app/src", "--concurrency=2", "--loglevel=info"]
# OCC 并行度伸缩(OCC_THROUGHPUT 方案 A,见 docs/topics/performance/OCC_THROUGHPUT.md):
# 每个 prefork 子进程各持一个串行 OCC 通道,concurrency 即并行分析数
# (调大时预算好每子进程内存与 PG 连接数);max-tasks-per-child 让子进程
# 定期重启,兜底回收 OCC 超时后滞留的线程。可在 compose/.env 覆盖。
ENV CELERY_CONCURRENCY=2 \
CELERY_MAX_TASKS_PER_CHILD=50
# sh -c + exec:既支持环境变量替换,又让 celery exec 接管 PID 1 正确接收 SIGTERM
CMD ["sh", "-c", "exec celery -A celery_app worker --workdir=/app/src --concurrency=${CELERY_CONCURRENCY:-2} --max-tasks-per-child=${CELERY_MAX_TASKS_PER_CHILD:-50} --loglevel=info"]
-1
View File
@@ -34,7 +34,6 @@ COPY migrations/ /app/migrations/
COPY alembic.ini /app/alembic.ini
COPY uploads/ /app/uploads/
COPY html_output/ /app/html_output/
ENV PYTHONPATH=/app/src
+8 -4
View File
@@ -65,8 +65,8 @@ services:
LLM_TIMEOUT: ${LLM_TIMEOUT:-60}
LLM_MAX_TOKENS: ${LLM_MAX_TOKENS:-2000}
AUTO_MIGRATE: ${AUTO_MIGRATE:-true}
# 共享卷过渡兜底(D6/D11):主链路已改走 RustFS,本地卷仅为
# RustFS 异常时的本地路径回退与 HTML 产物互通保留,后续批次移除
# 共享卷过渡兜底(D6/D11):主链路已改走 RustFS。uploads 供 RustFS 异常时
# 本地路径回退;html_output 仅作 /html 报告代理的存量兜底读(新产物不落本地)
volumes:
- uploads_data:/app/uploads
- html_data:/app/html_output
@@ -115,10 +115,13 @@ services:
LLM_MODEL: ${LLM_MODEL:-gpt-4o-mini}
LLM_TIMEOUT: ${LLM_TIMEOUT:-60}
LLM_MAX_TOKENS: ${LLM_MAX_TOKENS:-2000}
# 与 backend 共享本地卷(过渡兜底,见 D6/D11):worker 下载回退与 HTML 产物写读
# Celery/OCC 吞吐调优(OCC_THROUGHPUT 方案 A,默认值在 Dockerfile.celery)
CELERY_CONCURRENCY: ${CELERY_CONCURRENCY:-2}
CELERY_MAX_TASKS_PER_CHILD: ${CELERY_MAX_TASKS_PER_CHILD:-50}
# uploads_data 共享卷(D6 过渡兜底):RustFS 异常时 worker 回退本地路径下载。
# D11 后 worker 不再写 HTML 产物(直传 RustFS 报告键),无需 html_data 卷
volumes:
- uploads_data:/app/uploads
- html_data:/app/html_output
depends_on:
- backend
restart: unless-stopped
@@ -177,6 +180,7 @@ services:
LLM_TIMEOUT: ${LLM_TIMEOUT:-60}
LLM_MAX_TOKENS: ${LLM_MAX_TOKENS:-2000}
AUTO_MIGRATE: ${AUTO_MIGRATE:-true}
# html_output 仅作 /html 报告代理的存量兜底读(D11,新产物不落本地)
volumes:
- uploads_data:/app/uploads
- html_data:/app/html_output
+2 -1
View File
@@ -24,7 +24,7 @@
- **鉴权**:JWT Bearer(`Authorization: Bearer <token>`)。登录:`POST /api/auth/login`(表单)/ `POST /api/auth/login/json`(JSON);受保护路由通过 FastAPI 依赖 `get_current_active_user` 注入当前用户([shared/services/auth_service.py](../src/shared/services/auth_service.py))。`SECRET_KEY` 跨进程必须一致。
- **响应形态**:现状**无统一信封包装**——各端点直接返回业务 JSON;schema 以 `openapi.json` 的 components 为准。新增接口不建议另起信封风格,保持与所在模块一致。
- **错误**:FastAPI 标准 `HTTPException` 状态码语义;业务校验优先 Pydantic 请求模型自动 422。
- **业务路由前缀**:全部业务端点在 `/api` 下;顶层仅 `/health`(探活)、`/login`、`/users`(历史遗留入口,前端主链路用 `/api/auth/*`)。
- **业务路由前缀**:全部业务端点在 `/api` 下;顶层仅 `/health`(探活)、`/html/{filename}`(可视化报告代理)、`/login`、`/users`(历史遗留入口,前端主链路用 `/api/auth/*`)。
## 3. 端点总览(按域分组)
@@ -47,6 +47,7 @@
| 批量分析 | `/api/batch-upload`、`/api/batch/{batch_id}`(聚合状态以 PG 为准;响应含 `current_step`;他人批次 403、不存在 404) | batch_router.py |
| 任务状态 | `/api/status/{task_id}`(需登录;仅任务所有者可访问,他人/无主任务 403,不存在 404) | task_router.py |
| 历史结果 | `/api/history`、`/api/history/{filename}` | history_router.py |
| HTML 报告 | `/html/{filename}`(根路径,非 `/api` 前缀;D11 代理:RustFS 报告键 `html/reports/{filename}` 直取 → 遗留 JSON 包装对象 → 本地 `html_output/` 存量兜底 → 404。已知约束:不做认证——iframe 无法携带 Authorization 头,沿用 StaticFiles 时代既定姿态) | html_report_router.py |
| CAM | `/api/cam/plan`(Pydantic 请求模型;未提供的偏好回落任务持久化偏好再回落默认) | cam_router.py |
| 设计 | `/api/optimize-layout`、`/api/design-cooling`、`/api/design-gating`、`/api/design-mold-system`、`/api/detect-undercuts` | design_router.py |
| 成本估算 | `/api/cost-estimate` | cost_router.py |
+3 -3
View File
@@ -49,8 +49,8 @@ Celery worker(moldinsight 异步分析链路;本地从 `src` 目录跑,与
cd src && celery -A celery_app worker --concurrency=2 --loglevel=info
```
- `--concurrency=N` 即 OCC 并行分析数(每个 prefork 子进程持一个串行 OCC 通道,见 [topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md) 方案 A);调大时预算好每子进程内存与 PG 连接数。
- 建议生产加 `--max-tasks-per-child=M`(如 50):子进程定期重启,兜底回收 OCC 超时后滞留的线程。
- `--concurrency=N` 即 OCC 并行分析数:每个 prefork 子进程持一个常驻 OCC 工作进程(方案 B,见 [topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md))——每个 OCC 工作进程是独立的 Python + OCC 运行时,**N 增大时按「worker 子进程 + OCC 子进程」双份预算内存**,并预留 PG 连接数(按 celery 角色池随子进程倍增)。
- `--max-tasks-per-child=M`(如 50):worker 子进程定期重启,连带回收其 OCC 子进程(进程级兜底,方案 A)。
前端:
@@ -78,7 +78,7 @@ docker compose --profile inventory up -d # inventory 单模块栈
- **生产环境必须显式设置**:`SECRET_KEY`、`ADMIN_PASSWORD`、`DB_*`、`CORS_ORIGINS`、`RUSTFS_*`、`REDIS_PASSWORD`、`DEBUG=false`、`LOG_FORMAT=json`。
- **单数据库**:moldinsight 与 inventory 共享同一 PostgreSQL(刻意设计,不拆库)。
- **后台任务一律走 `task_dispatcher`** 与 Celery,不要在路由里 fire-and-forget。
- **uploads/ 与 html_output/ 为运行时产物目录**,不提交、不作为配置源头。
- **uploads/ 与 html_output/ 为运行时产物目录**,不提交、不作为配置源头。`html_output/` 自 D11 起仅作 `/html` 报告代理的**存量兜底读**(新产物直传 RustFS 报告键 `html/reports/`,worker 不再写本地卷)。
- `scripts/` 下的一次性脚本执行前先确认目标环境(多为不可逆数据迁移)。
## 6. 排障指针
+2 -2
View File
@@ -100,7 +100,7 @@ geMoldInsight 已从历史单体逐步演进为“双业务模块 + 共享平台
### P3:专项能力继续规范化
- 铝价模拟数据增加显式 `source: "simulated"`
- ~~铝价模拟数据增加显式 `source: "simulated"`~~(2026-09-18 完成:后端响应带 `source` 字段,前端按来源渲染标注,不再硬编码交易所名)
- 补专题文档的定位/边界说明
- 清理历史 checklist / tasks / report 文档的展示层级
@@ -121,7 +121,7 @@ geMoldInsight 已从历史单体逐步演进为“双业务模块 + 共享平台
**执行顺序建议**:批次 0 与批次 1 的 D6(RustFS 主链路)先行——前者是确认的安全漏洞,后者是部署根本性缺陷,两者互不依赖、改动可控。其余按批次顺序推进,每批完成同步 STATUS / TECH_DEBT / API_CONTRACT。
> 进度:批次 0 / 1 / 2 已于 2026-09-16 完成、批次 3 / 4 已于 2026-09-17 完成,§3.1 批次计划**全部执行完毕**(遗留:D13 的 pip 全量锁文件随下次镜像构建补齐;D11 留待后续批次,正确性已由批次 1 共享卷兜底;批次 4 遗留中期项——OCC 进程池方案 B 实施待独立排期,见 [TECH_DEBT.md](TECH_DEBT.md) D10 与 [topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md))。完成明细见 [STATUS.md](STATUS.md) 与 [TECH_DEBT.md](TECH_DEBT.md) §2.5–2.8。后续优先项回到 §3 P2 / P3 与主线方向。
> 进度:批次 0 / 1 / 2 已于 2026-09-16 完成、批次 3 / 4 已于 2026-09-17 完成,§3.1 批次计划**全部执行完毕**;批次 4 后续专项于 2026-09-18 完成——D11(HTML 报告 RustFS 单源 + `/html` 代理路由)与 **OCC 方案 B(`run_occ` 契约进程化 + 常驻进程池 kill-on-timeout)已清偿**(部署参数方案 A 一并落地,见 [TECH_DEBT.md](TECH_DEBT.md) D10 与 [topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md))。遗留:D13 的 pip 全量锁文件随下次镜像构建补齐。完成明细见 [STATUS.md](STATUS.md) 与 [TECH_DEBT.md](TECH_DEBT.md) §2.5–2.8。后续优先项回到 §3 P2 / P3 与主线方向。
---
+2
View File
@@ -3,6 +3,8 @@
> 文档定位:**唯一的「现在到哪了」**。README / AGENTS / 各主文档只链接到这里,不复制状态内容。
> 维护规则:每完整完成一个需求,**倒序在本文顶部加一条**(日期 + 主题 + 关键事实);其余主文档(架构 / 规划 / 技术债 / 部署)维护各自的"当前有效说法",本文只记录"什么时候做到了哪一步"。维护规则出处见根目录 [AGENTS.md](../AGENTS.md)。
> 2026-09-18(**批次 4 后续专项五项完成:D11 清偿 + 部署参数 + D2 诚实标注 + CI 门禁 + OCC 方案 B 实施**:① **D11 清偿**(TECH_DEBT P2)——可视化报告 RustFS 单源化:写侧 HTMLGenerator 每任务写临时目录,`.html`/`_summary.json`/`_data.json` 三件统一裸传 RustFS 报告键 `html/reports/{filename}`(文件名寻址),`/html` StaticFiles 本地挂载删除,新增 [html_report_router.py](../src/moldinsight/api/html_report_router.py) 根路径代理(报告键直取 → 遗留 `html/{hash}.json` JSON 包装解析 → 本地卷存量兜底 → 404;URL 形状 `/html/{filename}` 不变,持久化 cavity JSON 与前端 iframe 引用零迁移);celery 服务摘除 `html_data` 卷,Dockerfile.moldinsight 删除 `COPY html_output/`(构建机陈旧报告不再进镜像);已知约束:报告路由不做认证(iframe 无法携带 Authorization 头,沿用 StaticFiles 时代既定姿态,文档已声明);顺带删除 `get_stp_file_with_data` 的死数据块(`html_content` 组装零消费方,此前每次完成任务查询白下载数 MB 正文)。② **OCC 方案 A 参数落地**——`CELERY_CONCURRENCY`/`CELERY_MAX_TASKS_PER_CHILD` 进 [Dockerfile.celery](../deploy/Dockerfile.celery) ENV + compose 透传 + `.env.example`。③ **D2 清偿**——铝价响应带 `source: "simulated"`,[HomeView.vue](../frontend/src/modules/home/HomeView.vue) 按来源渲染"模拟数据 · 参考走势"标注(原硬编码"上海期货交易所"属虚假声明),死代码 `getAluminumPrice` 删除。④ **CI 门禁**——[.gitea/workflows/ci.yml](.gitea/workflows/ci.yml) 三 job:pytest 全量 / 前端构建(含 vue-tsc)/ openapi 漂移检测(conda pythonocc 环境重导出比对;已实测 pytest 与导出均不依赖 .env)。⑤ **OCC 方案 B 实施**(TECH_DEBT D10 清偿)——`run_occ(fn, *args)` → `run_occ(op_name, payload)`,执行器由线程池替换为常驻工作进程池 [occ_process_pool.py](../src/moldinsight/services/occ_process_pool.py) + 操作注册表 [occ_worker.py](../src/moldinsight/core/occ_worker.py):超时/崩溃 terminate 换新补位、任务级超时 recover 整体重建,**残留线程泄漏根治**(进程边界回收 C++ 栈);TopoDS 形状不跨进程(`generate_cavity` 分模 + 方案 STEP 持久化导出全在子进程内,返回 export_manifest);调用点全量迁移(解析/网格/型腔/分析/倒扣/STEP 转换),删除内存形状缓存链(`_cache_export_shapes`/`get_export_shapes`/`_persist_step_exports`)、`CADExporter.export_mold_results`(零调用方)、shape_loader(→ [stp_materializer.py](../src/moldinsight/services/stp_materializer.py));回归测试 [test_occ_process_pool.py](../tests/test_occ_process_pool.py)(OCC-gated,6 例含真实盒体 STP 解析/分模端到端)。**接口变更三件套随批完成**:openapi.json 重导出(76→77 paths,新增 `/html/{filename}`)+ 前端 `gen:api` 再生 + 前端构建通过(方案 B 接口面零变化,无路由/schema 变更)。**测试基线**:**143 passed, 0 skipped**(D11 8 项 + 铝价 2 项 + OCC 进程池 6 项;基线 129 中原 2 个 skip 已随本地环境补齐 celery/alembic 转为执行)。**下一步**:回到 §3 主线 P2/P3 长期方向——D3 剩余收敛(app_factory 参数收敛、identity/platform 语义)、inventory 服务下沉、D13 pip 锁文件随下次镜像构建补齐,见 [ROADMAP.md](ROADMAP.md) §3。)
> 2026-09-17(**批次 4(架构演进)完成,§3.1 治理批次全部执行完毕**:① D3 主体清偿——891 行的旧 `shared/models/database.py`(31 模型类三类同居,已删除)按归属拆为 [shared/models/base.py](../src/shared/models/base.py)(唯一 Base)+ [shared/models/identity.py](../src/shared/models/identity.py)(身份权限 7 表)+ [moldinsight/models/](../src/moldinsight/models/)(分析域 9 表)+ [inventory/models/](../src/inventory/models/)(进销存 15 表,catalog/warehouse/trading/finance 四文件);**三条跨模块 ORM relationship(`User.stp_files` / `STPFile.user` / `STPFile.product`)经全仓核实零使用,直接删除**——跨模块桥接收敛为裸 FK 硬规则([ARCHITECTURE.md](ARCHITECTURE.md) §5.1),单模块部署 mapper 可独立配置;约 45 处 import 全量改写(含 migrations/env.py 全量注册、scripts/ 两个一次性脚本),旧模块物理删除无兼容 facade;零调用方死方法 `db_manager.create_tables` 一并删除(拆分后会静默建残缺 schema);② D10 治理——`_reset_occ_executor` 补 `cancel_futures=True`(旧实现下"慢恢复"的旧线程会继续消化旧队列,与新 executor **并发操作非线程安全的 OCC**,属数据竞争而非单纯泄漏);吞吐方案设计先行定稿 [topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md)(短期 A:celery `--concurrency` 伸缩 + `--max-tasks-per-child` 兜底,启动参数已记 [OPERATIONS.md](OPERATIONS.md) §3;中期 B:`run_occ` 契约进程化 + kill-on-timeout,待独立排期);③ 新增 [tests/test_model_ownership.py](../tests/test_model_ownership.py) 锁定归属边界(31 表全量注册 / 单模块独立 mapper 配置 / 旧模块无 facade);④ 顺手清偿 D15——[vite.config.ts](../frontend/vite.config.ts) 删除未用的 `mode` 参数,`vue-tsc -b` 恢复通过,前端生产构建链路解除阻断。**接口面零变化**(无路由与 schema 变更,openapi.json 不触发重导出)。**测试基线**:**125 passed, 2 skipped**(基线 122 + 新增归属测试 3 项)。**下一步**:治理批次收尾后回到主线方向;遗留项 D11(HTML RustFS 单源)、D13(pip 锁文件)、OCC 方案 B 独立批次。)
> 2026-09-17(**批次 3(API 与代码结构)完成**:① D1 清偿——592 行 advanced_router 拆为 [design_router](../src/moldinsight/api/design_router.py) / [cost_router](../src/moldinsight/api/cost_router.py) / [machining_router](../src/moldinsight/api/machining_router.py) / [export_router](../src/moldinsight/api/export_router.py) 四个子路由(端点路径不变),请求体全量 Pydantic 模型化(`request.json()` 手动解析退役,校验统一 422);② 路由装载失败显式化:`ROUTE_MODULES` 清单 + [route_registry](../src/moldinsight/api/route_registry.py),失败经 `/api/health` 呈现 `degraded` 并列出清单(`pythonocc` 改真实探测),DEBUG 下 fail fast——此前失败仅 WARNING 后静默跳过,进程带病启动不可感知;③ 纯 Python 重计算端点(设计/加工/CAM 打包)统一 `asyncio.to_thread` 投放线程池,不再阻塞事件循环(OCC 仍走单线程 executor,D10 留批次 4);④ `StorageIntegrationService`(867 行)按职责拆为 [task_storage](../src/moldinsight/services/task_storage_service.py) / [analysis_storage](../src/moldinsight/services/analysis_storage_service.py) / [file_history](../src/moldinsight/services/file_history_service.py) 三服务,无调用方死代码 `log_user_activity` 删除;⑤ D14 收尾清偿——`MAX_FILE_SIZE` 接线生效(默认上限 50MB→100MB,以 .env 为准)、celery_app 复用 `Settings.redis_url`(连接串唯一拼装点)。**连带修复**:管理员重置密码改 JSON body `{ new_password }`(原裸 str 参数被解析为 query param,前端两个调用点均发 body,功能端到端断裂)+ [UsersView.vue](../frontend/src/modules/users/UsersView.vue) 同步;Dockerfile.celery 的 FROM 对齐 `gemold-backend:latest`(此前引用不存在的 tag,干净环境 celery 镜像必构建失败)。**接口变更三件套随批完成**:openapi.json 重导出(76 paths)+ 前端 `gen:api` 再生。**连带发现**:`npm run build` 因 vite.config.ts 既有 TS6133 失败(与本项目改动无关,登记 D15)。**测试基线**:**122 passed, 2 skipped**(新增 4 个测试文件共 17 项:[test_advanced_split_contract](../tests/test_advanced_split_contract.py) / [test_route_load_status](../tests/test_route_load_status.py) / [test_config_governance](../tests/test_config_governance.py) / [test_auth_password_reset](../tests/test_auth_password_reset.py);skips 为 alembic / celery 缺失环境)。**下一步**:批次 4(架构演进:共享 ORM 拆分、OCC 吞吐方案,见 [ROADMAP.md](ROADMAP.md) §3.1)。)
+19 -16
View File
@@ -173,30 +173,33 @@
~~原现状 / 影响~~:各存储方法内部自行 commit,型腔保存失败留半成品数据且任务仍 completed。
### D10. OCC 全局单线程串行 + 超时重建泄漏线程 —— 泄漏治理已落地,吞吐方案设计先行(2026-09-17,批次 4)
### D10. OCC 全局单线程串行 + 超时重建泄漏线程 —— 已清偿(2026-09-18,方案 B 实施)
已完成:
- `_reset_occ_executor` 补 `cancel_futures=True`:排队任务随重建丢弃——旧实现下"慢恢复"的旧线程会继续消化旧队列,与新 executor 并发操作非线程安全的 OCC;修复后残留收敛为"运行中线程滞留 1 个"(C++ 栈 Python 层不可杀,属客观边界)
- 吞吐与隔离方案定稿:[topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md)——短期方案 A(celery `--concurrency` 伸缩 + `--max-tasks-per-child` 进程回收兜底,零新代码,启动参数见 [OPERATIONS.md](OPERATIONS.md) §3);中期方案 B(`run_occ` 改操作名+payload 契约、常驻 OCC 进程池 kill-on-timeout 根治泄漏,待独立批次)
**方案 B(常驻 OCC 进程池,kill-on-timeout 根治泄漏)已实施**(2026-09-18):
- `run_occ(fn, *args)` → `run_occ(op_name, payload)`;执行器由进程内线程池替换为常驻工作进程池 [occ_process_pool.py](../src/moldinsight/services/occ_process_pool.py) + 操作注册表 [occ_worker.py](../src/moldinsight/core/occ_worker.py)(新增;详见 [topics/performance/OCC_THROUGHPUT.md](topics/performance/OCC_THROUGHPUT.md) §5)
- 超时/崩溃 = terminate() 换新补位——**残留线程泄漏根治**(C++ 栈由进程边界回收);OCC segfault 不再波及 API/worker 主进程
- 调用点全部迁移(解析/网格/型腔/分析/倒扣/STEP 转换),TopoDS 形状不跨进程(`generate_cavity` 的方案形状 STEP 导出改在子进程内持久化,返回 export_manifest)
- 顺带删除:内存形状缓存链(`_cache_export_shapes` / `get_export_shapes` / `_persist_step_exports`)、`CADExporter.export_mold_results`(零调用方)、shape_loader(→ [stp_materializer.py](../src/moldinsight/services/stp_materializer.py))
- 回归测试:[tests/test_occ_process_pool.py](../tests/test_occ_process_pool.py)(OCC-gated,6 例含真实盒体 STP 端到端)
**方案 A 部署参数落地**(2026-09-18):`CELERY_CONCURRENCY` / `CELERY_MAX_TASKS_PER_CHILD` 进 [Dockerfile.celery](../deploy/Dockerfile.celery) + compose + `.env.example`(`--max-tasks-per-child` 仍保留为进程回收兜底)。
保留为已知约束(非待修缺陷):
- 单进程内 OCC 串行是正确性要求(OCC 非线程安全),吞吐扩展走多进程(方案 A/B)
- 线程级超时的滞留线程由进程边界回收,根治依赖方案 B 落地
- 每个操作从 STP 原件重新加载形状(STEP 重载成本秒级)——进程隔离的设计取舍,见 OCC_THROUGHPUT §1.2/§5
优先级:**P3**(中期方案 B 实施前维持观察)
优先级:~~**P3**~~ **已清偿**
### D11. HTML 报告本地磁盘与 RustFS 双写双读
### D11. HTML 报告本地磁盘与 RustFS 双写双读 —— 已清偿(2026-09-18)
现状:
- 可视化 HTML/摘要同时写本地 `html_output/`(/html 静态挂载)与 RustFS
修复内容:
- **写侧**:可视化产物不再落节点本地 `html_output/`——HTMLGenerator 每任务写临时目录([processing_service.py](../src/moldinsight/services/processing_service.py)),`.html` / `_summary.json` / `_data.json` 三件统一裸传 RustFS 报告键 `html/reports/{filename}`(文件名寻址,同源同秒重复分析即覆盖刷新);`HTMLFile` 表仅存元数据
- **读侧**:`/html` StaticFiles 挂载删除,新增代理路由 [html_report_router.py](../src/moldinsight/api/html_report_router.py)(挂根路径保持 URL 形状——持久化 cavity JSON 与前端 iframe 均引用 `/html/{filename}`):RustFS 报告键直取 → 遗留 `html/{hash}.json` JSON 包装解析 → 本地卷存量兜底 → 404,防路径穿越(单段文件名校验)
- **部署**:celery 服务摘除 `html_data` 卷(不再写本地);Dockerfile.moldinsight 删除 `COPY html_output/`(构建机陈旧报告不再烤进镜像)
- **已知约束**(沿用 StaticFiles 时代既定姿态,非新引入):报告路由不做认证——iframe 无法携带 Authorization 头
- 回归测试:[tests/test_html_report_router.py](../tests/test_html_report_router.py)(8 例:四链路命中、新旧格式记录区分、媒体类型、404、穿越拒绝)
影响:
- 多副本下 /html 命中结果取决于负载均衡,跨副本文件不共享;同一份报告两套来源
建议:
- 统一 RustFS 为唯一来源,本地仅作按需缓存
优先级:**P2**
~~原现状 / 影响~~:可视化 HTML/摘要同时写本地与 RustFS,多副本下 /html 命中结果取决于负载均衡,跨副本文件不共享。
### D12. 应用启动时自动执行 alembic 迁移
+22 -6
View File
@@ -10,9 +10,10 @@
### 1.1 运行时事实
- 所有 OCC 操作(STEP 解析 / 布尔运算 / 三角化 / 倒扣检测等)统一经 [processing_service.py](../../../src/moldinsight/services/processing_service.py) 的 `run_occ` 投入 **进程内 `ThreadPoolExecutor(max_workers=1)`** 串行执行——OCC 非线程安全,串行是正确性要求,不是实现偷懒。
- Celery worker 为 prefork 模式,`processing_service` 是模块级单例:**每个 worker 子进程各持一个串行 OCC 通道**。因此 OCC 并行度 = worker 子进程数,与 web 进程数无关(web 侧 `run_occ` 仅服务于轻量同步调用,如倒扣检测)。
- 型腔生成超时后 `_reset_occ_executor` 重建 executor;已在运行的 C++ 线程在 Python 层**不可杀**,每次超时滞留 1 个线程(2026-09-17 起排队任务随 `cancel_futures=True` 丢弃,见 §4)。
- 所有 OCC 操作(STEP 解析 / 布尔运算 / 三角化 / 倒扣检测 / STEP 转换等)统一经 [processing_service.py](../../../src/moldinsight/services/processing_service.py) 的 `run_occ(op_name, payload)` 投入 **常驻 OCC 工作进程池**([occ_process_pool.py](../../../src/moldinsight/services/occ_process_pool.py),默认 1 进程 = 1 串行通道)——OCC 非线程安全,通道内串行是正确性要求,不是实现偷懒。
- 操作在 [occ_worker.py](../../../src/moldinsight/core/occ_worker.py) 以注册表形式实现(parse_stp / generate_mesh / generate_cavity / analyze_mold_design / detect_undercuts / convert_component_step 等);输入输出全部是**文件路径 + 普通字典**,TopoDS_Shape 不跨进程传输(方案 B 硬约束,见 §1.2)。
- Celery worker 为 prefork 模式,`processing_service` 是模块级单例:**每个 worker 子进程各持一个常驻 OCC 工作进程**。因此 OCC 并行度 = worker 子进程数,与 web 进程数无关(web 侧 `run_occ` 仅服务于轻量同步调用,如倒扣检测)。
- 超时/崩溃 = `terminate()` 该工作进程并换新补位——进程边界干净回收,无线程滞留。
### 1.2 硬约束(决定方案边界)
@@ -61,10 +62,25 @@
| 阶段 | 动作 | 状态 |
|---|---|---|
| 短期 | 方案 A:`--concurrency` 伸缩 + `--max-tasks-per-child` 兜底回收;`cancel_futures=True` 修复重建并发风险 | ✅ 代码侧 2026-09-17 完成;部署参数随下次 compose/镜像评审落地 |
| 中期 | 方案 B:`run_occ(op_name, payload)` 接口演进 + 常驻进程池,kill-on-timeout 根治泄漏 | 待排期(独立批次,工作量集中在调用点迁移与序列化设计) |
| 短期 | 方案 A:`--concurrency` 伸缩 + `--max-tasks-per-child` 兜底回收;`cancel_futures=True` 修复重建并发风险 | ✅ 部署参数 2026-09-18 落地(`CELERY_CONCURRENCY` / `CELERY_MAX_TASKS_PER_CHILD` 进 Dockerfile.celery + compose + .env.example) |
| 中期 | 方案 B:`run_occ(op_name, payload)` 接口演进 + 常驻进程池,kill-on-timeout 根治泄漏 | ✅ 2026-09-18 实施完成(见 §5;回归测试 [tests/test_occ_process_pool.py](../../../tests/test_occ_process_pool.py)) |
| 长期 | 方案 C:sidecar,仅在出现独立伸缩需求时启动 | 暂不启动 |
## 4. 本次已落地的缓解(2026-09-17,批次 4)
## 4. 已落地的缓解(2026-09-17,批次 4)
`_reset_occ_executor` 的 `shutdown(wait=False)` 补 `cancel_futures=True`。这不只是卫生问题:旧实现下旧 executor 的**排队任务不会消失**——若挂死线程后来"慢恢复",旧线程会继续消化旧队列,与新 executor **并发操作非线程安全的 OCC**(数据竞争 / 崩溃风险)。补参后排队任务即被丢弃,残留问题收敛为"运行中线程滞留 1 个",由方案 A 的进程回收兜底。
## 5. 方案 B 实施记录(2026-09-18)
**接口**:`run_occ(fn, *args)` → `run_occ(op_name, payload)`;执行器由进程内线程池替换为常驻进程池。
- **新增** [occ_process_pool.py](../../../src/moldinsight/services/occ_process_pool.py):`OccProcessPool`(默认 size=1)。每个 `_OccWorker` 是一个 spawn 出的常驻子进程 + 双工管道 + 独立 `asyncio.Lock`(通道串行);阻塞收发经 `asyncio.to_thread` 不卡事件循环。操作超时或进程死亡 → `terminate()` + 换新补位;任务级整体超时(`process_file_with_storage` 外层 wait_for)→ `recover()` 整体重建。`shutdown()` 供应用退出/测试清理。
- **新增** [occ_worker.py](../../../src/moldinsight/core/occ_worker.py):操作注册表 + `worker_main` 消息循环。OCC 模块在 handler 内惰性导入(父进程 pip 环境无 OCC 也可 import),进程内单例缓存(parser/planner/analyzer/mesh_gen 等)。全部操作输入输出为**文件路径 + 普通字典**,TopoDS 不跨进程。
- **调用点迁移**(processing_service):
- `parse_stp` = 原 load_step_file + analyze_geometry 两步合一(形状在子进程内即生即用)
- `generate_mesh` / `analyze_mold_design` / `detect_undercuts` / `convert_component_step` 同名对位
- `generate_cavity` = 分模 + 方案形状持久化 STEP 导出全在子进程内;`_export_shapes`(TopoDS)不再回主进程,返回 export_manifest(与旧 `_persist_step_exports` 结构一致,主进程原样存 export_artifacts)
- 旧 `_cache_export_shapes` / `get_export_shapes` / `_persist_step_exports` / `_export_shapes_cache` 及线程 executor / `_reset_occ_executor` 全部删除(跨进程本就不存在内存形状缓存,export_router 相应移除 `export_mold_results` 内存分支)
- [shape_loader.py](../../../src/moldinsight/services/shape_loader.py) → [stp_materializer.py](../../../src/moldinsight/services/stp_materializer.py):只把 STP 原件落盘临时文件,OCC 解析交给子进程操作
- **成本确认**:每个 spawn 子进程首次操作需 import OCC(秒级);进程常驻后后续操作复用缓存实例。每个操作从 STP 原件重新加载形状(STEP 重载成本,见 §1.2)——原线程方案跨步骤共享 shape 的内存优势让位于进程隔离,符合方案 B 设计取舍。
- **测试**:[tests/test_occ_process_pool.py](../../../tests/test_occ_process_pool.py)(OCC-gated,6 例):spawn+管道往返 / 未知操作错误回传 / 子进程异常浮出 / 超时换新补位 / 真实盒体 STP 解析端到端 / generate_cavity 分模+STEP 落盘端到端。
+15 -3
View File
@@ -9,9 +9,9 @@
<div class="card aluminum-price-card">
<div class="card-header">
<div class="card-title">
<span class="aluminum-icon">🪙</span> 铝金属价格 (SHFE)
<span class="aluminum-icon">🪙</span> 铝金属价格
</div>
<span class="aluminum-source">数据来源: 上海期货交易所 | 更新于 {{ aluminum.date }}</span>
<span class="aluminum-source">{{ aluminumSourceLabel }}</span>
</div>
<div class="aluminum-content">
<div class="aluminum-price-row">
@@ -78,7 +78,7 @@
</template>
<script setup lang="ts">
import { reactive, onMounted, nextTick, ref } from 'vue'
import { computed, reactive, onMounted, nextTick, ref } from 'vue'
import { useRouter } from 'vue-router'
import { apiRequest } from '@/shared/api'
import { Chart, registerables } from 'chart.js'
@@ -98,6 +98,8 @@ interface AluminumPrice {
low: number
prev_close: number
week_ago_price: number
// D2:数据来源声明——simulated 为模拟走势(参考数据,非实时行情)
source?: string
}
interface HistoryItem {
@@ -112,6 +114,7 @@ const aluminum = reactive({
change: 0,
change_percent: 0,
open: 0,
source: '',
high: 0,
low: 0,
prev_close: 0,
@@ -124,12 +127,21 @@ const formatAluminumPrice = (val: number | null) => {
return Math.round(val).toLocaleString('zh-CN')
}
// D2:来源标注以接口 source 字段为准,不再硬编码交易所名称——
// 此前显示"数据来源: 上海期货交易所"而数据实为模拟走势,属虚假来源声明
const aluminumSourceLabel = computed(() => {
if (aluminum.source === 'simulated') return '模拟数据 · 参考走势,非实时行情'
if (aluminum.source) return `数据来源: ${aluminum.source}`
return `更新于 ${aluminum.date}`
})
const loadAluminumPrice = async () => {
try {
const data = await apiRequest<AluminumPrice>('/api/aluminum-price/current')
aluminum.price = data.price
aluminum.unit = data.unit
aluminum.date = data.date
aluminum.source = data.source ?? ''
aluminum.change = data.change
aluminum.change_percent = data.change_percent
aluminum.open = data.open
-4
View File
@@ -403,8 +403,4 @@ export const moldinsightApi = {
body: JSON.stringify(data),
})
},
getAluminumPrice() {
return apiRequest<{ price: number; unit: string; updated_at: string }>('/api/aluminum-price/current')
},
}
+48
View File
@@ -666,6 +666,23 @@ export interface paths {
patch?: never;
trace?: never;
};
"/html/{filename}": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** 读取 HTML 可视化报告 */
get: operations["get_html_report_html__filename__get"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/api/products": {
parameters: {
query?: never;
@@ -4538,6 +4555,37 @@ export interface operations {
};
};
};
get_html_report_html__filename__get: {
parameters: {
query?: never;
header?: never;
path: {
filename: string;
};
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Successful Response */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": unknown;
};
};
/** @description Validation Error */
422: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["HTTPValidationError"];
};
};
};
};
list_products_api_products_get: {
parameters: {
query?: {
+38 -1
View File
@@ -1771,6 +1771,43 @@
}
}
},
"/html/{filename}": {
"get": {
"summary": "读取 HTML 可视化报告",
"operationId": "get_html_report_html__filename__get",
"parameters": [
{
"name": "filename",
"in": "path",
"required": true,
"schema": {
"type": "string",
"title": "Filename"
}
}
],
"responses": {
"200": {
"description": "Successful Response",
"content": {
"application/json": {
"schema": {}
}
}
},
"422": {
"description": "Validation Error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
}
}
}
}
},
"/api/products": {
"get": {
"tags": [
@@ -10292,4 +10329,4 @@
}
}
}
}
}
-1
View File
@@ -24,7 +24,6 @@ def _register_routers(app):
app = create_app(
title="Gemold - 进销存管理系统",
service_name="inventory",
mount_html=False,
serve_frontend_static=False,
register_routers=_register_routers,
)
+8 -1
View File
@@ -20,11 +20,18 @@ def _register_routers(app):
except Exception as e:
print(f"[WARN] MoldInsight 路由: {e}")
# D11:HTML 报告代理挂根路径,URL 形状与原 StaticFiles 保持一致(/html/{filename})
try:
from moldinsight.api.html_report_router import include_into as include_html_report
include_html_report(app)
except Exception as e:
print(f"[WARN] HTML 报告路由: {e}")
app = create_app(
title="Gemold - 模具分析引擎",
service_name="moldinsight",
mount_html=True,
connect_rustfs=True,
serve_frontend_static=False,
register_routers=_register_routers,
)
+8 -1
View File
@@ -20,6 +20,13 @@ def _register_routers(app):
except Exception as e:
print(f"[WARN] MoldInsight 路由: {e}")
# D11:HTML 报告代理挂根路径,URL 形状与原 StaticFiles 保持一致(/html/{filename})
try:
from moldinsight.api.html_report_router import include_into as include_html_report
include_html_report(app)
except Exception as e:
print(f"[WARN] HTML 报告路由: {e}")
try:
from inventory.api import inventory_router
app.include_router(inventory_router)
@@ -30,7 +37,7 @@ def _register_routers(app):
app = create_app(
title="Gemold - Unified Backend",
service_name="unified",
mount_html=True,
connect_rustfs=True,
serve_frontend_static=False,
register_routers=_register_routers,
)
+2 -1
View File
@@ -4,7 +4,8 @@
- 装载失败返回 None 且不缓存失败(与原实现一致,端点统一 503「服务不可用」)
- 实例缓存:设计/加工模块为纯 Python 计算(构造后无 self 突变,方法仅读入参),
可安全地被 asyncio.to_thread 并发调用;OCC 相关的 side_action_designer
必须经 processing_service.run_occ 的单线程 executor 使用
经 processing_service.run_occ 的常驻 OCC 进程池使用(方案 B,见
docs/topics/performance/OCC_THROUGHPUT.md)
"""
import threading
from typing import Optional
+17 -13
View File
@@ -3,8 +3,8 @@
- 请求体一律 Pydantic 模型(原 request.json() 手动解析退役,校验失败统一 422)
- 纯 Python 设计计算统一经 asyncio.to_thread 投放线程池,不阻塞事件循环;
OCC 相关的倒扣检测仍走 processing_service.run_occ 的单线程 executor
(PythonOCC 非线程安全)
OCC 相关的倒扣检测经 processing_service.run_occ 的常驻 OCC 进程池
(PythonOCC 非线程安全,进程内串行;见 OCC_THROUGHPUT.md 方案 B)
"""
import asyncio
from typing import List, Optional
@@ -165,17 +165,21 @@ async def detect_undercuts(
if not sd:
raise HTTPException(503, "服务不可用:核心模块未加载")
# 从持久化 STP 原件重建几何(此前传 shape=None 会被兜底吞掉,永远返回"无倒扣")
from moldinsight.services.shape_loader import get_shape_loader
shape = await get_shape_loader().load_shape_for_task(db_session, body.task_id)
if shape is None:
# 方案 B:从持久化 STP 原件落盘,倒扣分析在常驻 OCC 子进程内完成(形状不跨进程)
from moldinsight.services.stp_materializer import get_stp_materializer
stp_path = await get_stp_materializer().materialize_stp_for_task(db_session, body.task_id)
if stp_path is None:
raise HTTPException(410, "任务几何不可用:无法从存储重建 STP 形状,请重新上传分析")
# OCC 操作必须走单线程 executor(PythonOCC 非线程安全)
result = await processing_service.run_occ(
sd.analyze_and_design,
shape,
body.parting_direction,
body.mold_size.model_dump(),
)
try:
result = await processing_service.run_occ(
"detect_undercuts",
{
"stp_path": str(stp_path),
"parting_direction": body.parting_direction,
"mold_size": body.mold_size.model_dump(),
},
)
finally:
stp_path.unlink(missing_ok=True)
return {"status": "success", "data": result}
+35 -66
View File
@@ -178,76 +178,45 @@ async def export_mold_results(
},
}
cavity_shapes = processing_service.get_export_shapes(
task_id,
resolved_scheme_id,
)
# 方案 B 后主进程不再持有内存形状缓存(get_export_shapes 已删除):
# 未命中持久化清单时,从持久化单组件 STEP 现场转换缺失格式,用户无需重新分析
filename = task_data.get("filename", f"mold_{task_id}")
if not cavity_shapes:
# 内存 shape 缓存失效(如服务重启):从持久化的单组件 STEP
# 现场转换缺失格式,用户无需重新分析
artifacts = _get_export_artifacts(task_data)
scheme_data = (artifacts.get("schemes") or {}).get(resolved_scheme_id)
if scheme_data:
base_filename = scheme_data.get("base_filename") or Path(filename).stem
regenerated = await processing_service.regenerate_export_from_persisted(
task_id=task_id,
scheme_id=resolved_scheme_id,
formats=formats,
components=_expand_components(components),
base_filename=base_filename,
scheme_files=scheme_data.get("files", []),
)
if regenerated:
regenerated["files"] = _augment_export_files(
task_id, regenerated.get("files", [])
)
# 合并进持久化 manifest,后续请求直接命中持久化路径
merged_artifacts = _merge_export_artifacts(artifacts, regenerated)
await TaskStorageService().update_task_parameters(
db_session,
task_id,
{"export_artifacts": merged_artifacts},
)
# D9:存储方法已不再自行 commit,请求侧显式提交
await db_session.commit()
await redis_task_manager.update_task(
task_id, {"export_artifacts": merged_artifacts}
)
TaskQueryService.invalidate_task_view(task_id)
return {"status": "success", "data": regenerated}
raise HTTPException(
409,
"导出缓存已失效或任务尚未完成,请重新分析后再导出以保证方案一致性",
artifacts = _get_export_artifacts(task_data)
scheme_data = (artifacts.get("schemes") or {}).get(resolved_scheme_id)
if scheme_data:
base_filename = scheme_data.get("base_filename") or Path(filename).stem
regenerated = await processing_service.regenerate_export_from_persisted(
task_id=task_id,
scheme_id=resolved_scheme_id,
formats=formats,
components=_expand_components(components),
base_filename=base_filename,
scheme_files=scheme_data.get("files", []),
)
if regenerated:
regenerated["files"] = _augment_export_files(
task_id, regenerated.get("files", [])
)
# 合并进持久化 manifest,后续请求直接命中持久化路径
merged_artifacts = _merge_export_artifacts(artifacts, regenerated)
await TaskStorageService().update_task_parameters(
db_session,
task_id,
{"export_artifacts": merged_artifacts},
)
# D9:存储方法已不再自行 commit,请求侧显式提交
await db_session.commit()
await redis_task_manager.update_task(
task_id, {"export_artifacts": merged_artifacts}
)
TaskQueryService.invalidate_task_view(task_id)
base_filename = Path(filename).stem
result = cad_exporter.export_mold_results(
cavity_data=cavity_shapes,
base_filename=base_filename,
formats=formats,
components=components,
task_id=task_id,
scheme_id=resolved_scheme_id,
return {"status": "success", "data": regenerated}
raise HTTPException(
409,
"导出文件不可用:任务未生成持久化导出清单,请重新分析后再导出",
)
result["files"] = _augment_export_files(task_id, result.get("files", []))
result["source"] = "generated"
merged_artifacts = _merge_export_artifacts(_get_export_artifacts(task_data), result)
await TaskStorageService().update_task_parameters(
db_session,
task_id,
{"export_artifacts": merged_artifacts},
)
# D9:存储方法已不再自行 commit,请求侧显式提交
await db_session.commit()
await redis_task_manager.update_task(task_id, {"export_artifacts": merged_artifacts})
TaskQueryService.invalidate_task_view(task_id) # parameters 已变更,缓存视图失效
return {"status": "success", "data": result}
@router.get("/export-download/{filepath:path}")
+139
View File
@@ -0,0 +1,139 @@
# api/html_report_router.py
"""HTML 可视化报告读取代理(D11)。
报告产物唯一持久来源是 RustFS 报告键 html/reports/{filename}(文件名寻址),
本路由以 GET /html/{filename} 提供读取,替代原节点本地 html_output 的
StaticFiles 挂载——API 与 worker 容器文件系统不互通,本地盘从来不是
可依赖的读取来源。
解析顺序(逐级兜底,每次未命中记日志):
1. RustFS 报告键(新产物,裸 HTML / 裸 JSON)
2. HTMLFile 表记录(遗留 html/{hash}.json JSON 包装 {'content'})
3. 节点本地 html_output 目录(存量兜底,compose 共享卷;新产物不再写本地)
已知约束(沿用 StaticFiles 时代的既定姿态,非本次引入):本路由不做认证。
iframe 加载报告时浏览器不会携带 Authorization 头,无法套用 API 鉴权。
"""
import json
from pathlib import Path
from typing import Optional
from fastapi import APIRouter, HTTPException
from fastapi.responses import Response
from sqlalchemy import select
from shared.config.settings import settings
from shared.database.database import db_manager
from shared.utils.logger import get_logger
from moldinsight.models import HTMLFile
from moldinsight.storage.rustfs_storage import rustfs_manager
logger = get_logger(__name__)
router = APIRouter()
# 存量兜底目录(模块常量,测试可替换)
LOCAL_HTML_DIR = Path("html_output")
_MEDIA_TYPES = {
".html": "text/html; charset=utf-8",
".json": "application/json",
}
def include_into(app) -> None:
"""挂载到应用根路径——不能进 /api 前缀聚合:URL 形状必须保持
/html/{filename}(持久化 cavity JSON 与前端 iframe 均引用此形状)。
失败语义与 route_registry._safe_include 一致:非 DEBUG 记入
route_load_status['failed'](/api/health 呈现 degraded),DEBUG 直接抛错。
"""
try:
app.include_router(router)
except Exception as exc:
from moldinsight.api.route_registry import route_load_status
route_load_status["failed"].append(
{"label": "HTML报告", "module": __name__, "error": str(exc)}
)
logger.error(f"HTML 报告路由加载失败: {exc}")
if settings.DEBUG:
raise
def _validate_filename(filename: str) -> None:
"""防路径穿越:只允许单段文件名(报告键固定为 html/reports/ 一级平铺)。"""
if not filename or filename.startswith(".") or Path(filename).name != filename:
raise HTTPException(status_code=404, detail=f"报告不存在: {filename}")
def _media_type_for(filename: str) -> str:
return _MEDIA_TYPES.get(Path(filename).suffix.lower(), "application/octet-stream")
async def _download_from_rustfs(filename: str) -> Optional[bytes]:
"""新产物:html/reports/{filename} 裸文件直取。"""
if not rustfs_manager.is_connected:
return None
try:
return await rustfs_manager.download_report_artifact(filename)
except Exception as exc:
logger.debug(f"报告键未命中(继续遗留解析): {filename}: {exc}")
return None
async def _download_from_legacy_record(filename: str) -> Optional[bytes]:
"""遗留 HTMLFile 记录:html/{hash}.json JSON 包装 {'content'}。"""
if not rustfs_manager.is_connected:
return None
async with db_manager.session() as session:
result = await session.execute(
select(HTMLFile)
.where(HTMLFile.filename == filename)
.order_by(HTMLFile.id.desc())
)
record = result.scalars().first()
if record is None:
return None
data = await rustfs_manager.download_file("html_files", record.object_key)
if record.object_key.startswith(rustfs_manager.report_prefix + "/"):
# 新格式记录:报告键直取瞬时失败走到这里,裸文件原样返回
return data
wrapper = json.loads(data.decode("utf-8"))
content = wrapper.get("content")
if content is None:
raise ValueError(f"遗留报告对象缺少 content 字段: {record.object_key}")
return content.encode("utf-8")
def _download_from_local(filename: str) -> Optional[bytes]:
"""存量兜底:旧 worker 写入共享卷 html_output 的历史产物。"""
path = LOCAL_HTML_DIR / filename
if path.is_file():
return path.read_bytes()
return None
@router.get("/html/{filename:path}", summary="读取 HTML 可视化报告")
async def get_html_report(filename: str) -> Response:
_validate_filename(filename)
data = await _download_from_rustfs(filename)
source = "rustfs"
if data is None:
try:
data = await _download_from_legacy_record(filename)
except Exception as exc:
logger.debug(f"遗留记录解析失败(继续本地兜底): {filename}: {exc}")
source = "rustfs-legacy"
if data is None:
data = _download_from_local(filename)
source = "local-fallback"
if data is None:
raise HTTPException(status_code=404, detail=f"报告不存在: {filename}")
if source != "rustfs":
# 存量链路命中留痕,便于评估遗留对象与本地卷的清理时机
logger.info(f"报告经 {source} 链路命中: {filename}")
return Response(content=data, media_type=_media_type_for(filename))
-110
View File
@@ -374,116 +374,6 @@ class CADExporter:
logger.error(f"BRep 导出失败: {e}")
return False
def export_mold_results(self, cavity_data: Dict,
base_filename: str,
formats: List[str] = None,
components: List[str] = None,
task_id: Optional[str] = None,
scheme_id: Optional[str] = None) -> Dict[str, Any]:
if formats is None:
formats = ["step", "stl"]
if components is None:
components = ["cavity", "core"]
export_dir = self.build_export_dir(
base_filename=base_filename,
task_id=task_id,
scheme_id=scheme_id,
)
os.makedirs(export_dir, exist_ok=True)
results = {
"base_filename": base_filename,
"task_id": task_id,
"scheme_id": scheme_id,
"export_dir": export_dir,
"files": [],
"errors": [],
}
shape_map = self._SHAPE_MAP
shapes_to_export: List[Tuple[str, str, TopoDS_Shape]] = []
assembly_shapes: List[Tuple[TopoDS_Shape, str]] = []
for comp in components:
if comp == "all":
for key, (data_key, label) in shape_map.items():
shape = cavity_data.get(data_key)
if shape is not None:
shapes_to_export.append((key, label, shape))
assembly_shapes.append((shape, label))
break
elif comp in shape_map:
data_key, label = shape_map[comp]
shape = cavity_data.get(data_key)
if shape is not None:
shapes_to_export.append((comp, label, shape))
assembly_shapes.append((shape, label))
else:
results["errors"].append(f"{label}形状不可用")
# STEP: 所有组件合并为一个装配体文件
if "step" in formats and assembly_shapes:
filepath = os.path.join(export_dir, f"{base_filename}_mold.step")
success = self.export_assembly_step(assembly_shapes, filepath)
if success:
file_size = os.path.getsize(filepath)
relative_path = self.get_relative_path(filepath)
results["files"].append({
"component": "assembly",
"component_label": "模具装配体",
"format": "step",
"filepath": filepath,
"relative_path": relative_path,
"filename": os.path.basename(filepath),
"size_bytes": file_size,
"size_readable": self._format_file_size(file_size),
})
else:
results["errors"].append("装配体 STEP 导出失败")
# IGES / STL / BRep: 逐组件导出
non_assembly_formats = [f for f in formats if f != "step"]
for comp_name, label, shape in shapes_to_export:
for fmt in non_assembly_formats:
filepath = os.path.join(export_dir, f"{base_filename}_{comp_name}.{fmt}")
success = False
if fmt == "iges":
success = self.export_iges(shape, filepath)
elif fmt == "stl":
success = self.export_stl(shape, filepath)
elif fmt == "brep":
success = self.export_brep(shape, filepath)
else:
results["errors"].append(f"不支持的格式: {fmt}")
continue
if success:
file_size = os.path.getsize(filepath)
relative_path = self.get_relative_path(filepath)
results["files"].append({
"component": comp_name,
"component_label": label,
"format": fmt,
"filepath": filepath,
"relative_path": relative_path,
"filename": os.path.basename(filepath),
"size_bytes": file_size,
"size_readable": self._format_file_size(file_size),
})
else:
results["errors"].append(f"{label} ({fmt}) 导出失败")
results["total_files"] = len(results["files"])
results["total_errors"] = len(results["errors"])
logger.info(f"模具导出完成: {results['total_files']} 个文件, "
f"{results['total_errors']} 个错误")
return results
def export_assembly_step(self, shapes_with_names: List[Tuple[TopoDS_Shape, str]],
filepath: str,
schema: str = "AP214") -> bool:
+251
View File
@@ -0,0 +1,251 @@
# core/occ_worker.py
"""OCC 常驻工作进程入口(方案 B,见 docs/topics/performance/OCC_THROUGHPUT.md)。
子进程经 multiprocessing spawn 拉起后运行 worker_main 消息循环:父进程经管道
下发 (op_name, payload),本进程从 _OPS 注册表取 handler 执行并回传结果。
所有操作输入输出均为文件路径 + 可 pickle 的普通字典,**杜绝 pickle OCC 对象**
(TopoDS_Shape 为 C++ 原生内存对象,跨进程传输的唯一干净方式是经文件中转)。
OCC 模块在 handler 内惰性导入:本模块在无 OCC 的父进程(pip 环境)导入不报错,
子进程(conda 环境)首次执行某操作时才加载对应模块——保持 processing_service
可无 OCC 导入(OCC 契约测试在无 OCC 环境 skip)。
"""
import os
import traceback
from datetime import datetime
from pathlib import Path
from typing import Any, Callable, Dict
# ─── 操作注册表 ─────────────────────────────────────────────────
# 新增 OCC 操作必须在此注册:op_name -> 单 payload 字典的 handler。
# handler 内惰性 import OCC 相关核心模块。
_OPS: Dict[str, Callable[[Dict[str, Any]], Any]] = {}
def _op(name: str):
"""注册操作到 _OPS。"""
def deco(fn):
_OPS[name] = fn
return fn
return deco
# 进程内惰性单例缓存(OCC 实例化成本高,子进程常驻期间复用)
_OCC_CACHE: Dict[str, Any] = {}
def _cached(key: str, factory: Callable[[], Any]) -> Any:
if key not in _OCC_CACHE:
_OCC_CACHE[key] = factory()
return _OCC_CACHE[key]
def _get_parser():
from moldinsight.core.stp_parser import STPParser
return STPParser()
def _get_planner():
from moldinsight.core.multi_scheme_planner import MultiSchemeMoldPlanner
return MultiSchemeMoldPlanner()
def _get_geometry_analyzer():
from moldinsight.core.geometry_analyzer import GeometryAnalyzer
return GeometryAnalyzer()
def _get_mesh_generator():
from moldinsight.core.mesh_generator import MeshGenerator
return MeshGenerator(quality="medium")
def _get_side_action_designer():
from moldinsight.core.side_action_designer import SideActionDesigner
return SideActionDesigner()
def _get_cad_exporter(output_dir: str):
from moldinsight.core.cad_exporter import CADExporter
return CADExporter(output_dir=output_dir)
# ─── 操作实现 ───────────────────────────────────────────────────
@_op("ping")
def _op_ping(payload):
return {"pong": True}
@_op("sleep")
def _op_sleep(payload):
"""诊断/测试用:在子进程内挂起指定秒数。"""
import time
time.sleep(float(payload.get("seconds", 0)))
return {"slept": True}
@_op("warmup")
def _op_warmup(payload):
"""预热:子进程启动后触发,把 OCC 模块加载成本放到池启动而非首个分析任务。"""
_cached("parser", _get_parser)
_cached("planner", _get_planner)
_cached("geometry_analyzer", _get_geometry_analyzer)
_cached("mesh_generator", _get_mesh_generator)
return {"warmed": True}
@_op("parse_stp")
def _op_parse_stp(payload):
"""STP 解析 + 几何分析(原主进程 load_step_file → analyze_geometry 两步合一)。
形状在子进程内创建并即刻消费,不跨进程传输。
"""
parser = _cached("parser", _get_parser)
shape = parser.load_step_file(Path(payload["stp_path"]))
return parser.analyze_geometry(shape)
@_op("generate_mesh")
def _op_generate_mesh(payload):
parser = _cached("parser", _get_parser)
mesh_gen = _cached("mesh_generator", _get_mesh_generator)
shape = parser.load_step_file(Path(payload["stp_path"]))
return mesh_gen.generate_multi_lod_mesh(shape)
@_op("generate_cavity")
def _op_generate_cavity(payload):
"""多方案分模 + 方案形状 STEP 导出,全部在子进程内完成。
plan_result 里携带的 _export_shapes(TopoDS 对象)无法跨进程,子进程直接
经 CADExporter 落盘为持久化 STEP,返回文件 manifest——与旧 _persist_step_exports
产物结构一致,主进程原样存入 export_artifacts。
"""
parser = _cached("parser", _get_parser)
planner = _cached("planner", _get_planner)
shape = parser.load_step_file(Path(payload["stp_path"]))
plan_result = planner.generate_plan(
shape=shape,
material=payload["material"],
is_foam_material=payload.get("is_foam_material", False),
process_params=payload.get("process_params"),
)
export_shapes = plan_result.pop("_export_shapes", {}) or {}
export_manifest = _persist_export_shapes(payload, export_shapes)
return {"plan_result": plan_result, "export_manifest": export_manifest}
def _persist_export_shapes(payload: Dict[str, Any], export_shapes: Dict[str, Any]) -> Any:
"""在子进程内把各方案的 TopoDS 形状导出为持久化 STEP,返回 manifest。"""
if not export_shapes:
return None
task_id = payload["task_id"]
stp_path = payload["stp_path"]
export_out_dir = payload.get("export_out_dir")
if not export_out_dir:
raise ValueError("generate_cavity 缺少 export_out_dir")
# 每次按任务目录新建 exporter(output_dir 依任务固定;模块 import 由 importlib 缓存)
exporter = _get_cad_exporter(export_out_dir)
base_filename = Path(stp_path).stem or f"mold_{task_id}"
components = ["cavity", "core", "parting_surface", "product", "a_plate", "b_plate"]
manifest = {
"version": 1,
"task_id": task_id,
"generated_at": datetime.now().isoformat(),
"schemes": {},
}
for scheme_id, cavity_data in export_shapes.items():
try:
result = exporter.export_persisted_steps(
cavity_data=cavity_data,
base_filename=base_filename,
components=components,
task_id=task_id,
scheme_id=scheme_id,
)
manifest["schemes"][scheme_id] = {
"base_filename": result.get("base_filename"),
"generated_at": datetime.now().isoformat(),
"files": result.get("files", []),
"errors": result.get("errors", []),
"total_files": result.get("total_files", 0),
"total_errors": result.get("total_errors", 0),
}
except Exception as exc:
logger = _get_logger()
logger.warning("持久化 STEP 导出失败: task=%s scheme=%s error=%s", task_id, scheme_id, exc)
manifest["schemes"][scheme_id] = {
"base_filename": base_filename,
"generated_at": datetime.now().isoformat(),
"files": [],
"errors": [str(exc)],
"total_files": 0,
"total_errors": 1,
}
return manifest
def _get_logger():
from shared.utils.logger import get_logger
return get_logger(__name__)
@_op("analyze_mold_design")
def _op_analyze_mold_design(payload):
parser = _cached("parser", _get_parser)
analyzer = _cached("geometry_analyzer", _get_geometry_analyzer)
shape = parser.load_step_file(Path(payload["stp_path"]))
return analyzer.analyze_mold_design(
payload["geometry_data"],
product_material=payload.get("product_material", "ABS"),
shape=shape,
)
@_op("detect_undercuts")
def _op_detect_undercuts(payload):
parser = _cached("parser", _get_parser)
designer = _cached("side_action_designer", _get_side_action_designer)
shape = parser.load_step_file(Path(payload["stp_path"]))
return designer.analyze_and_design(
shape,
payload["parting_direction"],
payload["mold_size"],
)
@_op("convert_component_step")
def _op_convert_component_step(payload):
exporter = _get_cad_exporter(os.path.dirname(payload["step_path"]))
return exporter.convert_component_step(
payload["step_path"], payload["out_path"], payload["fmt"]
)
# ─── 子进程主循环 ───────────────────────────────────────────────
def worker_main(conn) -> None:
"""OCC 子进程消息循环。conn 为 multiprocessing.Connection(双工)。
收到 None 或管道断开即退出;单条操作异常回传 error 不杀循环
(进程级隔离意味着一个操作 segfault 才会杀死本进程,由父进程补位)。
"""
while True:
try:
msg = conn.recv()
except (EOFError, KeyboardInterrupt, ConnectionResetError, OSError):
break
if msg is None:
break
op_name, payload = msg
handler = _OPS.get(op_name)
if handler is None:
conn.send(("error", {"error": f"未知 OCC 操作: {op_name}"}))
continue
try:
result = handler(payload)
conn.send(("ok", result))
except BaseException as exc: # noqa: BLE001 子进程内兜底,保证循环存活
conn.send(("error", {"error": f"{type(exc).__name__}: {exc}", "traceback": traceback.format_exc()}))
@@ -59,6 +59,9 @@ def get_aluminum_current_price() -> Dict:
"low": round(price - random.uniform(10, 60), 0),
"prev_close": prev_price,
"week_ago_price": round(week_price, 0),
# D2:数据为模拟走势(见模块 docstring),必须显式声明来源,
# 前端按此字段展示"模拟/参考"标注,防止被当作实时行情
"source": "simulated",
}
@@ -87,6 +90,7 @@ def get_aluminum_price_history(days: int = 30) -> List[Dict]:
"high": high_price,
"low": low_price,
"close": close_price,
"source": "simulated", # D2:与 current 一致,逐项显式声明模拟来源
})
random.seed()
@@ -5,6 +5,7 @@
批次 3 自 storage_integration_rustfs.py 按职责拆分(见 task_storage_service.py 头注)。
"""
import json
from pathlib import Path
from typing import Optional, Dict, Any
from sqlalchemy import select
@@ -270,39 +271,29 @@ class AnalysisStorageService:
stp_file_id: int,
filename: str,
file_path: str,
html_content: Optional[str] = None,
visualization_type: str = "3d_viewer") -> HTMLFile:
"""保存HTML文件到PostgreSQL元数据 + RustFS对象存储"""
"""保存HTML文件:正文按报告键裸传 RustFS + PG 仅存元数据(D11)。
# 1. 获取文件哈希
stp_file = await session.get(STPFile, stp_file_id)
file_hash = stp_file.file_hash
上传键固定 html/reports/{filename}(文件名寻址),读侧 GET /html/{filename}
按文件名直取,不再写 JSON 包装对象(遗留 html/{hash}.json 仅由读侧兼容解析)。
file_path 为任务内临时目录路径,任务结束即删除,仅供追溯,不作为读取来源。
"""
html_file_path = Path(file_path)
if not html_file_path.exists():
raise RuntimeError(f"HTML 可视化文件缺失: {file_path}")
html_bytes = html_file_path.read_bytes()
# 2. 读取HTML内容(如果未提供)
if html_content is None:
try:
with open(file_path, 'r', encoding='utf-8') as f:
html_content = f.read()
except Exception as e:
logger.error(f"读取HTML文件失败: {e}")
html_content = ""
# 3. 上传到RustFS
html_json = {'content': html_content, 'filename': filename}
upload_result = await rustfs_manager.upload_json_data(
file_type='html_files',
json_data=html_json,
file_hash=file_hash
object_key = await rustfs_manager.upload_report_artifact(
filename, html_bytes, content_type="text/html; charset=utf-8"
)
# 4. 创建PostgreSQL记录
html_file = HTMLFile(
stp_file_id=stp_file_id,
object_key=upload_result['object_key'],
storage_bucket=upload_result['bucket'],
object_key=object_key,
storage_bucket=rustfs_manager.bucket_name,
filename=filename,
file_path=file_path, # 保留本地路径
# 停止双写完整 HTML 进 PG:读取路径走 RustFS(html_json.content),
file_path=file_path,
# 停止双写完整 HTML 进 PG:读取路径走 /html/{filename} 代理,
# PG 仅存对象键与文件名,避免大文本撑爆表
html_content=None,
visualization_type=visualization_type
@@ -313,7 +304,7 @@ class AnalysisStorageService:
await session.flush()
await session.refresh(html_file)
logger.info(f"HTML文件保存成功 RustFS: {html_file.id}")
logger.info(f"HTML文件保存成功 RustFS: {html_file.id} ({object_key})")
return html_file
async def save_features_and_recommendations(
@@ -386,7 +377,6 @@ class AnalysisStorageService:
'geometry_data': None,
'mesh_data': None,
'mold_cavity_data': None,
'html_content': None,
'features': [],
'recommendations': [],
'analysis_metrics': None # 新增分析指标字段
@@ -418,14 +408,10 @@ class AnalysisStorageService:
)
result['mesh_data'] = json.loads(mesh_bytes.decode('utf-8'))
# HTML文件
if stp_file.html_file:
html_bytes = await rustfs_manager.download_file(
file_type='html_files',
object_key=stp_file.html_file.object_key
)
html_json = json.loads(html_bytes.decode('utf-8'))
result['html_content'] = html_json.get('content', '')
# HTML 正文不再随本视图下载(D11):历史实现把整个 HTML 读进
# result['html_content'],但所有调用方只取 geometry/cavity/features,
# HTML 的读取入口是 /html/{filename} 代理路由——每次任务查询白下载
# 数 MB 正文纯属浪费,已删除
except Exception as e:
logger.error(f"从RustFS获取数据失败: {e}")
@@ -0,0 +1,144 @@
# services/occ_process_pool.py
"""常驻 OCC 工作进程池(方案 B,见 docs/topics/performance/OCC_THROUGHPUT.md)。
替代原进程内 `ThreadPoolExecutor(max_workers=1)`:
- 每个工作进程是一个独立 OCC 通道(OCC 非线程安全,通道内串行),常驻不随任务拉起
(spawn 下 import OCC 秒级,按任务拉起会把开销摊到每个任务上)
- 超时/崩溃 = terminate() 换新进程补位——进程边界干净回收(线程级无法击杀 C++ 栈,
旧方案每次超时滞留 1 个线程)
- 输入输出走文件路径 + 普通字典,杜绝 pickle OCC 对象(见 core/occ_worker.py)
"""
import asyncio
import multiprocessing
from typing import Any, Dict, Optional
from moldinsight.core.occ_worker import worker_main
from shared.utils.logger import get_logger
logger = get_logger(__name__)
class _OccWorker:
"""单个 OCC 工作进程的父进程侧封装。"""
def __init__(self, process, conn):
self.process = process
self.conn = conn
self.lock = asyncio.Lock() # 通道串行:同一进程同时只有一个操作在途
async def run(self, op_name: str, payload: Dict[str, Any], timeout: float):
# 阻塞式管道收发放 asyncio.to_thread,不卡事件循环;
# 超时后父进程 terminate() 子进程 → 管道 EOF → 该线程 recv 立即返回,无泄漏
result = await asyncio.wait_for(
asyncio.to_thread(self._run_blocking, op_name, payload),
timeout=timeout,
)
return result
def _run_blocking(self, op_name: str, payload: Dict[str, Any]):
self.conn.send((op_name, payload))
status, data = self.conn.recv()
if status == "error":
raise RuntimeError(data.get("error") or "OCC 子进程操作失败")
return data
class OccProcessPool:
"""OCC 工作进程池(默认 1 进程 = 1 串行通道,与旧单线程语义一致)。
池大小与 celery 并发解耦(celery 并发走多 worker 子进程,每个持自己的池)。
"""
def __init__(self, size: int = 1):
if size < 1:
raise ValueError("size 必须 >= 1")
self._size = size
self._ctx = multiprocessing.get_context("spawn")
self._workers: list[_OccWorker] = []
self._rr = 0
# 保护 workers 列表与轮转指针;操作执行期不持锁
self._pool_lock = asyncio.Lock()
# 任务级整体超时(process_file_with_storage 外层 wait_for)时的在途 worker 追踪
self._busy: Optional[_OccWorker] = None
async def run(self, op_name: str, payload: Dict[str, Any], timeout: float = 600):
for attempt in range(3):
async with self._pool_lock:
await self._ensure_started()
worker = self._workers[self._rr]
self._rr = (self._rr + 1) % len(self._workers)
self._busy = worker
try:
async with worker.lock:
return await worker.run(op_name, payload, timeout)
except asyncio.TimeoutError:
await self._replace(worker)
raise
except Exception as exc:
if not worker.process.is_alive():
# OCC segfault 等进程死亡:换新补位后重试该操作
logger.warning(f"OCC 工作进程异常退出,重试操作 {op_name}: {exc}")
await self._replace(worker)
continue
raise
finally:
if self._busy is worker:
self._busy = None
raise RuntimeError(f"OCC 工作进程连续异常,操作 {op_name} 未能完成")
async def recover(self):
"""任务级整体超时恢复:重建整个池,丢弃可能正卡在挂死 OCC 操作上的进程。
单通道池重建代价可忽略;重建后下次 run 自动按需补拉。
"""
async with self._pool_lock:
for worker in self._workers:
await self._terminate(worker)
self._workers = []
self._busy = None
logger.warning("OCC 进程池已整体重建(任务级超时恢复)")
async def shutdown(self):
async with self._pool_lock:
for worker in self._workers:
await self._terminate(worker)
self._workers = []
self._busy = None
async def _ensure_started(self):
if not self._workers:
for _ in range(self._size):
self._workers.append(self._spawn_one())
logger.info(f"OCC 进程池已启动: {self._size} 个常驻工作进程")
def _spawn_one(self) -> _OccWorker:
parent_conn, child_conn = self._ctx.Pipe(duplex=True)
proc = self._ctx.Process(target=worker_main, args=(child_conn,), daemon=True)
proc.start()
child_conn.close() # 父进程侧关闭子端,只留 parent_conn
return _OccWorker(proc, parent_conn)
async def _replace(self, worker: _OccWorker):
async with self._pool_lock:
await self._terminate(worker)
new_worker = self._spawn_one()
try:
idx = self._workers.index(worker)
except ValueError:
# 已被并发重建移除,新进程追加补位
self._workers.append(new_worker)
else:
self._workers[idx] = new_worker
logger.warning("OCC 工作进程已重建补位")
@staticmethod
async def _terminate(worker: _OccWorker):
try:
worker.process.terminate()
worker.process.join(timeout=5)
except Exception as exc:
logger.warning(f"终止 OCC 工作进程异常: {exc}")
try:
worker.conn.close()
except Exception:
pass
+135 -190
View File
@@ -6,8 +6,6 @@ import os
import shutil
import tempfile
import time
from collections import OrderedDict
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime
from pathlib import Path
from typing import Optional, Dict, Any, List, Tuple
@@ -15,11 +13,8 @@ from typing import Optional, Dict, Any, List, Tuple
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from moldinsight.core.stp_parser import STPParser
from moldinsight.core.geometry_analyzer import GeometryAnalyzer
from moldinsight.core.mesh_generator import MeshGenerator
from moldinsight.core.multi_scheme_planner import MultiSchemeMoldPlanner
from moldinsight.core.cad_exporter import CADExporter
from moldinsight.services.occ_process_pool import OccProcessPool
from moldinsight.services.task_storage_service import TaskStorageService
from moldinsight.services.analysis_storage_service import AnalysisStorageService
from moldinsight.storage.rustfs_storage import rustfs_manager
@@ -40,46 +35,29 @@ class ProcessingService:
"""核心处理流程编排 — 协调 STP 解析、网格、型腔、计算、保存、验证"""
def __init__(self):
self.stp_parser = STPParser()
self.geometry_analyzer = GeometryAnalyzer()
self.mesh_generator = MeshGenerator(quality="medium")
self.html_generator = HTMLGenerator()
# 批次 3 按职责拆分:任务/文件生命周期 与 分析结果数据(原 StorageIntegrationService)
self.task_storage = TaskStorageService()
self.analysis_storage = AnalysisStorageService()
self.multi_scheme_planner = MultiSchemeMoldPlanner()
# cad_exporter 仅用于导出文件条目构建/输出目录(export_artifacts 路径语义);
# 真正的 OCC 形状导出在子进程内完成(见 core/occ_worker.py)
self.cad_exporter = CADExporter()
# TopoDS_Shape 为 C++ 原生内存对象,LRU 上限防止长期运行内存只涨不降
self._export_shapes_cache: "OrderedDict[str, Dict[str, Dict[str, Any]]]" = OrderedDict()
self._export_shapes_cache_max = 32
# OCC 非线程安全,max_workers=1 保证所有 OCC 操作序列化执行,避免偶发崩溃
self._occ_executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="occ")
# D11:HTMLGenerator 不再持有常驻实例,可视化产物统一写任务内临时目录后
# 上传 RustFS 报告键(见 process_file_core)
# OCC 方案 B:所有几何操作(解析/布尔/三角化/倒扣/转换)经常驻 OCC 进程池,
# 进程边界干净回收超时/崩溃——替代原线程级单通道 executor(见 OCC_THROUGHPUT.md)
self._occ_pool = OccProcessPool()
# ─── 对外入口 ───
def _reset_occ_executor(self):
"""超时后重建 OCC executor。
async def run_occ(self, op_name: str, payload: Dict[str, Any],
timeout: float = 600) -> Any:
"""在常驻 OCC 工作进程中执行同步几何操作(方案 B)。
asyncio.wait_for 只能取消协程,正在执行 OCC 布尔运算的线程无法中断;
单 worker executor 中一个挂死线程会让后续任务永久排队直至重启。
cancel_futures=True 丢弃旧 executor 中尚未开跑的排队任务(否则旧线程
恢复后仍会继续消化旧队列,与新 executor 并发操作 OCC 必然崩溃)。
已在运行中的 C++ 线程在 Python 层不可杀,仍会滞留——这是已知残留
泄漏(每次超时 1 线程),根治需进程级 OCC 隔离,见 docs/OCC_THROUGHPUT.md。
OCC 非线程安全,所有几何计算统一经本入口在独立进程中串行执行;
超时/进程崩溃由进程池 terminate + 换新补位,干净回收(见 OCC_THROUGHPUT.md)。
op_name 须已在 occ_worker._OPS 注册;payload 只含文件路径 + 普通字典。
"""
old = self._occ_executor
self._occ_executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="occ")
old.shutdown(wait=False, cancel_futures=True)
logger.warning("OCC executor 已因处理超时重建(排队任务已丢弃,运行中线程可能滞留 1 个)")
async def run_occ(self, fn, *args):
"""在 OCC 单线程 executor 中执行同步几何操作。
OCC 非线程安全,所有几何计算(解析/布尔/三角化)统一经由本入口串行执行,
避免各调用方自行创建线程池造成并发崩溃。
"""
loop = asyncio.get_running_loop()
return await loop.run_in_executor(self._occ_executor, fn, *args)
return await self._occ_pool.run(op_name, payload, timeout=timeout)
async def _materialize_source_file(self, stp_file: STPFile) -> Tuple[Path, Optional[Path]]:
"""把待处理文件落到本地磁盘,返回 (本地路径, 临时目录或 None)。
@@ -164,8 +142,8 @@ class ProcessingService:
)
except asyncio.TimeoutError:
logger.error(f"处理超时: {task_id}")
# OCC 线程无法取消:抛弃整个 executor,避免挂死线程堵死后续所有任务
self._reset_occ_executor()
# OCC 进程无法取消:整体重建进程池,干净杀掉可能卡死的 OCC 操作
await self._occ_pool.recover()
raise Exception(f"处理超时,超过{timeout_seconds}秒未完成")
except Exception as e:
@@ -214,12 +192,9 @@ class ProcessingService:
)
stage_started = time.perf_counter()
loop = asyncio.get_running_loop()
shape = await loop.run_in_executor(
self._occ_executor, self.stp_parser.load_step_file, Path(file_path)
)
geometry_data = await loop.run_in_executor(
self._occ_executor, self.stp_parser.analyze_geometry, shape
# 方案 B:解析 + 几何分析在 OCC 子进程内一步完成(形状不跨进程)
geometry_data = await self.run_occ(
"parse_stp", {"stp_path": file_path}, timeout=timeout_seconds
)
stage_timings["parse_stp"] = round(time.perf_counter() - stage_started, 3)
@@ -230,7 +205,8 @@ class ProcessingService:
stage_started = time.perf_counter()
mesh_result = await self._step_generate_mesh(
shape, geometry_data, file_path, db_session, stp_file_id, task_id
geometry_data, file_path, db_session, stp_file_id, task_id,
timeout=timeout_seconds,
)
stage_timings["generate_mesh"] = round(time.perf_counter() - stage_started, 3)
@@ -246,21 +222,13 @@ class ProcessingService:
is_foam_material = MaterialService.is_foam_material(requested_material)
stage_started = time.perf_counter()
plan_result = await self._step_generate_cavity(
shape, selected_material, is_foam_material, process_params
plan_result, export_artifacts = await self._step_generate_cavity(
file_path, selected_material, is_foam_material, process_params,
task_id, timeout=timeout_seconds,
)
stage_timings["generate_cavity"] = round(time.perf_counter() - stage_started, 3)
export_shapes = {}
export_artifacts = None
if plan_result:
export_shapes = plan_result.pop("_export_shapes", {}) or {}
if export_shapes:
self._cache_export_shapes(task_id, export_shapes)
export_artifacts = self._persist_step_exports(
task_id=task_id,
original_filename=Path(file_path).name,
export_shapes=export_shapes,
)
# 方案 B:各方案形状的持久化 STEP 已由子进程导出并返回 manifest(export_artifacts),
# 主进程不再持有 TopoDS 引用(无跨进程传输)
# 4. 生成详细JSON数据 — 委托 CalculationService
await self.task_storage.update_task_status(
@@ -330,49 +298,60 @@ class ProcessingService:
lod_data = mesh_result
logger.info(f"LOD数据复用成功: {len(lods)} 级 (面数: {[lods[k]['face_count'] for k in sorted(lods.keys())]})")
detailed_cavity_json = await self._attach_scheme_previews(
detailed_cavity_json=detailed_cavity_json,
geometry_data=geometry_data,
stp_filename=Path(file_path).name,
pointcloud_data=pointcloud_data,
lod_data=lod_data,
)
# D11:可视化产物先写任务内临时目录,再统一上传 RustFS 报告键
# (html/reports/{filename}),不再落节点本地 html_output——
# API 与 worker 容器文件系统不互通,本地盘从来不是可依赖的读取来源
html_out_dir = Path(tempfile.mkdtemp(prefix="moldinsight_html_"))
try:
html_generator = HTMLGenerator(output_dir=str(html_out_dir))
best_scheme = CalculationService.get_best_scheme(detailed_cavity_json)
best_cavity_data = best_scheme.get("cavity_data", {}) if best_scheme else best_cavity_data
best_key_info = best_scheme.get("key_info", {}) if best_scheme else best_key_info
detailed_cavity_json = await self._attach_scheme_previews(
detailed_cavity_json=detailed_cavity_json,
geometry_data=geometry_data,
stp_filename=Path(file_path).name,
pointcloud_data=pointcloud_data,
lod_data=lod_data,
html_generator=html_generator,
)
# 8. 保存模具型腔数据(包含方案级预览链接)
await self.analysis_storage.save_mold_cavity_data(
db_session, stp_file_id, detailed_cavity_json
)
best_scheme = CalculationService.get_best_scheme(detailed_cavity_json)
best_cavity_data = best_scheme.get("cavity_data", {}) if best_scheme else best_cavity_data
best_key_info = best_scheme.get("key_info", {}) if best_scheme else best_key_info
html_file_path = self.html_generator.generate_and_save_visualization(
geometry_data,
Path(file_path).name,
cavity_data=best_cavity_data,
pointcloud_data=pointcloud_data,
lod_data=lod_data,
)
# 8. 保存模具型腔数据(包含方案级预览链接)
await self.analysis_storage.save_mold_cavity_data(
db_session, stp_file_id, detailed_cavity_json
)
await self.analysis_storage.save_html_file(
db_session,
stp_file_id,
Path(html_file_path).name,
html_file_path,
)
html_file_path = html_generator.generate_and_save_visualization(
geometry_data,
Path(file_path).name,
cavity_data=best_cavity_data,
pointcloud_data=pointcloud_data,
lod_data=lod_data,
)
await self._upload_report_artifacts(Path(html_file_path))
await self.analysis_storage.save_html_file(
db_session,
stp_file_id,
Path(html_file_path).name,
html_file_path,
)
finally:
shutil.rmtree(html_out_dir, ignore_errors=True)
stage_timings["persist_artifacts"] = round(time.perf_counter() - stage_started, 3)
# 9. 分析模具设计
stage_started = time.perf_counter()
loop = asyncio.get_running_loop()
analysis_result = await loop.run_in_executor(
self._occ_executor,
lambda: self.geometry_analyzer.analyze_mold_design(
geometry_data,
product_material=requested_material,
shape=shape,
),
analysis_result = await self.run_occ(
"analyze_mold_design",
{
"geometry_data": geometry_data,
"product_material": requested_material,
"stp_path": file_path,
},
timeout=timeout_seconds,
)
if analysis_result:
@@ -486,15 +465,15 @@ class ProcessingService:
# ─── 内部步骤 ───
async def _step_generate_mesh(
self, shape, geometry_data: dict, file_path: str,
self, geometry_data: dict, file_path: str,
db_session: AsyncSession, stp_file_id: int, task_id: str,
timeout: float = 600,
) -> Optional[Dict[str, Any]]:
"""生成多级LOD网格并持久化,一次OCC剖分+trimesh简化,失败不影响主流程"""
"""生成多级LOD网格并持久化(方案 B:子进程内一次 OCC 剖分),失败不影响主流程"""
mesh_result = None
try:
loop = asyncio.get_running_loop()
mesh_result = await loop.run_in_executor(
self._occ_executor, self.mesh_generator.generate_multi_lod_mesh, shape
mesh_result = await self.run_occ(
"generate_mesh", {"stp_path": file_path}, timeout=timeout
)
lod0 = mesh_result.get("lods", {}).get("0", {})
@@ -550,97 +529,31 @@ class ProcessingService:
return mesh_result
async def _step_generate_cavity(
self, shape, selected_material: dict, is_foam_material: bool, process_params: Dict[str, Any],
) -> Dict[str, Any]:
"""生成多方案分模结果。
self, file_path: str, selected_material: dict, is_foam_material: bool,
process_params: Dict[str, Any], task_id: str, timeout: float = 600,
) -> Tuple[Dict[str, Any], Optional[Dict[str, Any]]]:
"""生成多方案分模结果(方案 B:子进程内完成分模 + 方案形状 STEP 导出)。
D8:型腔是任务的核心产出,生成失败必须让任务 failed——
此前异常在此被吞掉置 plan_result=None 继续主流程,最终任务
completed,"完成"状态不可信。异常直接向编排层传播。
异常直接向编排层传播。返回 (plan_result, export_manifest)。
"""
if not shape:
raise RuntimeError("无有效几何 shape,无法生成模具型腔")
loop = asyncio.get_running_loop()
plan_result = await loop.run_in_executor(
self._occ_executor,
lambda: self.multi_scheme_planner.generate_plan(
shape=shape,
material=selected_material,
is_foam_material=is_foam_material,
process_params=process_params,
),
result = await self.run_occ(
"generate_cavity",
{
"stp_path": file_path,
"task_id": task_id,
"material": selected_material,
"is_foam_material": is_foam_material,
"process_params": process_params,
"export_out_dir": os.path.abspath(self.cad_exporter.output_dir),
},
timeout=timeout,
)
plan_result = result["plan_result"]
logger.info(
f"多方案分模完成: 生成 {len(plan_result.get('candidate_schemes', []))} 套方案"
)
return plan_result
def _cache_export_shapes(self, task_id: str, export_shapes: Dict[str, Dict[str, Any]]):
self._export_shapes_cache[task_id] = export_shapes
self._export_shapes_cache.move_to_end(task_id)
# 逐出最旧任务的形状缓存(连原生 OCC shape 引用一起释放)
while len(self._export_shapes_cache) > self._export_shapes_cache_max:
self._export_shapes_cache.popitem(last=False)
def _persist_step_exports(
self,
task_id: str,
original_filename: str,
export_shapes: Dict[str, Dict[str, Any]],
) -> Optional[Dict[str, Any]]:
if not export_shapes:
return None
base_filename = Path(original_filename).stem or f"mold_{task_id}"
manifest = {
"version": 1,
"task_id": task_id,
"generated_at": datetime.now().isoformat(),
"schemes": {},
}
components = ["cavity", "core", "parting_surface", "product", "a_plate", "b_plate"]
for scheme_id, cavity_data in export_shapes.items():
try:
# 持久化装配体 STEP + 逐组件 STEP(后者是重启后按需
# 重导出其他格式的几何来源,见 regenerate_export_from_persisted)
result = self.cad_exporter.export_persisted_steps(
cavity_data=cavity_data,
base_filename=base_filename,
components=components,
task_id=task_id,
scheme_id=scheme_id,
)
manifest["schemes"][scheme_id] = {
"base_filename": result.get("base_filename"),
"generated_at": datetime.now().isoformat(),
"files": result.get("files", []),
"errors": result.get("errors", []),
"total_files": result.get("total_files", 0),
"total_errors": result.get("total_errors", 0),
}
except Exception as exc:
logger.warning("持久化 STEP 导出失败: task=%s scheme=%s error=%s", task_id, scheme_id, exc)
manifest["schemes"][scheme_id] = {
"base_filename": base_filename,
"generated_at": datetime.now().isoformat(),
"files": [],
"errors": [str(exc)],
"total_files": 0,
"total_errors": 1,
}
return manifest
def get_export_shapes(self, task_id: str, scheme_id: Optional[str] = None) -> Optional[Dict[str, Any]]:
scheme_map = self._export_shapes_cache.get(task_id)
if not scheme_map:
return None
self._export_shapes_cache.move_to_end(task_id) # LRU 热点保活
if scheme_id:
return scheme_map.get(scheme_id)
return next(iter(scheme_map.values()), None)
return plan_result, result["export_manifest"]
async def regenerate_export_from_persisted(
self,
@@ -651,11 +564,12 @@ class ProcessingService:
base_filename: str,
scheme_files: List[Dict[str, Any]],
) -> Optional[Dict[str, Any]]:
"""内存导出缓存失效(如服务重启)后,从持久化 STEP 重建导出文件。
"""持久化 STEP 缓存失效(如服务重启)后,从持久化 STEP 重建导出文件。
分析期已为每个方案持久化装配体 + 逐组件 STEP;
缺失格式(STL/IGES/BRep)读回单组件 STEP 现场转换,
用户无需重新分析。所有组件均不可用时返回 None。
方案 B 后主进程不再持有 TopoDS 形状(导出持久化发生在子进程),
内存缓存路径已随旧 _cache_export_shapes/get_export_shapes 一并删除;
本方法读回单组件 STEP 现场转换缺失格式,用户无需重新分析。
所有组件均不可用时返回 None。
"""
format_list = list(dict.fromkeys(formats or ["step", "stl"]))
step_files = {
@@ -693,7 +607,8 @@ class ProcessingService:
os.path.dirname(step_path), f"{base_filename}_{comp}.{fmt}"
)
ok = await self.run_occ(
self.cad_exporter.convert_component_step, step_path, out_path, fmt
"convert_component_step",
{"step_path": str(step_path), "out_path": str(out_path), "fmt": fmt},
)
if ok:
files.append(
@@ -763,8 +678,13 @@ class ProcessingService:
stp_filename: str,
pointcloud_data: Optional[Dict[str, Any]] = None,
lod_data: Optional[Dict[str, Any]] = None,
html_generator: HTMLGenerator = None,
) -> Dict[str, Any]:
"""为候选分模方案生成轻量摘要链接(完整HTML仅最优方案按需生成)"""
if html_generator is None:
raise ValueError(
"html_generator 不能为空(D11:摘要产物统一经任务临时目录上传 RustFS)"
)
candidate_schemes = detailed_cavity_json.get("candidate_schemes", [])
if not candidate_schemes:
return detailed_cavity_json
@@ -777,10 +697,15 @@ class ProcessingService:
base_stem = Path(stp_filename).stem.replace(" ", "_")
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
summary_name = f"mold_{base_stem}_{suffix}_{ts}_summary.json"
summary_content = self.html_generator.generate_3d_viewer_summary(
summary_content = html_generator.generate_3d_viewer_summary(
geometry_data, cavity_data
)
self.html_generator.save_data_file(summary_content, summary_name)
# D11:摘要 JSON 写任务临时目录后直传 RustFS 报告键,不落本地 html_output
summary_path = html_generator.save_data_file(summary_content, summary_name)
await rustfs_manager.upload_report_artifact(
summary_name, Path(summary_path).read_bytes(),
content_type="application/json",
)
scheme["summary_file"] = f"/html/{summary_name}"
best_scheme = CalculationService.get_best_scheme(detailed_cavity_json)
@@ -789,6 +714,26 @@ class ProcessingService:
return detailed_cavity_json
async def _upload_report_artifacts(self, html_file_path: Path):
"""D11:任务临时目录中的可视化产物(.html / _summary.json / _data.json)
上传 RustFS 报告键(html/reports/{filename})。
HTML 内嵌相对 DATA_URL/SUMMARY_URL 引用两个 JSON,
三者必须同键前缀可达(读侧 /html/{filename} 直取)。
"""
stem = html_file_path.with_suffix("")
artifacts = [
(html_file_path, "text/html; charset=utf-8"),
(stem.with_name(stem.name + "_summary.json"), "application/json"),
(stem.with_name(stem.name + "_data.json"), "application/json"),
]
for path, content_type in artifacts:
if not path.exists():
raise RuntimeError(f"可视化产物缺失: {path.name}")
await rustfs_manager.upload_report_artifact(
path.name, path.read_bytes(), content_type=content_type
)
# ─── 指标持久化 ───
async def _save_analysis_metrics(self, session: AsyncSession, stp_file_id: int, analysis_result: dict):
-88
View File
@@ -1,88 +0,0 @@
# services/shape_loader.py
"""按 task_id 从持久化存储重建 OCC 几何形状。
任务完成后 TopoDS_Shape 不驻留内存/Redis(原生内存与体积原因),
需要几何的端点(倒扣检测、按需重导出等)通过 STP 原件重建:
PG(object_key) -> RustFS 下载 -> 临时文件 -> OCC 单线程 executor 解析。
"""
import tempfile
from pathlib import Path
from typing import Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from moldinsight.models import ProcessingTask, STPFile
from shared.utils.logger import get_logger
logger = get_logger(__name__)
class ShapeLoader:
"""任务几何重建器"""
def __init__(self):
from moldinsight.core.stp_parser import STPParser
from moldinsight.services.processing_service import processing_service
self._parser = STPParser()
self._processing = processing_service
async def load_shape_for_task(
self, db_session: AsyncSession, task_id: str
) -> Optional["object"]:
"""重建任务的产品几何。任务不存在或 STP 原件不可用时返回 None。"""
result = await db_session.execute(
select(ProcessingTask, STPFile)
.join(STPFile, ProcessingTask.stp_file_id == STPFile.id)
.where(ProcessingTask.task_id == task_id)
)
row = result.first()
if not row:
logger.warning(f"几何重建失败:任务不存在 {task_id}")
return None
_, stp_file = row
if not stp_file.object_key:
logger.warning(f"几何重建失败:任务缺少 object_key {task_id}")
return None
from moldinsight.storage.rustfs_storage import rustfs_manager
try:
data = await rustfs_manager.download_file(
file_type="stp_files", object_key=stp_file.object_key
)
except Exception as exc:
logger.error(f"几何重建失败:STP 原件下载失败 {task_id}: {exc}")
return None
with tempfile.NamedTemporaryFile(suffix=".stp", delete=False) as tmp:
tmp.write(data)
tmp_path = Path(tmp.name)
try:
shape = await self._processing.run_occ(
self._parser.load_step_file, tmp_path
)
return shape
except Exception as exc:
logger.error(f"几何重建失败:STP 解析失败 {task_id}: {exc}")
return None
finally:
try:
tmp_path.unlink(missing_ok=True)
except Exception:
pass
# 惰性单例:__init__ 会实例化 STPParser 并校验 OCC 可用性,
# 延迟到首次真实使用,避免模块导入期失败拖垮路由加载
_shape_loader: Optional[ShapeLoader] = None
def get_shape_loader() -> ShapeLoader:
global _shape_loader
if _shape_loader is None:
_shape_loader = ShapeLoader()
return _shape_loader
@@ -0,0 +1,76 @@
# services/stp_materializer.py
"""按 task_id 把 STP 原件从持久化存储落盘为临时文件(方案 B)。
任务完成后 TopoDS_Shape 不驻留内存/Redis;需要几何的端点(倒扣检测、
按需重导出等)通过 STP 原件重建:PG(object_key) -> RustFS 下载 -> 临时文件,
OCC 解析在常驻子进程内完成(occ_worker 的 detect_undercuts 等操作,见
occ_process_pool)。本模块只负责把原件落到调用方可传路径的临时文件。
"""
import tempfile
from pathlib import Path
from typing import Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from moldinsight.models import ProcessingTask, STPFile
from shared.utils.logger import get_logger
logger = get_logger(__name__)
class STPMaterializer:
"""任务 STP 原件落盘器"""
async def materialize_stp_for_task(
self, db_session: AsyncSession, task_id: str
) -> Optional[Path]:
"""重建任务的产品几何原件为临时文件。
任务不存在或 STP 原件不可用时返回 None;返回的临时文件由调用方
finally 清理(unlink)。
"""
result = await db_session.execute(
select(ProcessingTask, STPFile)
.join(STPFile, ProcessingTask.stp_file_id == STPFile.id)
.where(ProcessingTask.task_id == task_id)
)
row = result.first()
if not row:
logger.warning(f"STP 原件落盘失败:任务不存在 {task_id}")
return None
_, stp_file = row
if not stp_file.object_key:
logger.warning(f"STP 原件落盘失败:任务缺少 object_key {task_id}")
return None
from moldinsight.storage.rustfs_storage import rustfs_manager
try:
data = await rustfs_manager.download_file(
file_type="stp_files", object_key=stp_file.object_key
)
except Exception as exc:
logger.error(f"STP 原件落盘失败:下载失败 {task_id}: {exc}")
return None
original_name = Path(stp_file.original_filename or "model.stp").name or "model.stp"
tmp = tempfile.NamedTemporaryFile(
suffix=Path(original_name).suffix or ".stp", prefix=f"mold_{task_id}_", delete=False
)
tmp.write(data)
tmp.close()
logger.debug(f"STP 原件已落盘: {tmp.name}")
return Path(tmp.name)
# 惰性单例
_stp_materializer: Optional[STPMaterializer] = None
def get_stp_materializer() -> STPMaterializer:
global _stp_materializer
if _stp_materializer is None:
_stp_materializer = STPMaterializer()
return _stp_materializer
+67
View File
@@ -33,6 +33,8 @@ class RustFSManager:
'html_files': 'html',
'user_files': 'user-files'
}
# 可视化报告产物固定前缀(D11:文件名寻址,读侧 /html/{filename} 代理按键直取)
self.report_prefix = "html/reports"
async def connect(self, endpoint: str, access_key: str, secret_key: str, timeout: int = 30):
"""连接到 RustFS 服务"""
@@ -223,6 +225,71 @@ class RustFSManager:
logger.error(f"RustFS JSON上传失败: {e}")
raise
async def upload_report_artifact(self, filename: str, data: bytes,
content_type: str = "application/octet-stream") -> str:
"""可视化报告产物上传(D11:RustFS 为报告唯一持久来源)。
固定键 html/reports/{filename}——文件名寻址(非生成键),
读侧 /html/{filename} 代理按键直取,无需元数据表参与。
filename 由调用方保证全局唯一(命名含源文件 stem + 时间戳);
同源同秒重复分析覆盖旧对象(重新分析语义即报告刷新)。
"""
import asyncio
if not self.is_connected:
raise RuntimeError("RustFS 未连接")
object_key = self._report_object_key(filename)
def _upload():
return self.client.put_object(
self.bucket_name,
object_key,
BytesIO(data),
length=len(data),
content_type=content_type,
)
try:
await asyncio.to_thread(_upload)
logger.info(f"报告产物上传成功 RustFS: {self.bucket_name}/{object_key} ({len(data)} bytes)")
return object_key
except S3Error as e:
logger.error(f"RustFS 报告产物上传失败: {e}")
raise
async def download_report_artifact(self, filename: str) -> bytes:
"""按文件名读取报告产物;对象不存在/未连接时抛异常,由调用方决定回退。"""
import asyncio
if not self.is_connected:
raise RuntimeError("RustFS 未连接")
object_key = self._report_object_key(filename)
def _download():
resp = self.client.get_object(self.bucket_name, object_key)
try:
return resp.read()
finally:
resp.close()
resp.release_conn()
try:
data = await asyncio.to_thread(_download)
logger.debug(f"报告产物下载成功: {self.bucket_name}/{object_key}")
return data
except S3Error as e:
logger.error(f"RustFS 报告产物下载失败: {e}")
raise
def _report_object_key(self, filename: str) -> str:
"""报告产物键构造:防路径穿越,键必须落在 html/reports/ 一级之下。"""
safe_name = Path(filename).name
if safe_name != filename or filename.startswith("."):
raise ValueError(f"非法报告文件名: {filename!r}")
return f"{self.report_prefix}/{safe_name}"
async def download_file(self, file_type: str, object_key: str) -> bytes:
"""从 RustFS 下载文件"""
import asyncio
+5 -11
View File
@@ -27,7 +27,7 @@ def create_app(
title: str,
service_name: str,
version: str = "4.0.0",
mount_html: bool = False,
connect_rustfs: bool = False,
serve_frontend_static: bool = False,
startup_hooks: Optional[List[Callable[[], Awaitable[None]]]] = None,
register_routers: Optional[Callable[[FastAPI], None]] = None,
@@ -38,7 +38,9 @@ def create_app(
title: 应用标题
service_name: 服务名(用于 /health 响应)
version: 版本号
mount_html: 是否挂载 /html 静态目录(moldinsight 需要)
connect_rustfs: 是否连接 RustFS 对象存储(moldinsight 需要)。
D11:/html 报告读取由 moldinsight.api.html_report_router 代理
(RustFS 报告键直取 + 遗留对象/本地卷兜底),不再挂本地 StaticFiles。
serve_frontend_static: 是否由后端托管 /static 与 SPA fallback(默认关闭,前端独立部署)
startup_hooks: 额外的 startup 钩子列表(在数据库/RustFS/Redis 初始化后执行)
register_routers: 回调函数,用于注册业务路由
@@ -96,8 +98,6 @@ def create_app(
Path("uploads").mkdir(exist_ok=True)
if serve_frontend_static:
Path("static").mkdir(exist_ok=True)
if mount_html:
Path("html_output").mkdir(exist_ok=True)
# ── 静态文件挂载 ─────────────────────────────────────────────
if serve_frontend_static:
@@ -106,12 +106,6 @@ def create_app(
StaticFiles(directory=os.path.join(os.getcwd(), "static")),
name="static",
)
if mount_html:
app.mount(
"/html",
StaticFiles(directory=os.path.join(os.getcwd(), "html_output")),
name="html",
)
# ── Startup ──────────────────────────────────────────────────
@app.on_event("startup")
@@ -122,7 +116,7 @@ def create_app(
print(f"[{'OK' if success else 'FAIL'}] 数据库初始化")
# RustFS(仅 moldinsight 需要)
if mount_html:
if connect_rustfs:
try:
from moldinsight.storage.rustfs_storage import rustfs_manager
await rustfs_manager.connect(
+20
View File
@@ -0,0 +1,20 @@
"""D2:铝价数据必须显式声明模拟来源。
数据为模拟走势(aluminum_price_service 模块 docstring 自述),
此前接口不声明来源、前端硬编码"数据来源: 上海期货交易所",属虚假来源声明。
"""
def test_current_price_declares_simulated_source():
from moldinsight.services.aluminum_price_service import get_aluminum_current_price
data = get_aluminum_current_price()
assert data["source"] == "simulated"
def test_history_items_declare_simulated_source():
from moldinsight.services.aluminum_price_service import get_aluminum_price_history
history = get_aluminum_price_history(days=30)
assert len(history) > 0
assert all(item["source"] == "simulated" for item in history)
+176
View File
@@ -0,0 +1,176 @@
"""D11:/html/{filename} 报告代理路由。
解析链:RustFS 报告键(新产物裸文件)→ HTMLFile 遗留记录(JSON 包装)
→ 节点本地 html_output(存量兜底)→ 404;路径穿越一律 404。
"""
import json
from types import SimpleNamespace
import pytest
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
from moldinsight.api import html_report_router
from moldinsight.storage.rustfs_storage import rustfs_manager
@pytest.fixture
def app():
application = FastAPI()
application.include_router(html_report_router.router)
return application
@pytest.fixture
async def client(app):
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as c:
yield c
@pytest.fixture(autouse=True)
def _isolate_external_sources(monkeypatch, tmp_path):
"""默认隔离:RustFS 未连接、本地兜底目录指向空临时目录,由各测试按需打开。"""
monkeypatch.setattr(rustfs_manager, "is_connected", False)
monkeypatch.setattr(html_report_router, "LOCAL_HTML_DIR", tmp_path / "html_output")
# ── 链路 1:RustFS 报告键(新产物) ──────────────────────────────
async def test_report_served_from_rustfs_report_key(client, monkeypatch):
async def fake_download_report(filename):
assert filename == "mold_demo_20260918.html"
return b"<html>fresh</html>"
monkeypatch.setattr(rustfs_manager, "is_connected", True)
monkeypatch.setattr(rustfs_manager, "download_report_artifact", fake_download_report)
resp = await client.get("/html/mold_demo_20260918.html")
assert resp.status_code == 200
assert resp.content == b"<html>fresh</html>"
assert "text/html" in resp.headers["content-type"]
# ── 链路 2:HTMLFile 遗留记录(html/{hash}.json JSON 包装) ──────
class _FakeScalars:
def __init__(self, record):
self._record = record
def first(self):
return self._record
class _FakeResult:
def __init__(self, record):
self._record = record
def scalars(self):
return _FakeScalars(self._record)
class _FakeSession:
def __init__(self, record):
self._record = record
async def __aenter__(self):
return self
async def __aexit__(self, *exc):
return False
async def execute(self, stmt):
return _FakeResult(self._record)
class _FakeDBManager:
def __init__(self, record):
self._record = record
def session(self):
return _FakeSession(self._record)
async def test_falls_back_to_legacy_json_wrapper(client, monkeypatch):
async def fake_download_report(filename):
raise RuntimeError("报告键未命中")
async def fake_download_file(file_type, object_key):
assert file_type == "html_files"
assert object_key == "html/abc123.json"
return json.dumps(
{"content": "<html>legacy</html>", "filename": "mold_old.html"}
).encode("utf-8")
record = SimpleNamespace(object_key="html/abc123.json", filename="mold_old.html")
monkeypatch.setattr(rustfs_manager, "is_connected", True)
monkeypatch.setattr(rustfs_manager, "download_report_artifact", fake_download_report)
monkeypatch.setattr(rustfs_manager, "download_file", fake_download_file)
monkeypatch.setattr(html_report_router, "db_manager", _FakeDBManager(record))
resp = await client.get("/html/mold_old.html")
assert resp.status_code == 200
assert resp.content == b"<html>legacy</html>"
async def test_legacy_record_with_report_key_returns_raw(client, monkeypatch):
"""新格式记录(键在报告前缀下)必须按裸文件返回,不能当 JSON 包装解析。"""
async def fake_download_report(filename):
raise RuntimeError("报告键直取瞬时失败")
async def fake_download_file(file_type, object_key):
assert object_key == "html/reports/mold_new.html"
return b"<html>raw-by-report-key</html>"
record = SimpleNamespace(object_key="html/reports/mold_new.html", filename="mold_new.html")
monkeypatch.setattr(rustfs_manager, "is_connected", True)
monkeypatch.setattr(rustfs_manager, "download_report_artifact", fake_download_report)
monkeypatch.setattr(rustfs_manager, "download_file", fake_download_file)
monkeypatch.setattr(html_report_router, "db_manager", _FakeDBManager(record))
resp = await client.get("/html/mold_new.html")
assert resp.status_code == 200
assert resp.content == b"<html>raw-by-report-key</html>"
# ── 链路 3:节点本地 html_output(存量兜底) ─────────────────────
async def test_falls_back_to_local_dir(client, tmp_path):
local_dir = tmp_path / "html_output"
local_dir.mkdir()
(local_dir / "mold_local.html").write_text("<html>local</html>", encoding="utf-8")
resp = await client.get("/html/mold_local.html")
assert resp.status_code == 200
assert resp.content == b"<html>local</html>"
async def test_json_media_type_from_extension(client, tmp_path):
local_dir = tmp_path / "html_output"
local_dir.mkdir()
(local_dir / "mold_a_data.json").write_bytes(b"{}")
resp = await client.get("/html/mold_a_data.json")
assert resp.status_code == 200
assert "application/json" in resp.headers["content-type"]
# ── 未命中与防护 ────────────────────────────────────────────────
async def test_404_when_all_sources_miss(client):
resp = await client.get("/html/missing.html")
assert resp.status_code == 404
async def test_path_traversal_rejected(client):
resp = await client.get("/html/%2e%2e/secret.txt")
assert resp.status_code == 404
async def test_hidden_filename_rejected(client):
resp = await client.get("/html/%2e%2eetc%2fpasswd")
assert resp.status_code == 404
+116
View File
@@ -0,0 +1,116 @@
"""OCC 方案 B(常驻进程池)契约测试,见 docs/topics/performance/OCC_THROUGHPUT.md。
覆盖:spawn 子进程 + 管道往返、操作错误回传、超时换新补位、真实 OCC 解析
(盒体 STP 端到端)。OCC(pythonocc)仅 conda 环境提供,无 OCC 环境自动跳过。
"""
import os
from pathlib import Path
import pytest
pytest.importorskip("OCC", reason="需要 pythonocc 运行 OCC 进程池测试")
from moldinsight.services.occ_process_pool import OccProcessPool
@pytest.fixture(scope="module", autouse=True)
def _child_pythonpath():
"""spawn 子进程不继承 pytest 的 sys.path,需经 PYTHONPATH 传递 src 目录。"""
src_dir = str(Path(__file__).resolve().parent.parent / "src")
prev = os.environ.get("PYTHONPATH", "")
os.environ["PYTHONPATH"] = src_dir if not prev else f"{src_dir}{os.pathsep}{prev}"
yield
if prev:
os.environ["PYTHONPATH"] = prev
else:
os.environ.pop("PYTHONPATH", None)
@pytest.fixture
async def pool():
_pool = OccProcessPool()
try:
yield _pool
finally:
await _pool.shutdown()
async def test_ping_roundtrip(pool):
result = await pool.run("ping", {})
assert result == {"pong": True}
async def test_unknown_op_raises(pool):
with pytest.raises(RuntimeError, match="未知 OCC 操作"):
await pool.run("no_such_op", {})
async def test_child_error_surfaces(pool):
# parse_stp 指向不存在的文件:子进程内抛错,父进程应收到 RuntimeError
with pytest.raises(RuntimeError):
await pool.run("parse_stp", {"stp_path": "D:/no/such/file.stp"})
async def test_timeout_replaces_worker(pool):
# 子进程挂 15s,父进程 1s 超时 → 换新补位后池仍可用
with pytest.raises(Exception, match=".*"):
await pool.run("sleep", {"seconds": 15}, timeout=1)
# 新 worker 上 ping 正常
result = await pool.run("ping", {})
assert result == {"pong": True}
def _write_box_stp(path: Path) -> Path:
from OCC.Core.BRepPrimAPI import BRepPrimAPI_MakeBox
from OCC.Core.IFSelect import IFSelect_RetDone
from OCC.Core.STEPControl import STEPControl_AsIs, STEPControl_Writer
from OCC.Core.gp import gp_Pnt
box = BRepPrimAPI_MakeBox(gp_Pnt(0, 0, 0), 100.0, 60.0, 40.0).Shape()
writer = STEPControl_Writer()
writer.Transfer(box, STEPControl_AsIs)
assert writer.Write(str(path)) == IFSelect_RetDone
return path
async def test_parse_stp_end_to_end(pool, tmp_path):
stp_path = _write_box_stp(tmp_path / "box.stp")
geometry = await pool.run("parse_stp", {"stp_path": str(stp_path)})
assert geometry["volume"] == pytest.approx(100 * 60 * 40, rel=0.01)
assert "bounding_box" in geometry
async def test_generate_cavity_persists_export_steps(pool, tmp_path):
"""最复杂的迁移点:分模 + 方案形状 STEP 导出全部在子进程内完成,
TopoDS 不跨进程传输,export_manifest 携带落盘文件清单。"""
from moldinsight.services.material_service import MaterialService
stp_path = _write_box_stp(tmp_path / "box.stp")
export_dir = tmp_path / "exports"
result = await pool.run(
"generate_cavity",
{
"stp_path": str(stp_path),
"task_id": "test-task",
"material": dict(MaterialService.get_material("ABS")),
"is_foam_material": False,
"process_params": None,
"export_out_dir": str(export_dir),
},
timeout=120,
)
plan_result = result["plan_result"]
schemes = plan_result["candidate_schemes"]
assert 1 <= len(schemes) <= 3
assert plan_result["best_scheme_id"] == schemes[0]["scheme_id"]
# TopoDS 形状不得跨进程(子进程已 pop 掉 _export_shapes)
assert "_export_shapes" not in plan_result
manifest = result["export_manifest"]
assert manifest is not None
assert manifest["schemes"]
best = manifest["schemes"][plan_result["best_scheme_id"]]
assert best["total_files"] >= 1
# 落盘文件真实存在于子进程写入的 export_out_dir
assert any((export_dir / f.get("relative_path", "")).exists() for f in best["files"])