From f441f1810edf3005429b5436109a903d4c9c2d37 Mon Sep 17 00:00:00 2001 From: Yuyi-Oak <1722157266@qq.com> Date: Sun, 30 Aug 2026 21:13:31 +0800 Subject: [PATCH] =?UTF-8?q?feat(i18n):=20=E6=8E=A5=E5=85=A5=E7=BF=BB?= =?UTF-8?q?=E8=AF=91=20provider=20worker?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #44 --- CURRENT_STATUS.md | 14 +- PROJECT_PLAN.md | 16 +- README.md | 4 +- USERGUIDE.md | 8 +- api/openapi/bat-api.yaml | 170 +- docs/architecture/assetbundle.md | 4 +- .../architecture/official-resource-backend.md | 5 +- docs/guides/baseline.md | 4 +- docs/guides/bat-workflows.md | 38 +- docs/guides/development.md | 4 + docs/reference/rpc-backend-api.md | 48 +- docs/reports/CURRENT_GAPS.md | 8 +- docs/reports/GO_STATUS.md | 2 +- infrastructure/src/bin/bat/app.rs | 548 ++++++- infrastructure/src/bin/bat/app_tests.rs | 159 ++ infrastructure/src/bin/bat/task_registry.rs | 94 +- .../src/bin/bat/workflow_commands.rs | 37 + infrastructure/src/downloader.rs | 12 +- infrastructure/src/lib.rs | 17 +- infrastructure/src/translation_tasks.rs | 705 ++++++++- infrastructure/src/translation_worker.rs | 1374 +++++++++++++++++ internal/api/admin.go | 168 +- internal/api/api_test.go | 124 +- internal/api/openapi.go | 104 +- internal/api/rpc_release.go | 15 + internal/api/server.go | 4 + internal/backendrpc/client.go | 66 + internal/backendrpc/client_test.go | 157 ++ web/admin/README.md | 36 +- 29 files changed, 3815 insertions(+), 130 deletions(-) create mode 100644 infrastructure/src/translation_worker.rs diff --git a/CURRENT_STATUS.md b/CURRENT_STATUS.md index 75a89f4..3f3d8bd 100644 --- a/CURRENT_STATUS.md +++ b/CURRENT_STATUS.md @@ -28,9 +28,9 @@ Rust 侧官方日服资源链路已经从实验验证推进到正式入口： 11. 官方同步会拒绝危险输出目录、路径逃逸和现有 symlink 路径组件；下载目标、`.part`、manifest、snapshot、PID、status、log 和控制锁文件不会跟随 symlink，daemon 状态类文件默认以 `0600` 权限创建。 12. `/official-version-state.json` 会明确保存当前已完成版本、正在拉取版本、上一个可用版本和失败版本；同一 app version、bundle version 和 Addressables root 的失败只保留最新一条，同一版本开始重新拉取或后续发布成功时会清理对应失败记录；`bat status` 会显示最后成功时间、下次检查时间、最后错误摘要、当前阶段、当前下载 URL 进度、版本状态摘要、最近历史失败版本和原因、结构化日志路径和轮转日志路径，人类可读输出不会把完整版本状态 JSON 内联打印。 13. 资源导入链路已支持 CAS + `ResourceRepository` 索引写入，官方同步可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后触发导入，默认 CAS 路径为 `/.cas`、SQLite 索引为 `/resources.sqlite`，也可通过 `--import-cas-root`、`--import-resource-db`、`BAT_IMPORT_CAS_ROOT`、`BAT_IMPORT_RESOURCE_DB` 覆盖；`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 分页查询现有索引，数据库不存在时返回 `available=false` 且不会创建空库。`Resource` metadata 已通过 `metadata_json` 兼容迁移保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式；当前/上一个/结构变化 catalog、失败 staging 复用、403/404、hash mismatch、CRC、metadata 迁移与 UnityFS 边界校验均有离线回归 fixture 或单测覆盖。 -14. 非 dry-run 官方同步在校验完成并发布后，会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`，在当前 release 下写入 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`；同一 destination 只有 size 或 BLAKE3 改变才算 modified，仅 URL/CDN 根变化但内容相同不会触发解析/翻译候选。随后刷新 `official-parse-cache.json` 和 `official-textunit-index.json`，并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`；删除资源只进入差异记录，不进入 TextUnit/Crowdin 队列。`parse.text_units` 和 `parse.errors` RPC/CLI 可按 destination、archive entry、path id、class id、field path 和 format 查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` RPC/CLI 可按 release、destination、archive entry、任务状态、parse status、TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因；`translation.task.update` 可回写 provider worker 状态，`translation.proofread` 可把汉化 workflow 标记为人工校对中；TextUnit 已包含 class id、field path、字段 offset/byte size 等可追溯定位。Crowdin 当前仅预留本地离线队列，不发网络请求；up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要，不重复解析。官方同步报告默认 `localized_release_status=not_localized`，含义是原版资源已经发布、汉化资源未发布；UnityFS TextAsset patch 发布成功后会写 `localized-patch-manifest.json`，校验 hash/size/diff/rollback 后才允许 `localized.status` 返回 `status=published`、`status_code=localized.published` 和 `localized_release_status=localized`。`bat` 首次启动会在二进制所在目录释放 `.env` 配置模板（`0600`），之后每次启动自动加载（不覆盖已存在的环境变量），支持 `BAT_OUTPUT`/`BAT_LOCALIZED_OUTPUT`/`BAT_IMPORT_REPOSITORY`/`BAT_IMPORT_CAS_ROOT`/`BAT_IMPORT_RESOURCE_DB`/`BAT_STATE_DIR`/`BAT_AUTO_DISCOVER`/`BAT_WATCH`/`BAT_DAEMON`/`BAT_PROXY` 等键，实现编辑 `.env` 后无参启动；优先级为命令行参数 > 进程环境变量 > `.env` > 内置默认值，`BAT_SKIP_ENV_FILE=1` 可整体禁用；Redis 键为预留。daemon 任务历史持久化在 `/bat-tasks.json`（版本化、`0600` 原子写），重启后任务经 `task.*` 仍可查，中断任务标记 `task_interrupted`（`BAT-ERR-700005`）。 +14. 非 dry-run 官方同步在校验完成并发布后，会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`，在当前 release 下写入 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`；同一 destination 只有 size 或 BLAKE3 改变才算 modified，仅 URL/CDN 根变化但内容相同不会触发解析/翻译候选。随后刷新 `official-parse-cache.json` 和 `official-textunit-index.json`，并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`；删除资源只进入差异记录，不进入 TextUnit/Crowdin 队列。`parse.text_units` 和 `parse.errors` RPC/CLI 可按 destination、archive entry、path id、class id、field path 和 format 查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` RPC/CLI 可按 release、destination、archive entry、任务状态、parse status、TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因；`translation.task.update` 可回写 provider worker 状态，`translation.worker.run` 可触发 Rust provider worker 独立 claim/lease/retry 并落库 TextUnit 级译文结果，`translation.proofread` 可把汉化 workflow 标记为人工校对中；TextUnit 已包含 class id、field path、字段 offset/byte size 等可追溯定位。Crowdin provider 通过 `CROWDIN_*` 环境变量接入，mock provider 支持本地 fixture；up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要，不重复解析。官方同步报告默认 `localized_release_status=not_localized`，含义是原版资源已经发布、汉化资源未发布；UnityFS TextAsset patch 发布成功后会写 `localized-patch-manifest.json`，校验 hash/size/diff/rollback 后才允许 `localized.status` 返回 `status=published`、`status_code=localized.published` 和 `localized_release_status=localized`。`bat` 首次启动会在二进制所在目录释放 `.env` 配置模板（`0600`），之后每次启动自动加载（不覆盖已存在的环境变量），支持 `BAT_OUTPUT`/`BAT_LOCALIZED_OUTPUT`/`BAT_IMPORT_REPOSITORY`/`BAT_IMPORT_CAS_ROOT`/`BAT_IMPORT_RESOURCE_DB`/`BAT_STATE_DIR`/`BAT_AUTO_DISCOVER`/`BAT_WATCH`/`BAT_DAEMON`/`BAT_PROXY` 和 `BAT_TRANSLATION_*` 等键，实现编辑 `.env` 后无参启动；优先级为命令行参数 > 进程环境变量 > `.env` > 内置默认值，`BAT_SKIP_ENV_FILE=1` 可整体禁用；Redis 键为预留。daemon 任务历史持久化在 `/bat-tasks.json`（版本化、`0600` 原子写），重启后任务经 `task.*` 仍可查，中断任务标记 `task_interrupted`（`BAT-ERR-700005`）。 -15. issue 43 已补齐 Rust `bat` 的 `res` / `parse` / `i18n` 工作流入口：支持资源拉取、解析刷新、可再生解析缓存清理、离线翻译工作台、人工文本查看/修改/清空、工作台发布前校验、有限 TextAsset 汉化发布、人工校对状态标记、既有 patch 能力的批量重打包、单次/限定次数/周期执行和版本化 schedule CRUD。schedule 查询现在按一级工作流过滤，删除/执行会校验作用域，单轮执行可限制计划数；schedule CRUD、翻译任务状态回写和 `translation.proofread` 状态标记已通过 `bat.sock` 的 RPC 以及 `bat-api` 的鉴权管理接口暴露，dashboard 不维护第二套状态。该例外只编排已有解析和 patch 能力，不扩大解析器覆盖；完整 AssetBundle 重打包、真实 provider worker 和 Web 前端仍是后续工作。G-008（产品级 Go 同步 CLI）已决策关闭。真实官方网络全量拉取 smoke 已固化（G-018 已关闭）；真实大文件与运行报告默认在 `/tmp` 隔离目录，不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md`。 +15. issue 43 已补齐 Rust `bat` 的 `res` / `parse` / `i18n` 工作流入口：支持资源拉取、解析刷新、可再生解析缓存清理、离线翻译工作台、人工文本查看/修改/清空、工作台发布前校验、有限 TextAsset 汉化发布、人工校对状态标记、既有 patch 能力的批量重打包、单次/限定次数/周期执行和版本化 schedule CRUD。issue 44 已接入 `translation.worker.run` provider worker：默认并发 8、范围 `1..=256`，每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果。schedule 查询现在按一级工作流过滤，删除/执行会校验作用域，单轮执行可限制计划数；schedule CRUD、翻译任务查询/交接视图、翻译任务状态回写、provider worker 触发和 `translation.proofread` 状态标记已通过 `bat.sock` 的 RPC 以及 `bat-api` 的鉴权管理接口暴露，dashboard 不维护第二套状态。该例外只编排已有解析和 patch 能力，不扩大解析器覆盖；完整 AssetBundle 重打包和 Web 前端仍是后续工作。G-008（产品级 Go 同步 CLI）已决策关闭。真实官方网络全量拉取 smoke 已固化（G-018 已关闭）；真实大文件与运行报告默认在 `/tmp` 隔离目录，不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md`。 当前翻译交接还包括 `translation-tasks.sqlite` 和版本化 `translation-handoff.json`； `translation.tasks` 查询单项 worker 状态，`translation.handoff` 查询完整 @@ -146,7 +146,7 @@ job/unit/provider run 状态。当前下载实现使用默认 8 个独立 worker 待完成： -- 将 `crowdin-textunit-queue.json` 接入真实 Crowdin/provider worker、翻译记忆和 Patch 构建；围绕 `translation.task.update` 完成真实 worker 集成与失败原因落库验证，并继续扩展 CAS 诊断查询面。 +- 基于已接入的 `translation.worker.run` 继续推进翻译记忆、完整 Patch 构建/rollback；继续扩展 CAS 诊断查询面。 - 真实线上全量下载 smoke 已固化为 `scripts/official-full-pull-smoke.sh` 和 `make official-smoke`；实际运行报告由脚本写入隔离输出目录。 - 增加更多权限和极端文件系统场景测试。 @@ -271,7 +271,7 @@ cargo run -p bat-infrastructure --bin bat -- \ --watch ``` -资源 HTTP bootstrap / 只读分发入口是 Go `cmd/bat-api`。生产拓扑下它与 Rust `bat` 同环境运行，经 `bat.sock` RPC 获取当前 `resource_root`，不在配置里写死资源目录；本地开发不能全量跑 `bat` 时用 fixture 和 Go 门禁验证。`internal/api/testdata/contract/` 已固化来自 Rust 输出并经归一化的 `catalog.status`、`resource.manifest` 和 `official-sync-snapshot.json` contract fixture，Go mirror 测试会防止字段名、null 语义和 `game_main_config_bootstrap` 再次漂移。`bat-api` 已补 launcher 资源引导兼容端点、玩家-facing HTTP 控制面和鉴权调度管理接口（token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单；`reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh` 及 `schedule.*` 可经 Web 转发），响应只来自已发布 snapshot/RPC，不提供登录、网关、鉴权或完整 package update manifest。 +资源 HTTP bootstrap / 只读分发入口是 Go `cmd/bat-api`。生产拓扑下它与 Rust `bat` 同环境运行，经 `bat.sock` RPC 获取当前 `resource_root`，不在配置里写死资源目录；本地开发不能全量跑 `bat` 时用 fixture 和 Go 门禁验证。`internal/api/testdata/contract/` 已固化来自 Rust 输出并经归一化的 `catalog.status`、`resource.manifest` 和 `official-sync-snapshot.json` contract fixture，Go mirror 测试会防止字段名、null 语义和 `game_main_config_bootstrap` 再次漂移。`bat-api` 已补 launcher 资源引导兼容端点、玩家-facing HTTP 控制面和鉴权调度/translation 管理接口（token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单；`reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`、`schedule.*`、`translation.tasks` / `translation.handoff` 查询、`translation.task.update`、`translation.worker.run` 和 `translation.proofread` 可经 Web 转发），响应只来自已发布 snapshot/RPC，不提供登录、网关、鉴权或完整 package update manifest。 生产要求： @@ -292,8 +292,8 @@ GitHub issue 状态：#1 已关闭；#17 的历史决定不代表当前下载实 下一阶段必须优先完成： 1. Issue #1（P0，主体已实现）：`bat.sock` Unix socket JSON-RPC 已扩展为面向 Go 服务层的 Rust Resource Backend API。统一 envelope（`ok`、`status`、`error`、`data`、`request_id`）与 `BAT-ERR` 错误码模型已落地；`daemon.*`（status/logs/stop/restart/reload/refresh/doctor）、`resource.*`（state/sync/verify/repair/manifest/list/index）、`schedule.*`（list/add/update/remove/run）、`parse.*`（status/text_units/errors）、`translation.*`（tasks/handoff/task.update/proofread）、`localized.*`（status）、`catalog.*`（status/refresh/diff/versions）、`task.*`（status/list/cancel/logs）、文件级 `patch.apply` 与 `unityfs.patch_text_asset` / `unityfs.patch_string_field` / `unityfs.patch_field` 已实现，长任务返回 `task_id` 可轮询（任务执行器单 worker FIFO，与 watch 循环互斥；任务历史持久化于 `/bat-tasks.json`，daemon 重启后仍可查，中断任务标记 `task_interrupted`）；错误码已接入下载、launcher/metadata、server-info/marker 与配置校验路径。剩余：发布级 `patch build` / `patch rollback`、复杂 `unityfs.*` 语义编辑、`task.create`（按设计由语义方法创建）、`daemon.clean-stable`（由 CLI 侧按进程生命周期显式执行，live RPC 内不做在线清理）、Redis 任务后端（`.env` 已预留配置键，接入时机另议）。Go 层通过 RPC 调用 Rust backend，不走 FFI（FFI 降级说明见 `docs/architecture/official-resource-backend.md` §7）。 -2. Go 侧：进度见 `docs/reports/GO_STATUS.md`。G-008 已关闭；`bat-api` 资源 bootstrap/分发 MVP 已落地，已含 `/v1/bootstrap`、`/v1/launcher/bootstrap`、launcher 资源 metadata 兼容、HTTP 鉴权/限流/日志/反代适配、动态 JSON no-store、OpenAPI、管理控制白名单、CDN Range/缓存头、RPC 周期刷新和 USERGUIDE 基础章节；仓库内 Rust/Go snapshot contract fixture 与同机 live smoke 已落地。失效 `resource_root` 或 manifest 不完整时会清空旧索引并使 `/readyz` 返回 `503`；编码 dot-segment 会在路由规范化前拒绝。refresh mtime/size 增量缓存和可选持久化另议。 -3. 文本提取 / 翻译队列 / Patch 输入：`official-textunit-index.json`、`official-textunit-tasks.json` 与 `crowdin-textunit-queue.json` 已生成；TextUnit 明细、解析错误和离线翻译任务状态/失败原因已可查询，`translation.task.update` 已提供 Rust CLI、Go typed helper 和 bat-api dashboard 状态回写入口，`translation.proofread` 已提供人工校对状态标记入口。剩余为真实 Crowdin/provider worker、翻译记忆和 Patch 构建。 +2. Go 侧：进度见 `docs/reports/GO_STATUS.md`。G-008 已关闭；`bat-api` 资源 bootstrap/分发 MVP 已落地，已含 `/v1/bootstrap`、`/v1/launcher/bootstrap`、launcher 资源 metadata 兼容、HTTP 鉴权/限流/日志/反代适配、动态 JSON no-store、OpenAPI、管理控制白名单、translation admin 查询/控制、CDN Range/缓存头、RPC 周期刷新和 USERGUIDE 基础章节；仓库内 Rust/Go snapshot contract fixture 与同机 live smoke 已落地。失效 `resource_root` 或 manifest 不完整时会清空旧索引并使 `/readyz` 返回 `503`；编码 dot-segment 会在路由规范化前拒绝。refresh mtime/size 增量缓存和可选持久化另议。 +3. 文本提取 / 翻译队列 / Patch 输入：`official-textunit-index.json`、`official-textunit-tasks.json` 与 `crowdin-textunit-queue.json` 已生成；TextUnit 明细、解析错误和离线翻译任务状态/失败原因已可查询，`translation.tasks` / `translation.handoff` 已提供 Go typed helper 和 bat-api dashboard 查询入口，`translation.task.update` 已提供 Rust CLI、Go typed helper 和 bat-api dashboard 状态回写入口，`translation.worker.run` 已提供 mock/Crowdin provider worker、lease、重试和 TextUnit 译文结果落库，`translation.proofread` 已提供人工校对状态标记入口。剩余为翻译记忆、完整 Patch 构建/rollback 和复杂重打包。 4. Issue #3（已完成当前目标）：AssetBundle UnityFS 基础解析校验已覆盖 header、block、directory、LZ4/LZMA、alignment、总大小、计数、路径安全、重复 directory、边界和隔离 UnityPy 真实样本；复杂对象解析与发布级重打包继续跟踪 G-005。 5. Issue #2（已完成当前目标）：Addressables JSON/compact catalog 已提取 bundle provider、name、hash/size/CRC、资源类型和依赖关系，并贯通 ResourceEntry、SQLite 与 resource.index 输出；独立二进制格式仍明确拒绝。 6. 通用 Binary/JSON/Text Patch 基础已落地；复杂 AssetBundle 重打包和真实翻译系统仍应后置，UnityFS TextAsset patch 发布前置已具备回归测试。 @@ -315,4 +315,4 @@ GitHub issue 状态：#1 已关闭；#17 的历史决定不代表当前下载实 - **当前总体完成度**：不再固定写单一百分比，以各模块状态、`GO_STATUS.md` 和 issue 为准。 - **当前基线状态**：Rust `bat` 同步闭环可用；Go `bat-api` 资源 bootstrap/分发 MVP + 玩家-facing HTTP 控制面 + launcher 资源引导兼容 + RPC 周期刷新/诊断 + readiness + `backendrpc` 可用；CAS 用户级导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、通用 Binary/JSON/Text Patch 基础和 UnityFS TextAsset patch 发布前置可用；完整 AssetBundle 重打包未完成。 -- **下一工程里程碑**：真实 Crowdin/provider worker / 翻译记忆、复杂 AssetBundle 解析和重打包；`bat-api` 同机 live 联调已完成。 +- **下一工程里程碑**：翻译记忆、完整 localized Patch 构建/rollback、复杂 AssetBundle 解析和重打包；`bat-api` 同机 live 联调已完成。 diff --git a/PROJECT_PLAN.md b/PROJECT_PLAN.md index d76915b..60151b8 100644 --- a/PROJECT_PLAN.md +++ b/PROJECT_PLAN.md @@ -36,7 +36,7 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付 8. `bat-ffi` 已提供 Manifest inspect 和官方 sync plan 的可选无状态粗粒度 JSON C ABI helper。 9. 官方原版资源默认发布到 `./bat-resources`，汉化产物默认发布到独立的 `./bat-localized`；当前官方同步报告会标记 `localized_release_status=not_localized`，表示原版资源已发布、汉化资源未发布；`translation.proofread` 可把汉化 workflow 标记为人工校对中，但不会覆盖已发布的汉化 release。 10. 官方同步校验完成并发布新 release 后会生成 `official-resource-changes.json`、`crowdin-translation-handoff.json`、`official-parse-cache.json`、`official-textunit-index.json`、`official-textunit-tasks.json` 和 `crowdin-textunit-queue.json`，用 Added/Modified 资源驱动后续解析/翻译增量；up-to-date 轮询在已有有效缓存、TextUnit 明细索引和队列时只读取摘要，不重复解析。 -11. issue 43 已补齐 Rust `bat` 的 `res` / `parse` / `i18n` 工作流：单次/限定次数/周期执行、版本化 schedule CRUD 与作用域过滤、解析缓存清理、翻译工作台校验、离线翻译工作台、人工文本查看/修改/清空、翻译任务状态回写、人工校对状态标记、既有 patch 能力的批量重打包和独立汉化 release 发布；schedule CRUD、`translation.task.update` 和 `translation.proofread` 已经通过 `bat.sock` 和 `bat-api` 管理接口暴露，dashboard 不维护第二套状态；该能力不扩大冻结期解析器覆盖。 +11. issue 43 已补齐 Rust `bat` 的 `res` / `parse` / `i18n` 工作流：单次/限定次数/周期执行、版本化 schedule CRUD 与作用域过滤、解析缓存清理、翻译工作台校验、离线翻译工作台、人工文本查看/修改/清空、翻译任务状态回写、人工校对状态标记、既有 patch 能力的批量重打包和独立汉化 release 发布；issue 44 已接入 `translation.worker.run` provider worker，默认并发 8、范围 `1..=256`，每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果；schedule CRUD、`translation.tasks` / `translation.handoff`、`translation.task.update`、`translation.worker.run` 和 `translation.proofread` 已经通过 `bat.sock` 和 `bat-api` 管理接口暴露，dashboard 不维护第二套状态；该能力不扩大冻结期解析器覆盖。 12. 文档已整理：根目录保留入口文档，历史报告进入 `docs/reports/historical/`，误嵌套的 `docs/docs` 已合并。 ### 仍是骨架或占位 @@ -45,8 +45,8 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付 2. `bat-patch` 已具备确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、通用 Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据；文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放，当前发布级可用的是 `bat-assetbundle` + `LocalizedPatchService` 的 UnityFS TextAsset patch 前置链路。 3. Go 侧边界已冻结（见 `docs/reports/GO_STATUS.md`）：同步/运维命令行 = Rust `bat`；资源分发 = `cmd/bat-api` MVP；`internal/backendrpc` 完成；`cmd/bat` 仅为试验（`bin/bat-go`）。完整游戏业务 API / Web / SDK 仍未完成。 4. Addressables parser 已覆盖当前真实形态 fixture/golden，但还不是完整 Unity Addressables/SBP catalog 兼容层。 -5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository，并通过 `resource.index` RPC/CLI 查询；Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要，资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format；单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`，可通过 `parse.text_units` / `parse.errors` 查询；离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询，`translation.task.update` 已提供 worker 状态回写 contract，`translation.proofread` 已提供汉化 workflow 人工校对标记 contract，真实 provider worker 集成仍需继续推进。 -6. 汉化 Patch 发布前置已具备 UnityFS TextAsset manifest/apply/diff/rollback/完整性校验和 `localized.status` 严格校验；真实 Crowdin worker、翻译记忆到完整汉化文件集合的构建仍未完成。 +5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository，并通过 `resource.index` RPC/CLI 查询；Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要，资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format；单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`，可通过 `parse.text_units` / `parse.errors` 查询；离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询，`translation.task.update` 已提供 worker 状态回写 contract，`translation.worker.run` 已提供真实 provider worker 触发、lease/retry 和结果落库 contract，`translation.proofread` 已提供汉化 workflow 人工校对标记 contract。 +6. 汉化 Patch 发布前置已具备 UnityFS TextAsset manifest/apply/diff/rollback/完整性校验和 `localized.status` 严格校验；真实 provider worker 已接入，翻译记忆到完整汉化文件集合的构建仍未完成。 7. 真实官方网络全量下载 smoke 已固化为可重复脚本和 runbook（G-018 已关闭）；真实运行记录处于长期运行测试阶段，报告待后续提供。 8. 完整 Web、数据库迁移、插件加载机制尚未实现；`bat-api` 资源 bootstrap/分发 OpenAPI 已通过 `/openapi.yaml` 提供，完整游戏业务 API 的 OpenAPI 仍未完成。 9. 原 Git 历史未恢复；当前仓库以新初始化基线为准。 @@ -179,7 +179,7 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付 4. Rust 自动更新入口：**已完成当前生产入口**。`bat` 支持 snapshot、marker diff、bootstrap cache、one-shot、`--watch`、`--daemon`、默认 1 小时间隔、北京时间固定强制刷新，以及 Unix socket JSON-RPC 后台运维命令返回。 5. Go 入口边界：**已冻结**。同步命令行 = Rust `bat`（G-008 关闭）；资源分发 = `bat-api` MVP（G-009 部分完成）。详见 `docs/reports/GO_STATUS.md`。 6. 用户级 `sync`、`manifest inspect`、`cache status`：**未完成**。Rust `bat --json` 是当前稳定进程边界；`bat-ffi` 只提供可选兼容用的 Manifest inspect 和 sync plan JSON helper。 -7. 下载结果写入 CAS + ResourceRepository：**部分完成**。CAS 和 SQLite ResourceRepository 已存在，官方同步入口可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后导入；`resource.index` 可按资源级 release、平台、destination、archive entry、parse status 和 TextUnit format 查询现有索引和资源 metadata；`parse.text_units` / `parse.errors` 可查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` 可查询离线 TextUnit 翻译任务状态和跳过/失败原因。剩余工作是真实 provider worker 集成、CAS 诊断入口和面向大索引的查询优化。 +7. 下载结果写入 CAS + ResourceRepository：**部分完成**。CAS 和 SQLite ResourceRepository 已存在，官方同步入口可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后导入；`resource.index` 可按资源级 release、平台、destination、archive entry、parse status 和 TextUnit format 查询现有索引和资源 metadata；`parse.text_units` / `parse.errors` 可查询当前 release 的 TextUnit 明细与解析错误；`translation.tasks` 可查询离线 TextUnit 翻译任务状态、跳过/失败原因和 worker 结果。剩余工作是 CAS 诊断入口和面向大索引的查询优化。 8. Linux 生产同步不依赖已安装官方启动器：**已完成当前 Rust 入口**。`--auto-discover` 只使用官方 HTTP metadata 和临时目录解析 `GameMainConfig`。 9. 真实官方网络全量下载 smoke test：**命令已固化（G-018 已关闭）**。`scripts/official-full-pull-smoke.sh` / `make official-smoke` 已固化 dry-run、首次下载、二次 up-to-date 和本地损坏 repair 的可重复流程；真实运行处于长期运行测试阶段，报告待后续提供。 10. 官方发布后的增量 handoff 与解析缓存：**已完成基础入口**。新 release 发布后先生成 `official-resource-changes.json` 和 `crowdin-translation-handoff.json`，新增+变更资源进入解析/翻译候选；`official-parse-cache.json` 基于下载 manifest 覆盖直接 UnityFS bundle、zip 内 UnityFS 条目和非候选资源记录；随后生成 `official-textunit-index.json`、`official-textunit-tasks.json` 与 `crowdin-textunit-queue.json`，本地文件未变化且缓存/索引有效时跳过重复解析。 @@ -404,8 +404,8 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付 2. G-008 已决策关闭：同步/运维命令行 = 近乎全自动的 Rust `bat`。 3. G-009 / issue #19：`bat-api` 资源分发和同机 `bat.sock` live 联调已完成；拉取仍在 Rust `bat`。 4. 继续 Addressables 结构变体与 UnityFS 复杂对象能力（issue #3 / G-005 的后续阶段）。 -5. 将 `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 -6. 继续扩展 G-011 剩余查询面：真实 provider worker 集成与状态落库验证、`doctor cas` 诊断入口和面向大索引的查询优化。 +5. 基于 `translation.worker.run` 继续推进翻译记忆和 Patch 构建。 +6. 继续扩展 G-011 剩余查询面：`doctor cas` 诊断入口和面向大索引的查询优化。 --- @@ -459,9 +459,9 @@ BlueArchiveToolkit 不是一次性脚本，也不是演示项目。最终交付 按最终目标计算，当前总体完成度不再固定写单一百分比，以模块状态和 issue 收敛情况为准。 -已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、通用 Binary/JSON/Text Patch 基础、UnityFS TextAsset patch 发布前置，以及 Go `bat-api` 资源分发和同机 live 联调。下一阶段的关键是真实 Crowdin worker、翻译记忆、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。 +已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、provider worker、通用 Binary/JSON/Text Patch 基础、UnityFS TextAsset patch 发布前置，以及 Go `bat-api` 资源分发和同机 live 联调。下一阶段的关键是翻译记忆、完整 localized Patch 构建/rollback、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。 --- - **下一份应更新文档**：真实官方网络 smoke 记录 -- **下一项工程任务**：推进真实 Crowdin worker / 翻译记忆、复杂 AssetBundle 解析，并持续执行官方资源长期运行 smoke。 +- **下一项工程任务**：推进翻译记忆、完整 localized Patch 构建/rollback、复杂 AssetBundle 解析，并持续执行官方资源长期运行 smoke。 diff --git a/README.md b/README.md index f66c692..033c9a2 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ - `bat-infrastructure` CAS 适配层、SQLite Resource Repository、资源导入服务、官方资源 pull/update 服务。 - `bat`：官方资源自动发现、全量拉取、原子发布到 `current -> versions/`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、默认并发 8（可配置 `1..=256`，report 按 plan 顺序、进度按完成数单调上报）、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json`、`--watch` 常驻更新、`--daemon` 后台运行，以及 Unix socket JSON-RPC live control/backend 方法（`daemon.*`、`resource.*`、`parse.*`、`translation.tasks/handoff/task.update`、`localized.status`、`catalog.*`、`task.*`、`patch.apply`、`unityfs.patch_*`）。 - `internal/backendrpc`：Go 侧 typed Unix socket JSON-RPC client，是 `bat-api` 调用 Rust daemon 的默认路径。 -- `cmd/bat-api`：资源 bootstrap + 分发 HTTP MVP（issue #19 / G-009）；`/v1/bootstrap` 和 `/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口，launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导，`/healthz` 暴露 RPC refresh 诊断，`/readyz` 做 release readiness，CDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头；玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI 和管理控制白名单；`.env` 配置端口/RPC socket/刷新周期；生产资源根来自 RPC，不负责自动拉取。 +- `cmd/bat-api`：资源 bootstrap + 分发 HTTP MVP（issue #19 / G-009）；`/v1/bootstrap` 和 `/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口，launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导，`/healthz` 暴露 RPC refresh 诊断，`/readyz` 做 release readiness，CDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头；玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单和 translation admin 查询/控制入口；`.env` 配置端口/RPC socket/刷新周期；生产资源根来自 RPC，不负责自动拉取。 - Go 边界权威说明：[`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)（G-008 已关闭：同步 CLI = Rust `bat`）。 - 官方同步会维护 `/official-version-state.json`，明确记录当前已完成版本、正在拉取版本、上一个可用版本和失败版本。 - 资源导入链路可配置为在官方 release 发布后写入 CAS + `ResourceRepository`，资源 metadata 会记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式，TextAsset/Table/Media 会按类型分类索引；`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、archive entry、parse status 和 TextUnit format 分页查询索引。 @@ -186,7 +186,7 @@ BlueArchiveToolkit/ 1. 维护并联调 Go `bat-api` 资源 bootstrap/分发入口；`cmd/bat` 仅有 `doctor`、`manifest inspect`、`sync plan` 试验能力，不应误写成完整产品 CLI。 2. 补齐 AssetBundle UnityFS 引擎级解析。 3. 扩展 Addressables catalog 解析覆盖，继续用真实形态 fixture/golden 锁定行为。 -4. 将 `official-textunit-tasks.json` / `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 +4. 基于 `translation.worker.run` provider worker 推进翻译记忆、完整 Patch 构建和发布/回滚闭环。 5. 按 smoke runbook 在具备网络和磁盘窗口的环境中执行真实官方全量拉取，并保留本地报告。 不建议在 Go 产品入口、资源解析和文本提取基础能力完成前优先开发 Web UI。 diff --git a/USERGUIDE.md b/USERGUIDE.md index 4a4a0ac..3ed15b5 100644 --- a/USERGUIDE.md +++ b/USERGUIDE.md @@ -58,6 +58,7 @@ HTTPS_PROXY=http://user:pass@127.0.0.1:7890 bat --auto-discover --daemon | `i18n handoff` | 查询当前翻译交接视图 | | `i18n status` | 显示当前汉化 release 状态 | | `i18n task update` | 回写 provider worker 任务状态 | +| `i18n worker run` | 运行真实 provider worker；支持单次、限定次数和周期执行 | | `i18n publish` | 校验工作台并发布独立汉化 release；`--force` 使用新的手动 release ID | | `i18n schedule` | 管理翻译和汉化发布计划 | | `refresh` | 执行一次更新检查；若有 live daemon，则通过 RPC 请求其刷新 | @@ -137,13 +138,15 @@ BAT_API_SKIP_ENV_FILE=1 go run ./cmd/bat-api \ | `GET /openapi.yaml` | bat-api OpenAPI 文档 | | `GET /admin/` | 管理控制入口与允许操作列表 | | `GET /admin/schedules?id=...&group=...&enabled=...` | 读取/过滤 Rust `bat` 调度计划；需要管理 token | +| `GET /admin/translation/tasks?limit=100&worker_status=failed` | 读取/过滤 Rust 翻译任务和 provider worker 状态；需要管理 token | +| `GET /admin/translation/handoff` | 读取当前 release 的完整翻译交接视图；需要管理 token | | `POST /admin/control/{action}` | 经白名单转发 Rust `bat` 控制请求；见下文 | launcher 兼容端点只服务启动前资源发现。它们复用 Rust `bat` snapshot 中的 `launcher_metadata` 和 `game_main_config_bootstrap`，显式标记 `scope=resource_bootstrap_only` / `package_update_manifest=false`。`bat-api` 不下载 launcher 包，不生成官方 PC package update manifest，也不仿造登录、账号、网关、鉴权或游戏业务协议。 生产面对玩家分发时，应启用 HTTP token 鉴权、限流和访问日志： -- `BAT_API_AUTH_TOKEN`：启用 `Authorization: Bearer `、`X-BAT-Token` 或 query fallback 鉴权；token 推荐由 secret manager 或进程环境提供，不建议写入提交文件。`/admin/control/*` 和 `/admin/schedules` 需要此 token。 +- `BAT_API_AUTH_TOKEN`：启用 `Authorization: Bearer `、`X-BAT-Token` 或 query fallback 鉴权；token 推荐由 secret manager 或进程环境提供，不建议写入提交文件。`/admin/control/*`、`/admin/schedules` 和 `/admin/translation/*` 需要此 token。 - `BAT_API_AUTH_QUERY_PARAM`：query fallback 参数名，默认 `bat_token`；兼容不能写 header 的客户端，访问日志不会记录 query。 - `BAT_API_AUTH_EXEMPT_PATHS`：逗号分隔的免鉴权 path 或 slash-prefix，例如 `/healthz,/readyz`。 - `BAT_API_RATE_LIMIT_RPS` / `BAT_API_RATE_LIMIT_BURST`：按客户端 IP 的进程内 token bucket 限流；边缘反代/CDN 仍应配置独立限流。 @@ -167,6 +170,8 @@ launcher 兼容端点只服务启动前资源发现。它们复用 Rust `bat` sn | `schedule-remove` | `schedule.remove` | `{ "id": "..." }` | `202` + Rust schedule report | | `schedule-run` | `schedule.run` | 可选 `{ "id": "...", "force": true }` | `202` + 执行报告 | | `translation-task-update` | `translation.task.update` | `{ "task_id": "...", "status": "...", "failure_reason": "...", "provider_run_id": "..." }` | `202` + 当前任务记录 | +| `translation-worker-run` | `translation.worker.run` | `{ "provider": "mock", "concurrency": 8, "max_tasks": 2 }` | `202` + worker task | +| `translation-proofread` | `translation.proofread` | 无 | `202` + 汉化状态 | `stop`、`clean-stable`、patch 和 UnityFS 写入命令不会经 HTTP 暴露。 @@ -420,6 +425,7 @@ curl -i -H 'Range: bytes=0-1023' \ | `translation.tasks` | ✅ | 查询离线 TextUnit 翻译任务及 worker 状态 | | `translation.handoff` | ✅ | 查询完整 job/unit/provider run 交接视图 | | `translation.task.update` | ✅ | 回写当前 release 的 provider worker 状态 | +| `translation.worker.run` | ✅ | 触发 Rust provider worker，落库 TextUnit 译文结果、lease、失败分类和重试状态 | | `translation.proofread` | ✅ | 将当前汉化 workflow 标记为人工校对中 | | `localized.status` | ✅ | 查询汉化 release 与当前官方 release 的匹配状态 | | `catalog.status` | ✅ | 当前已发布版本的 catalog 概览（app/bundle 版本、addressables 根、端点与 marker 计数、launcher 元数据） | diff --git a/api/openapi/bat-api.yaml b/api/openapi/bat-api.yaml index 3b830bf..dc13d50 100644 --- a/api/openapi/bat-api.yaml +++ b/api/openapi/bat-api.yaml @@ -108,16 +108,114 @@ paths: responses: "200": description: Admin links and allowlisted control actions. + /admin/schedules: + get: + summary: List Rust-owned resource workflow schedules + parameters: + - name: id + in: query + schema: + type: string + - name: group + in: query + schema: + type: string + enum: [res, parse, i18n] + - name: enabled + in: query + schema: + type: boolean + responses: + "200": + description: Current schedule JSON report. + "401": + description: Missing or invalid admin token. + "503": + description: Rust bat schedule backend is unavailable. + /admin/translation/tasks: + get: + summary: List Rust-owned translation task worker status + parameters: + - name: offset + in: query + schema: + type: integer + minimum: 0 + - name: limit + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + - name: task_id + in: query + schema: + type: string + - name: release_id + in: query + schema: + type: string + - name: destination + in: query + schema: + type: string + - name: archive_entry + in: query + schema: + type: string + - name: status + in: query + schema: + type: string + - name: worker_status + in: query + schema: + type: string + - name: parse_status + in: query + schema: + type: string + - name: format + in: query + schema: + type: string + - name: has_reason + in: query + schema: + type: boolean + - name: has_failure_reason + in: query + schema: + type: boolean + responses: + "200": + description: Current translation task JSON report from Rust bat. + "400": + description: Invalid translation task query. + "401": + description: Missing or invalid admin token. + "503": + description: Rust bat translation backend is unavailable. + /admin/translation/handoff: + get: + summary: Read Rust-owned translation handoff state + responses: + "200": + description: Current translation handoff JSON report from Rust bat. + "401": + description: Missing or invalid admin token. + "503": + description: Rust bat translation backend is unavailable. /admin/control/{action}: post: - summary: Forward an allowlisted control action to Rust bat + summary: Forward an allowlisted control or schedule action to Rust bat parameters: - name: action in: path required: true schema: type: string - enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh] + enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, translation-task-update, translation-worker-run, translation-proofread] requestBody: required: false content: @@ -128,6 +226,74 @@ paths: properties: force: type: boolean + id: + type: string + group: + type: string + action: + type: string + args: + type: array + items: + type: string + next_run_unix_seconds: + type: integer + format: int64 + delay_seconds: + type: integer + format: int64 + every_seconds: + type: integer + format: int64 + count: + type: integer + format: int64 + max_runs: + type: integer + format: int64 + minimum: 1 + clear_args: + type: boolean + clear_every: + type: boolean + enabled: + type: boolean + task_id: + type: string + status: + type: string + failure_reason: + type: string + provider_run_id: + type: string + provider: + type: string + enum: [mock, crowdin] + fixture_path: + type: string + concurrency: + type: integer + format: int64 + minimum: 1 + maximum: 256 + max_attempts: + type: integer + format: int64 + minimum: 1 + lease_seconds: + type: integer + format: int64 + minimum: 1 + retry_backoff_seconds: + type: integer + format: int64 + minimum: 0 + max_tasks: + type: integer + format: int64 + minimum: 1 + worker_id: + type: string responses: "202": description: Rust bat accepted the control request. diff --git a/docs/architecture/assetbundle.md b/docs/architecture/assetbundle.md index 79eef27..9e3c108 100644 --- a/docs/architecture/assetbundle.md +++ b/docs/architecture/assetbundle.md @@ -208,6 +208,6 @@ catalog 仍按“明确不支持”处理，不把低保真路径伪装成完整 1. 完成 Addressables Windows/Android 当前版本 catalog 样本集合，关闭 G-007 当前阶段。 2. 完成 TypeTree 字段 reader 和 MonoBehaviour/ScriptableObject 遍历，推进 G-005。 -3. 将 `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 -4. 将翻译任务状态接入 CAS/ResourceRepository 查询面，推进 G-011。 +3. 基于 `translation.worker.run` 推进翻译记忆和 Patch 构建。 +4. 扩展翻译任务结果在 CAS/ResourceRepository 查询面的索引，推进 G-011。 5. 在通用 Binary/JSON/Text Patch 基础上继续扩展复杂 AssetBundle 重打包和发布流程统一，保留当前 UnityFS TextAsset patch 发布前置链路。 diff --git a/docs/architecture/official-resource-backend.md b/docs/architecture/official-resource-backend.md index 5530b48..a3b8c2c 100644 --- a/docs/architecture/official-resource-backend.md +++ b/docs/architecture/official-resource-backend.md @@ -179,8 +179,9 @@ `resource.index` RPC / CLI 只读查询现有 SQLite 索引；索引不存在时返回 `available=false`，不会因为查询创建空库。发布后的 TextUnit 队列还会在当前 release 根目录写入 `translation-tasks.sqlite`，由版本化 `schema_migrations` -管理 queued/running/failed/completed/skipped 和 failure reason。`translation.tasks` -优先查询这份状态库，`translation.task.update` 供 provider worker 回写状态； +管理 queued/running/failed/completed/skipped、provider run、lease、失败分类、 +重试计划和 TextUnit 级译文结果。`translation.tasks` 优先查询这份状态库， +`translation.worker.run` 由 Rust worker 回写状态；`translation.task.update` 仍供外部 provider 流程回写状态； 没有状态库的旧 release 才回退到 immutable JSON 队列。G-011 剩余工作是 CAS 诊断入口和面向大索引的查询优化。 diff --git a/docs/guides/baseline.md b/docs/guides/baseline.md index f30f04d..6bb8b03 100644 --- a/docs/guides/baseline.md +++ b/docs/guides/baseline.md @@ -92,8 +92,8 @@ CAS V1 和 Rust 官方同步闭环完成后，下一阶段优先推进： 1. 继续联调 Go `bat-api` 与 Rust daemon 的资源分发路径；Go 同步 CLI 不再作为产品目标。 2. 按 `docs/guides/official-full-pull-smoke.md` 在隔离目录执行真实官方网络全量下载 smoke，并保留运行报告。 -3. 将 `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 -4. 扩展 ResourceRepository 查询面：真实 provider worker 集成与状态落库验证、CAS 诊断入口和更丰富 TextUnit 查询。 +3. 基于 `translation.worker.run` 推进翻译记忆和 Patch 构建。 +4. 扩展 ResourceRepository 查询面：CAS 诊断入口、面向大索引的查询优化和更丰富 TextUnit 查询。 5. 继续完善 AssetBundle 复杂对象解析、复杂对象重打包和 Patch 发布流程统一；通用 Binary/JSON/Text Patch 基础与 UnityFS TextAsset patch 发布前置链路已可用。 优先阅读： diff --git a/docs/guides/bat-workflows.md b/docs/guides/bat-workflows.md index 9fdb28e..80a35f5 100644 --- a/docs/guides/bat-workflows.md +++ b/docs/guides/bat-workflows.md @@ -133,9 +133,30 @@ bat i18n publish \ 发布只接受当前实现支持的直接 TextAsset 条目；TypeTree 字段和 zip 内 bundle 使用 `parse repack` 的 spec 单独处理。`--force` 不覆盖已有目录，而是生成独立的 `-manual-` 汉化 release ID；也可以用 `--localized-release-id` 显式指定新 ID。因此强制发布仍保留旧 release 和 rollback 信息。 -当前 `i18n run` 是离线工作流：刷新 TextUnit 队列，并可用 `--translation-file` 导出工作台；不会假装调用未接入的外部 provider。 +当前 `i18n run` 是离线工作流：刷新 TextUnit 队列，并可用 `--translation-file` 导出工作台；真实 provider 由单独的 worker 命令消费 `translation-tasks.sqlite`。 -provider worker 消费 handoff 后，可以用 `i18n task update` 回写当前 release 的任务状态： +运行一次 mock provider worker： + +```bash +bat i18n worker run \ + --output /tmp/bat-resources \ + --provider mock \ + --worker-concurrency 8 +``` + +`--provider` 支持 `mock` 和 `crowdin`。`mock` 可通过 `--translation-fixture` +读取本地 JSON fixture；`crowdin` 从环境变量 `CROWDIN_PROJECT_ID`、 +`CROWDIN_LANGUAGE_ID`、`CROWDIN_API_TOKEN` 读取配置。worker 默认并发为 8， +范围 `1..=256`；每个 worker 完成当前任务后立即从 SQLite 队列领取下一项， +不会等待当前一批 worker 全部结束后再重新分配。 + +可用参数包括 `--worker-max-attempts`、`--worker-lease-seconds`、 +`--worker-retry-backoff` / `--worker-retry-backoff-seconds`、 +`--worker-max-tasks` 和 `--worker-id`。worker 支持 `--run-count` 与 +`--watch`，因此可以单次、限定次数或周期执行；`--run-count > 1` 时仍必须 +显式指定 `--interval`。 + +外部 provider 或人工流程也可以用 `i18n task update` 回写当前 release 的任务状态： ```bash bat i18n task update \ @@ -224,11 +245,24 @@ contract：`id`、`group`、`action`、`args`、`next_run_unix_seconds`、 `enabled`；`schedule.list` 额外接受 `id`、`group`、`enabled` 过滤， `schedule.run` 额外接受 `group`、`force` 和 `max_runs`。 +翻译任务状态可由已鉴权的 dashboard 通过 `GET /admin/translation/tasks` +查询，query 过滤项包括 `offset`、`limit`、`task_id`、`release_id`、 +`destination`、`path_pattern`、`archive_entry`、`status`、`worker_status`、 +`parse_status`、`format`、`has_reason` 和 `has_failure_reason`。完整 provider +run 交接视图通过 `GET /admin/translation/handoff` 查询。两个查询接口都只转发 +Rust `translation.tasks` / `translation.handoff`，不在 Go 侧维护状态。 + 翻译任务状态也可由已鉴权的 dashboard 通过 `POST /admin/control/translation-task-update` 回写，请求字段为 `task_id`、`status`，以及可选的 `failure_reason`、`provider_run_id`； 该接口只转发 `translation.task.update`。 +`POST /admin/control/translation-worker-run` 会触发 Rust 侧 +`translation.worker.run`，请求字段为 `provider`、`fixture_path`、 +`concurrency`、`max_attempts`、`lease_seconds`、`retry_backoff_seconds`、 +`max_tasks` 和 `worker_id`。bat-api 只做鉴权、JSON 解码和基础范围校验； +任务状态、lease、重试和 provider 结果仍由 Rust 持久化。 + `POST /admin/control/translation-proofread` 会把当前汉化 workflow 标记为人工校对中； 该接口只转发 `translation.proofread`，不会改动已发布汉化 release 指针。 diff --git a/docs/guides/development.md b/docs/guides/development.md index 5949c69..9a78b54 100644 --- a/docs/guides/development.md +++ b/docs/guides/development.md @@ -258,6 +258,7 @@ cargo run -p bat-infrastructure --bin bat -- parse-errors --limit 50 cargo run -p bat-infrastructure --bin bat -- translation-tasks --task-status skipped_parse_failed --has-reason --limit 50 cargo run -p bat-infrastructure --bin bat -- translation-tasks --worker-status failed --has-failure-reason --limit 50 cargo run -p bat-infrastructure --bin bat -- translation-handoff +cargo run -p bat-infrastructure --bin bat -- i18n worker run --provider mock --worker-concurrency 8 --worker-max-tasks 10 cargo run -p bat-infrastructure --bin bat -- localized-status cargo run -p bat-infrastructure --bin bat -- resource-index --limit 50 cargo run -p bat-infrastructure --bin bat -- resource-index --release-id --platform windows --archive-entry --format json --limit 50 @@ -270,6 +271,9 @@ cargo run -p bat-infrastructure --bin bat -- resource-index --release-id -- worker 状态、parse status、TextUnit format、队列 reason 和 provider failure reason 查询离线 TextUnit 翻译任务状态与跳过/失败原因；发布后的状态保存在当前 release 根目录的 `translation-tasks.sqlite`，旧 release 没有状态库时回退到 JSON 队列； +`i18n worker run` / `translation.worker.run` 会由 Rust provider worker 独立 claim +下一项任务并落库 lease、失败分类、重试计划和 TextUnit 级译文结果，默认并发为 8， +范围 `1..=256`； `translation-handoff` / `translation.handoff` 会动态合并版本化 `translation-handoff.json` 与 SQLite 状态，返回 job、unit、provider run 的完整交接 视图； diff --git a/docs/reference/rpc-backend-api.md b/docs/reference/rpc-backend-api.md index 4b5b23b..3a2dd07 100644 --- a/docs/reference/rpc-backend-api.md +++ b/docs/reference/rpc-backend-api.md @@ -150,11 +150,12 @@ SQLite `ResourceRepository`，索引不存在时返回 `ok=true` 且 本地文件未变且索引有效时复用，不重复解析。 - `official-textunit-tasks.json`：只由 added + modified 资源、parse cache 和 TextUnit 明细索引派生，记录 TextUnit 任务、跳过原因和解析诊断。 -- `crowdin-textunit-queue.json`：只包含已产生 TextUnit 的离线任务，预留给后续 - Crowdin worker；当前不会发出网络请求。 +- `crowdin-textunit-queue.json`：只包含已产生 TextUnit 的离线任务；官方同步阶段 + 不发出 provider 网络请求。 - `translation-tasks.sqlite`：当前 release 的可变 worker 状态库，记录 queued / running / failed / completed / skipped、attempt count、provider run - ID 和 failure reason；schema 由 `schema_migrations` 版本表管理。 + ID、provider、TextUnit 级译文结果、lease、失败分类、可重试标记和 + next attempt；schema 由 `schema_migrations` 版本表管理。 - `translation-handoff.json`：当前 release 的版本化 job/unit/provider run 交接 快照；worker 更新后的实时状态仍以 `translation-tasks.sqlite` 为准。 @@ -206,6 +207,7 @@ SQLite `ResourceRepository`，索引不存在时返回 `ok=true` 且 | `translation.tasks` | 已实现 | `{ "offset": 0, "limit": 100, "task_id": "...", "release_id": "...", "destination": "...", "archive_entry": "...", "status": "skipped_parse_failed", "parse_status": "failed", "format": "json", "has_reason": true }` | 当前官方 release 的离线 TextUnit 翻译任务状态分页。 | | `translation.handoff` | 已实现 | `null` | 当前官方 release 的 job、unit、provider run 交接视图；动态合并队列和 SQLite worker 状态。 | | `translation.task.update` | 已实现 | `{ "task_id": "...", "status": "failed", "failure_reason": "...", "provider_run_id": "..." }` | 写入当前 release 的 provider worker 状态，返回可回查任务记录。 | +| `translation.worker.run` | 已实现 | provider worker 参数 | 异步触发 Rust provider worker，返回 `{ "task_id": "...", "kind": "translation.worker.run", "worker": {...} }`。 | | `translation.proofread` | 已实现 | `null` | 将当前汉化 workflow 标记为人工校对中，返回工作流状态报告。 | `parse.status` 是只读查询；没有当前 release 或没有解析缓存时返回 @@ -215,8 +217,8 @@ SQLite `ResourceRepository`，索引不存在时返回 `ok=true` 且 `textunit_index_summary`；存在 `official-textunit-tasks.json` 时，响应会包含 `textunit_queue_available=true`、`textunit_task_queue_path` 和 `textunit_task_summary`。当 TextUnit 队列存在且有离线任务时， -`translation_status_code=translation.queued_offline`；真实 Crowdin worker -尚未接入时不会返回翻译完成状态。 +`translation_status_code=translation.queued_offline`；provider worker 完成任务后， +同一查询面会返回已落库的 worker 状态和 TextUnit 级译文结果。 `parse.text_units` / `parse.errors` 是只读查询；没有当前 release 或没有 `official-textunit-index.json` 时返回 `ok=true` 且 `data.available=false`。 @@ -260,6 +262,31 @@ bat-api 可通过 `translation.tasks` 查询单项任务，也可通过 不会触发下载或 provider 网络请求；没有当前 release 或任务队列时返回 `data.available=false`。 +`translation.worker.run` 通过 daemon 任务队列异步启动 Rust provider worker。 +provider worker 会先同步当前 release 的 TextUnit 队列到 +`translation-tasks.sqlite`，回收过期 lease，然后由 `concurrency` 个独立 worker +循环 claim 下一项任务；任一 worker 完成当前任务后会立即领取下一项，不等待 +其他 worker 完成本轮批次。默认并发为 8，范围 `1..=256`。 + +provider worker 参数： + +| 字段 | 类型 | 默认 | 说明 | +|---|---|---|---| +| `provider` | string | `mock` | `mock` 或 `crowdin`。 | +| `fixture_path` | string/null | `null` | mock provider fixture；别名为 `translation_fixture`、`provider_fixture`、`mock_fixture`、`fixture`。 | +| `concurrency` | uint | `8` | 独立 worker 数，范围 `1..=256`；别名为 `worker_concurrency`、`translation_concurrency`。 | +| `max_attempts` | uint | `3` | 单个任务最大 claim 次数，必须大于 0。 | +| `lease_seconds` | uint | `300` | claim lease 秒数，必须大于 0。 | +| `retry_backoff_seconds` | uint | `5` | 可重试 provider 失败的 next attempt 间隔，可为 0。 | +| `max_tasks` | uint/null | `null` | 本轮最多 claim 的任务数，设置时必须大于 0。 | +| `worker_id` | string | `bat-rpc-worker` | lease 诊断用 worker ID 前缀。 | + +数字字段必须是 JSON number；字符串数字、负数和越界值会返回 +`BAT-ERR-700002`。`mock` provider 在没有 fixture 时把 source text 写成可诊断的 +mock 译文；`crowdin` provider 从 `CROWDIN_PROJECT_ID`、`CROWDIN_LANGUAGE_ID`、 +`CROWDIN_API_TOKEN` 读取配置，可选 `CROWDIN_API_BASE_URL` 和 `BAT_CURL`。 +token 不会进入报告、任务记录或调试输出。 + ### localized | 方法 | 状态 | params | data | @@ -397,6 +424,7 @@ CLI 对应关系： | `bat translation-handoff` | `translation.handoff` | | `bat i18n task list` / `bat i18n task status` | `translation.tasks` | | `bat i18n task update` | `translation.task.update` | +| `bat i18n worker run` | `translation.worker.run` | | `bat i18n proofread` | `translation.proofread` | | `bat localized-status` | `localized.status` | | `bat resource-index` | `resource.index` | @@ -424,10 +452,10 @@ CLI 对应关系： - typed helper 已覆盖 daemon 已实现方法（`status/logs/stop/restart/reload/refresh/doctor`）、 `resource.state/sync/verify/repair/manifest/list`、`schedule.list/add/update/remove/run`、 `catalog.*`、`parse.*`、 - `localized.status`、`translation.task.update`、`translation.proofread`、`task.*` 和三个 - `unityfs.patch_*` 方法。 -- `resource.index`、`translation.tasks`、`translation.handoff` 和 - `patch.apply` 当前没有专用 typed helper；需要直接使用 `Call`，并仍须遵守 + `localized.status`、`translation.tasks`、`translation.handoff`、 + `translation.task.update`、`translation.worker.run`、`translation.proofread`、 + `task.*` 和三个 `unityfs.patch_*` 方法。 +- `resource.index` 和 `patch.apply` 当前没有专用 typed helper；需要直接使用 `Call`，并仍须遵守 本契约的参数和响应定义。 `internal/api` 对 bat-api 生产路径进一步收窄接口： @@ -437,7 +465,7 @@ CLI 对应关系： | `Backend` | `daemon.status`、`daemon.doctor`、`resource.state`、`catalog.status`、`resource.manifest` | 启动发现、周期刷新和资源分发 | | `ControlBackend` | `daemon.restart`、`daemon.reload`、`daemon.refresh`、`resource.sync`、`resource.verify`、`resource.repair`、`catalog.refresh` | 鉴权后的管理控制白名单 | | `ScheduleBackend` | `schedule.list`、`schedule.add`、`schedule.update`、`schedule.remove`、`schedule.run` | 鉴权后的 dashboard 调度计划控制 | -| `TranslationBackend` | `translation.task.update`、`translation.proofread` | 鉴权后的 dashboard 翻译任务状态回写与人工校对标记 | +| `TranslationBackend` | `translation.tasks`、`translation.handoff`、`translation.task.update`、`translation.worker.run`、`translation.proofread` | 鉴权后的 dashboard 翻译任务查询、交接视图、状态回写、provider worker 触发与人工校对标记 | `daemon.stop`、`daemon.clean-stable` 和任意通用 RPC 不属于 bat-api 管理控制面。 Rust dispatch、Go transport 和 bat-api 接口的权威实现位置分别是 diff --git a/docs/reports/CURRENT_GAPS.md b/docs/reports/CURRENT_GAPS.md index 06568ff..12df15b 100644 --- a/docs/reports/CURRENT_GAPS.md +++ b/docs/reports/CURRENT_GAPS.md @@ -232,8 +232,8 @@ issue 43 例外说明：新增的 `parse repack` 只编排已有 TextAsset、Typ - `cmd/bat-api`：组织 Rust `bat` 已发布 release 的启动前资源入口，并只读分发官方 CDN host/path 形态资源。 - **拉取归属 Rust `bat`**；`bat-api` 不做下载器。 - 发现经 `bat.sock`：先 `daemon.status`，再 `daemon.doctor`，再 `catalog.status` / `resource.manifest`。 -- Rust RPC 的 `resource.state`、`catalog.status`、`parse.status` 和 `localized.status` 会返回短状态 `status` 与稳定生命周期状态码 `status_code`；`backendrpc` 已提供对应 client 能力，当前 `bat-api` 发现流程只消费 `resource.state`、`catalog.status`、`resource.manifest`，parse/localized 不由 HTTP surface 暴露；错误原因仍以 `BAT-ERR-*` 为准。 -- `bat-api` 可通过受限 Web 控制面转发 `reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`，这些控制动作均走 Rust live RPC；`parse.*`、`localized.status` 和文件级 `unityfs.patch_*` 属于 `backendrpc` 能力，但当前不由 HTTP surface 暴露；`daemon.clean-stable` 等危险或离线生命周期命令不经 Web 转发。 +- Rust RPC 的 `resource.state`、`catalog.status`、`parse.status` 和 `localized.status` 会返回短状态 `status` 与稳定生命周期状态码 `status_code`；`backendrpc` 已提供对应 client 能力，当前 `bat-api` 发现流程只消费 `resource.state`、`catalog.status`、`resource.manifest`，translation task/handoff 通过受鉴权的 admin 查询端点暴露，parse/localized 不由 HTTP surface 暴露；错误原因仍以 `BAT-ERR-*` 为准。 +- `bat-api` 可通过受限 Web 控制面转发 `reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`，并通过 translation admin 端点查询/回写任务、触发 provider worker 和标记人工校对；这些动作均走 Rust live RPC；`parse.*`、`localized.status` 和文件级 `unityfs.patch_*` 属于 `backendrpc` 能力，但当前不由 HTTP surface 暴露；`daemon.clean-stable` 等危险或离线生命周期命令不经 Web 转发。 - 生产与 Rust `bat` 同环境运行，资源根来自 RPC 返回的 `resource_root`；`--resource-root` 仅用于 fixture 或应急只读诊断。 - `.env` 配置端口 / public base / RPC socket / RPC 刷新周期；预留 database/redis。 - `/v1/bootstrap` 返回 RPC 健康、release 摘要、server-info URL、client-patch base 和改写后的 Addressables root。 @@ -294,9 +294,9 @@ issue 43 例外说明：新增的 `parse repack` 只编排已有 TextAsset、Typ - 官方 release 发布后还会写入版本化 `translation-handoff.json`；`translation.handoff` RPC/CLI 动态合并该快照与 SQLite worker 状态，暴露 job、unit、provider run、attempt 和 failure reason。 - `translation-tasks.sqlite` 由 `schema_migrations` 管理 durable task state、attempt count、provider run ID 和 failure reason；重复同步会保留已有 worker 状态。 - `translation.tasks` RPC/CLI 优先查询 `translation-tasks.sqlite`，旧 release 没有状态库时回退到 `official-textunit-tasks.json`；返回队列 `status`、worker `task_status`、failure reason 和时间/尝试次数。 -- `translation.task.update` 已提供 queued/running/failed/completed/skipped 状态回写契约，并已通过 `bat i18n task update`、Go typed helper 和 bat-api 鉴权控制面暴露；`translation.proofread` 已提供人工校对状态标记契约，provider worker 可在消费 handoff 后按 task_id 更新并由同一查询接口反查。 +- `translation.tasks` / `translation.handoff` 已提供 Go typed helper 和 bat-api 鉴权查询端点；`translation.task.update` 已提供 queued/running/failed/completed/skipped 状态回写契约，并已通过 `bat i18n task update`、Go typed helper 和 bat-api 鉴权控制面暴露；`translation.worker.run` 已提供 Rust provider worker，支持 mock/Crowdin、独立 claim、lease、失败分类、重试和 TextUnit 级译文结果落库；`translation.proofread` 已提供人工校对状态标记契约。 - `i18n get/set/unset` 可查看、修改或清空单条工作台译文，`i18n validate` 可在发布前校验工作台 release、source text、重复 patch 目标，并区分可直接发布的 TextAsset 与必须进入 `parse repack` 的条目；`i18n proofread` 可把当前汉化 workflow 标记为人工校对中，且不会遮蔽已发布汉化 release；`parse clear-cache` 只删除可再生解析/队列 JSON，保留 `translation-tasks.sqlite` 的 worker 状态。 -- 真实 Crowdin 网络 worker、翻译记忆和完整 localized repack 仍属于后续翻译系统工作，不在当前 Rust 离线状态仓储范围内。 +- 翻译记忆和完整 localized repack 仍属于后续翻译系统工作，不在当前 provider worker 状态闭环范围内。 验收： diff --git a/docs/reports/GO_STATUS.md b/docs/reports/GO_STATUS.md index 3cf44f4..0fa506c 100644 --- a/docs/reports/GO_STATUS.md +++ b/docs/reports/GO_STATUS.md @@ -95,7 +95,7 @@ | 组件 | 路径 | 状态 | 说明 | |---|---|---|---| | Module | `go.mod` → `bat-api` | 已用 | 服务层模块名 | -| RPC client | `internal/backendrpc` | **完成** | Unix socket JSON-RPC transport + typed helper；typed helper 覆盖 daemon 已实现控制/查询、`resource.state/sync/verify/repair/manifest/list`、`catalog.*`、`parse.*`、`localized.status`、`task.*`、`translation.task.update`、`translation.proofread` 和文件级 UnityFS patch 调用；`resource.index`、其余 `translation.*`、`patch.apply` 仍通过通用 `Call` 走同一 contract；fake transport 单测，配合 `internal/api/testdata/contract/` 固化 Rust 输出 mirror | +| RPC client | `internal/backendrpc` | **完成** | Unix socket JSON-RPC transport + typed helper；typed helper 覆盖 daemon 已实现控制/查询、`resource.state/sync/verify/repair/manifest/list`、`catalog.*`、`parse.*`、`localized.status`、`task.*`、`translation.tasks`、`translation.handoff`、`translation.task.update`、`translation.worker.run`、`translation.proofread` 和文件级 UnityFS patch 调用；`resource.index`、`patch.apply` 仍通过通用 `Call` 走同一 contract；fake transport 单测，配合 `internal/api/testdata/contract/` 固化 Rust 输出 mirror | | 资源 bootstrap/分发 | `cmd/bat-api` + `internal/api` | **MVP+生产控制面** | RPC 发现 + 周期刷新/诊断 + `/v1/bootstrap` + `/v1/launcher/bootstrap` + launcher 资源 metadata 兼容 + `/readyz` + CDN Range/缓存头 + 鉴权/限流/访问日志/反代适配 + OpenAPI + 管理控制白名单 + `.env` | | 试验 CLI | `cmd/bat` | **试验** | doctor 固定 ok；manifest/sync 走 FFI | | FFI | `internal/ffi` | **可选** | 需 `build-ffi` | diff --git a/infrastructure/src/bin/bat/app.rs b/infrastructure/src/bin/bat/app.rs index c906e80..67ce70e 100644 --- a/infrastructure/src/bin/bat/app.rs +++ b/infrastructure/src/bin/bat/app.rs @@ -23,14 +23,16 @@ use bat_infrastructure::{ OfficialUpdateService, OfficialUpdateSnapshot, OfficialUpdateStatus, OfficialVerificationSummary, OfficialVersionRecord, OfficialVersionState, PatchApplyKind, PatchApplyParams, PatchApplyReport, ReleaseFlowStatusCode, RepackReport, - SqliteResourceRepository, SqliteTranslationTaskRepository, TranslationTaskStatus, - UnityFsFieldPatchParams, UnityFsPatchReport, UnityFsStringFieldPatchParams, - UnityFsTextAssetPatchParams, CROWDIN_TEXTUNIT_QUEUE_FILE, LOCALIZED_CURRENT_LINK, + SqliteResourceRepository, SqliteTranslationTaskRepository, TranslationProviderKind, + TranslationTaskStatus, TranslationWorkerConfig, UnityFsFieldPatchParams, UnityFsPatchReport, + UnityFsStringFieldPatchParams, UnityFsTextAssetPatchParams, CROWDIN_TEXTUNIT_QUEUE_FILE, + DEFAULT_TRANSLATION_CONCURRENCY, DEFAULT_TRANSLATION_LEASE_SECONDS, + DEFAULT_TRANSLATION_MAX_ATTEMPTS, DEFAULT_TRANSLATION_RETRY_BACKOFF, LOCALIZED_CURRENT_LINK, LOCALIZED_PATCH_MANIFEST_FILE, LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING, LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING_LABEL, LOCALIZED_VERSIONS_DIR, - LOCALIZED_VERSION_STATE_FILE, MAX_DOWNLOAD_CONCURRENCY, MIN_DOWNLOAD_CONCURRENCY, - OFFICIAL_PARSE_CACHE_FILE, OFFICIAL_TEXTUNIT_INDEX_FILE, OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE, - PRIVATE_FILE_MODE, + LOCALIZED_VERSION_STATE_FILE, MAX_DOWNLOAD_CONCURRENCY, MAX_TRANSLATION_CONCURRENCY, + MIN_DOWNLOAD_CONCURRENCY, MIN_TRANSLATION_CONCURRENCY, OFFICIAL_PARSE_CACHE_FILE, + OFFICIAL_TEXTUNIT_INDEX_FILE, OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE, PRIVATE_FILE_MODE, }; use serde::de::DeserializeOwned; use serde::{Deserialize, Serialize}; @@ -85,6 +87,7 @@ use workflow_commands::{ run_parse_clear_cache, run_parse_once, run_publish_localized, run_repack, run_translate_once, run_translation_get, run_translation_proofread, run_translation_set, run_translation_task_update, run_translation_unset, run_translation_validate, + run_translation_worker, }; const EXIT_ERROR: i32 = 1; @@ -229,6 +232,10 @@ fn run() -> anyhow::Result { run_translation_proofread(&options)?; Ok(0) } + CliCommand::TranslationWorker => { + run_repeated_workflow(&options, "translation-worker", run_translation_worker)?; + Ok(0) + } CliCommand::Repack => { run_repack(&options)?; Ok(0) @@ -378,6 +385,15 @@ struct CliOptions { translation_text_file: Option, translation_failure_reason: Option, translation_provider_run_id: Option, + translation_provider: Option, + translation_fixture: Option, + worker_concurrency: usize, + worker_max_attempts: u32, + worker_lease_seconds: u64, + worker_retry_backoff: Duration, + worker_max_tasks: Option, + worker_id: Option, + translation_worker_option_explicit: bool, localized_release_id: Option, repack_spec: Option, schedule_group: Option, @@ -463,6 +479,15 @@ impl Default for CliOptions { translation_text_file: None, translation_failure_reason: None, translation_provider_run_id: None, + translation_provider: None, + translation_fixture: None, + worker_concurrency: DEFAULT_TRANSLATION_CONCURRENCY, + worker_max_attempts: DEFAULT_TRANSLATION_MAX_ATTEMPTS, + worker_lease_seconds: DEFAULT_TRANSLATION_LEASE_SECONDS, + worker_retry_backoff: DEFAULT_TRANSLATION_RETRY_BACKOFF, + worker_max_tasks: None, + worker_id: None, + translation_worker_option_explicit: false, localized_release_id: None, repack_spec: None, schedule_group: None, @@ -546,6 +571,7 @@ enum CliCommand { TranslationGet, TranslationUnset, TranslationTaskUpdate, + TranslationWorker, TranslationProofread, Repack, PublishLocalized, @@ -1025,6 +1051,7 @@ const RPC_METHOD_TRANSLATION_TASKS: &str = "translation.tasks"; const RPC_METHOD_TRANSLATION_HANDOFF: &str = "translation.handoff"; const RPC_METHOD_TRANSLATION_TASK_UPDATE: &str = "translation.task.update"; const RPC_METHOD_TRANSLATION_PROOFREAD: &str = "translation.proofread"; +const RPC_METHOD_TRANSLATION_WORKER_RUN: &str = "translation.worker.run"; const RPC_METHOD_LOCALIZED_STATUS: &str = "localized.status"; const RPC_METHOD_CATALOG_STATUS: &str = "catalog.status"; const RPC_METHOD_CATALOG_VERSIONS: &str = "catalog.versions"; @@ -2027,6 +2054,13 @@ fn dispatch_rpc_method( mark_localized_manual_proofreading_report(state_dir, &tasks.base_config) .and_then(|report| serde_json::to_value(report).map_err(anyhow::Error::from)), ), + RPC_METHOD_TRANSLATION_WORKER_RUN => { + let config = match rpc_translation_worker_config(request.params.as_ref()) { + Ok(config) => config, + Err(error) => return rpc_envelope_error(request_id, error), + }; + enqueue_translation_worker_envelope(tasks, config, request_id) + } RPC_METHOD_LOCALIZED_STATUS => rpc_envelope_from_result( request_id, "localized.status", @@ -2211,7 +2245,9 @@ fn enqueue_task_envelope( .unwrap_or_else(|| Arc::new(AtomicBool::new(false))); let job = TaskJob { id: task_id.clone(), + kind, config, + translation_worker_config: None, cancel, }; if tasks.queue.send(job).is_err() { @@ -2237,6 +2273,50 @@ fn enqueue_task_envelope( ) } +fn enqueue_translation_worker_envelope( + tasks: &DaemonTaskContext, + worker_config: TranslationWorkerConfig, + request_id: String, +) -> RpcEnvelope { + let config = TaskKind::TranslationWorker.build_config(&tasks.base_config, false); + let task_id = tasks.registry.create(TaskKind::TranslationWorker); + let cancel = tasks + .registry + .cancel_flag(&task_id) + .unwrap_or_else(|| Arc::new(AtomicBool::new(false))); + let job = TaskJob { + id: task_id.clone(), + kind: TaskKind::TranslationWorker, + config, + translation_worker_config: Some(worker_config.clone()), + cancel, + }; + if tasks.queue.send(job).is_err() { + tasks.registry.update(&task_id, |record| { + record.status = "failed"; + record.finished_at = Some(unix_seconds_now()); + record.error = Some(ApiError::new( + ErrorCode::INTERNAL, + "task.enqueue", + "任务执行器不可用", + )); + }); + return rpc_envelope_error( + request_id, + ApiError::new(ErrorCode::INTERNAL, "task.enqueue", "任务执行器不可用"), + ); + } + rpc_envelope_ok( + request_id, + "accepted", + serde_json::json!({ + "task_id": task_id, + "kind": RPC_METHOD_TRANSLATION_WORKER_RUN, + "worker": worker_config, + }), + ) +} + fn rpc_ack_value( command: &'static str, message: &'static str, @@ -3070,6 +3150,207 @@ fn rpc_struct_params( }) } +fn rpc_translation_worker_config( + params: Option<&serde_json::Value>, +) -> Result { + let empty = serde_json::json!({}); + let params = params.unwrap_or(&empty); + if !params.is_object() { + return Err(ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + "params 必须是 JSON object", + )); + } + let provider = rpc_translation_worker_string_param( + params, + &["provider", "translation_provider"], + "provider", + )? + .unwrap_or_else(|| TranslationProviderKind::Mock.as_str().to_string()); + let provider = TranslationProviderKind::parse(&provider).ok_or_else(|| { + ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + format!("不支持的 translation provider：{provider}"), + ) + })?; + let concurrency = rpc_translation_worker_usize_param( + params, + &[ + "concurrency", + "worker_concurrency", + "translation_concurrency", + ], + "concurrency", + )? + .unwrap_or(DEFAULT_TRANSLATION_CONCURRENCY); + let max_attempts = rpc_translation_worker_u32_param( + params, + &[ + "max_attempts", + "worker_max_attempts", + "translation_max_attempts", + ], + "max_attempts", + )? + .unwrap_or(DEFAULT_TRANSLATION_MAX_ATTEMPTS); + let lease_seconds = rpc_translation_worker_u64_param( + params, + &[ + "lease_seconds", + "worker_lease_seconds", + "translation_lease_seconds", + ], + "lease_seconds", + )? + .unwrap_or(DEFAULT_TRANSLATION_LEASE_SECONDS); + let retry_backoff_seconds = rpc_translation_worker_u64_param( + params, + &[ + "retry_backoff_seconds", + "worker_retry_backoff_seconds", + "translation_retry_backoff_seconds", + ], + "retry_backoff_seconds", + )? + .unwrap_or(DEFAULT_TRANSLATION_RETRY_BACKOFF.as_secs()); + let max_tasks = rpc_translation_worker_usize_param( + params, + &["max_tasks", "worker_max_tasks", "translation_max_tasks"], + "max_tasks", + )?; + let config = TranslationWorkerConfig { + provider, + fixture_path: rpc_translation_worker_string_param( + params, + &[ + "fixture_path", + "translation_fixture", + "provider_fixture", + "mock_fixture", + "fixture", + ], + "fixture_path", + )? + .map(PathBuf::from), + concurrency, + max_attempts, + lease_seconds, + retry_backoff: Duration::from_secs(retry_backoff_seconds), + max_tasks, + worker_id: rpc_translation_worker_string_param( + params, + &["worker_id", "translation_worker_id"], + "worker_id", + )? + .unwrap_or_else(|| "bat-rpc-worker".to_string()), + }; + config.validate().map_err(|error| { + ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + error.to_string(), + ) + })?; + Ok(config) +} + +fn rpc_translation_worker_string_param( + params: &serde_json::Value, + aliases: &[&str], + label: &str, +) -> Result, ApiError> { + for key in aliases { + let Some(value) = params.get(*key) else { + continue; + }; + if value.is_null() { + return Ok(None); + } + let Some(text) = value.as_str() else { + return Err(ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + format!("{label} 必须是字符串"), + )); + }; + let text = text.trim(); + if text.is_empty() { + return Ok(None); + } + return Ok(Some(text.to_string())); + } + Ok(None) +} + +fn rpc_translation_worker_u64_param( + params: &serde_json::Value, + aliases: &[&str], + label: &str, +) -> Result, ApiError> { + for key in aliases { + let Some(value) = params.get(*key) else { + continue; + }; + if value.is_null() { + return Ok(None); + } + if value.as_i64().is_some_and(|number| number < 0) { + return Err(ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + format!("{label} 不能为负数"), + )); + } + let Some(number) = value.as_u64() else { + return Err(ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + format!("{label} 必须是非负整数 JSON number"), + )); + }; + return Ok(Some(number)); + } + Ok(None) +} + +fn rpc_translation_worker_usize_param( + params: &serde_json::Value, + aliases: &[&str], + label: &str, +) -> Result, ApiError> { + rpc_translation_worker_u64_param(params, aliases, label)? + .map(|number| { + usize::try_from(number).map_err(|error| { + ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + format!("{label} 无效：{error}"), + ) + }) + }) + .transpose() +} + +fn rpc_translation_worker_u32_param( + params: &serde_json::Value, + aliases: &[&str], + label: &str, +) -> Result, ApiError> { + rpc_translation_worker_u64_param(params, aliases, label)? + .map(|number| { + u32::try_from(number).map_err(|error| { + ApiError::new( + ErrorCode::RPC_INVALID_PARAMS, + RPC_METHOD_TRANSLATION_WORKER_RUN, + format!("{label} 无效：{error}"), + ) + }) + }) + .transpose() +} + /// 分页参数：`offset` 默认 0；`limit` 默认 100，范围 1..=1000。 fn rpc_page_params(params: Option<&serde_json::Value>) -> anyhow::Result<(usize, usize)> { let offset = params @@ -4221,6 +4502,32 @@ impl HumanReport for bat_infrastructure::LocalizedTranslationWorkflowReport { } } +impl HumanReport for bat_infrastructure::TranslationWorkerReport { + fn print_human(&self) -> anyhow::Result<()> { + print_title("翻译 provider worker"); + print_field("命令", self.command); + print_field("状态", self.status); + print_field("官方 release", &self.official_release_id); + print_field("provider", &self.provider); + print_field("回收 lease", self.recovered_lease_count); + print_field("领取任务", self.claimed_count); + print_field("完成任务", self.completed_count); + print_field("失败任务", self.failed_count); + print_field("已安排重试", self.retry_scheduled_count); + print_field("剩余任务", self.remaining_count); + for failure in &self.failures { + println!( + " - {} [{}] retryable={} {}", + failure.task_id, + failure.failure_class, + format_bool(failure.retryable), + failure.failure_reason + ); + } + Ok(()) + } +} + fn print_human_json_value(value: &serde_json::Value) -> anyhow::Result<()> { if value.get("running").is_some() && value.get("state_dir").is_some() { print_title("后台状态"); @@ -6321,6 +6628,17 @@ BAT_AUTO_DISCOVER=1 #BAT_DOWNLOAD_CONCURRENCY=8 #BAT_UNZIP=unzip +# ---- 翻译 provider worker ---- +# provider：mock（本地 fixture）或 crowdin（读取 CROWDIN_* 环境变量） +#BAT_TRANSLATION_PROVIDER=mock +#BAT_TRANSLATION_FIXTURE= +#BAT_TRANSLATION_CONCURRENCY=8 +#BAT_TRANSLATION_MAX_ATTEMPTS=3 +#BAT_TRANSLATION_LEASE_SECONDS=300 +#BAT_TRANSLATION_RETRY_BACKOFF_SECONDS=5 +#BAT_TRANSLATION_MAX_TASKS= +#BAT_TRANSLATION_WORKER_ID= + # ---- 输出 ---- # 设为 1 时输出机器可读 JSON（默认人类可读） #BAT_JSON=0 @@ -6504,6 +6822,38 @@ fn apply_bat_env_overrides( if let Some(v) = value("BAT_UNZIP") { options.config.unzip_command = PathBuf::from(v); } + if let Some(v) = value("BAT_TRANSLATION_PROVIDER") { + options.translation_provider = Some(v); + } + if let Some(v) = value("BAT_TRANSLATION_FIXTURE") { + options.translation_fixture = Some(PathBuf::from(v)); + } + if let Some(v) = value("BAT_TRANSLATION_CONCURRENCY") { + options.worker_concurrency = + parse_translation_worker_concurrency(&v, "环境变量 BAT_TRANSLATION_CONCURRENCY")?; + } + if let Some(v) = value("BAT_TRANSLATION_MAX_ATTEMPTS") { + options.worker_max_attempts = + parse_positive_u32(&v, "环境变量 BAT_TRANSLATION_MAX_ATTEMPTS")?; + } + if let Some(v) = value("BAT_TRANSLATION_LEASE_SECONDS") { + options.worker_lease_seconds = + parse_positive_u64(&v, "环境变量 BAT_TRANSLATION_LEASE_SECONDS")?; + } + if let Some(v) = value("BAT_TRANSLATION_RETRY_BACKOFF_SECONDS") { + options.worker_retry_backoff = Duration::from_secs(v.parse::().map_err(|error| { + anyhow::anyhow!("环境变量 BAT_TRANSLATION_RETRY_BACKOFF_SECONDS 的秒数无效：{error}") + })?); + } + if let Some(v) = value("BAT_TRANSLATION_MAX_TASKS") { + options.worker_max_tasks = Some(parse_positive_usize( + &v, + "环境变量 BAT_TRANSLATION_MAX_TASKS", + )?); + } + if let Some(v) = value("BAT_TRANSLATION_WORKER_ID") { + options.worker_id = Some(v); + } if let Some(v) = value("BAT_PROXY") { options.config.curl_proxy = parse_proxy_config(&v)?; } @@ -6725,6 +7075,55 @@ fn parse_args_with_env( "--provider-run-id" => { options.translation_provider_run_id = Some(next_option_value(&mut args, &flag)?); } + "--translation-provider" | "--provider" => { + options.translation_provider = Some(next_option_value(&mut args, &flag)?); + options.translation_worker_option_explicit = true; + } + "--translation-fixture" | "--provider-fixture" | "--mock-fixture" | "--fixture" => { + options.translation_fixture = + Some(PathBuf::from(next_option_value(&mut args, &flag)?)); + options.translation_worker_option_explicit = true; + } + "--worker-concurrency" | "--translation-concurrency" => { + options.worker_concurrency = parse_translation_worker_concurrency( + &next_option_value(&mut args, &flag)?, + &flag, + )?; + options.translation_worker_option_explicit = true; + } + "--worker-max-attempts" | "--translation-max-attempts" => { + options.worker_max_attempts = + parse_positive_u32(&next_option_value(&mut args, &flag)?, &flag)?; + options.translation_worker_option_explicit = true; + } + "--worker-lease-seconds" | "--translation-lease-seconds" => { + options.worker_lease_seconds = + parse_positive_u64(&next_option_value(&mut args, &flag)?, &flag)?; + options.translation_worker_option_explicit = true; + } + "--worker-retry-backoff" | "--translation-retry-backoff" => { + options.worker_retry_backoff = + parse_duration(&next_option_value(&mut args, &flag)?)?; + options.translation_worker_option_explicit = true; + } + "--worker-retry-backoff-seconds" | "--translation-retry-backoff-seconds" => { + let seconds = next_option_value(&mut args, &flag)? + .parse::() + .map_err(|error| anyhow::anyhow!("{flag} 的秒数无效：{error}"))?; + options.worker_retry_backoff = Duration::from_secs(seconds); + options.translation_worker_option_explicit = true; + } + "--worker-max-tasks" | "--translation-max-tasks" => { + options.worker_max_tasks = Some(parse_positive_usize( + &next_option_value(&mut args, &flag)?, + &flag, + )?); + options.translation_worker_option_explicit = true; + } + "--worker-id" | "--translation-worker-id" => { + options.worker_id = Some(next_option_value(&mut args, &flag)?); + options.translation_worker_option_explicit = true; + } "--localized-release-id" => { options.localized_release_id = Some(next_option_value(&mut args, &flag)?); } @@ -7185,6 +7584,12 @@ fn parse_args_with_env( )); } + if options.command != CliCommand::TranslationWorker + && options.translation_worker_option_explicit + { + return Err(anyhow::anyhow!("翻译 worker 参数只适用于 i18n worker run")); + } + match options.command { CliCommand::Status | CliCommand::Stop | CliCommand::Logs => { if options.sync_option_explicit @@ -7406,6 +7811,56 @@ fn parse_args_with_env( options.progress = false; options.banner = false; } + CliCommand::TranslationWorker => { + if options.daemon || options.daemon_child { + return Err(anyhow::anyhow!( + "i18n worker run 使用 --watch 或 --run-count，不支持 daemon" + )); + } + if options.config.force + || options.config.dry_run + || options.schedule_option_explicit + || options.query_option_explicit + || options.write_patch_option_explicit + || options.translation_file.is_some() + || options.translation_id.is_some() + || options.translation_text.is_some() + || options.translation_text_file.is_some() + || options.translation_failure_reason.is_some() + || options.translation_provider_run_id.is_some() + || options.localized_release_id.is_some() + || options.repack_spec.is_some() + || options.proxy_option_explicit + || tools_are_non_default(&options.config, &options.env_baseline_config) + { + return Err(anyhow::anyhow!( + "i18n worker run 只接受 --output、--resource-root、--state-dir、worker 参数和轮询参数" + )); + } + let provider = options + .translation_provider + .as_deref() + .unwrap_or(TranslationProviderKind::Mock.as_str()); + if TranslationProviderKind::parse(provider).is_none() { + return Err(anyhow::anyhow!( + "i18n worker run 的 provider 无效：{provider}" + )); + } + TranslationWorkerConfig { + provider: TranslationProviderKind::parse(provider).expect("provider 已在上方校验"), + fixture_path: options.translation_fixture.clone(), + concurrency: options.worker_concurrency, + max_attempts: options.worker_max_attempts, + lease_seconds: options.worker_lease_seconds, + retry_backoff: options.worker_retry_backoff, + max_tasks: options.worker_max_tasks, + worker_id: options + .worker_id + .clone() + .unwrap_or_else(|| "bat-worker-validation".to_string()), + } + .validate()?; + } CliCommand::TranslationProofread => { if options.watch || options.daemon || options.daemon_child { return Err(anyhow::anyhow!("i18n proofread 只支持单次执行或 RPC 调用")); @@ -7506,6 +7961,7 @@ fn parse_args_with_env( CliCommand::Pull | CliCommand::Parse | CliCommand::Translate + | CliCommand::TranslationWorker | CliCommand::PublishLocalized ) { @@ -7515,7 +7971,11 @@ fn parse_args_with_env( } if matches!( options.command, - CliCommand::Pull | CliCommand::Parse | CliCommand::Translate | CliCommand::PublishLocalized + CliCommand::Pull + | CliCommand::Parse + | CliCommand::Translate + | CliCommand::PublishLocalized + | CliCommand::TranslationWorker ) { if options.watch && options.run_count.is_some() { return Err(anyhow::anyhow!( @@ -7632,6 +8092,9 @@ fn parse_translation_command( if action == "workbench" || action == "wb" { return parse_translation_workbench_command(args, options); } + if action == "worker" { + return parse_translation_worker_command(args, options); + } let command = match action.as_str() { "run" => CliCommand::Translate, "export" => CliCommand::Translate, @@ -7655,6 +8118,20 @@ fn parse_translation_command( Ok(()) } +fn parse_translation_worker_command( + args: &mut impl Iterator, + options: &mut CliOptions, +) -> anyhow::Result<()> { + let action = next_option_value(args, "translation worker")?; + let command = match action.as_str() { + "run" => CliCommand::TranslationWorker, + other => return Err(anyhow::anyhow!("未知 translation worker 二级命令：{other}")), + }; + ensure_command_not_set(options.command, &format!("translation worker {action}"))?; + options.command = command; + Ok(()) +} + fn parse_translation_workbench_command( args: &mut impl Iterator, options: &mut CliOptions, @@ -7738,6 +8215,48 @@ fn parse_download_concurrency(value: &str, source: &str) -> anyhow::Result anyhow::Result { + let parsed = value + .parse::() + .map_err(|error| anyhow::anyhow!("{source} 无效：{error}"))?; + if !(MIN_TRANSLATION_CONCURRENCY..=MAX_TRANSLATION_CONCURRENCY).contains(&parsed) { + return Err(anyhow::anyhow!( + "{source} 必须在 {MIN_TRANSLATION_CONCURRENCY}..={MAX_TRANSLATION_CONCURRENCY} 范围内" + )); + } + Ok(parsed) +} + +fn parse_positive_usize(value: &str, source: &str) -> anyhow::Result { + let parsed = value + .parse::() + .map_err(|error| anyhow::anyhow!("{source} 无效：{error}"))?; + if parsed == 0 { + return Err(anyhow::anyhow!("{source} 必须大于 0")); + } + Ok(parsed) +} + +fn parse_positive_u32(value: &str, source: &str) -> anyhow::Result { + let parsed = value + .parse::() + .map_err(|error| anyhow::anyhow!("{source} 无效：{error}"))?; + if parsed == 0 { + return Err(anyhow::anyhow!("{source} 必须大于 0")); + } + Ok(parsed) +} + +fn parse_positive_u64(value: &str, source: &str) -> anyhow::Result { + let parsed = value + .parse::() + .map_err(|error| anyhow::anyhow!("{source} 无效：{error}"))?; + if parsed == 0 { + return Err(anyhow::anyhow!("{source} 必须大于 0")); + } + Ok(parsed) +} + fn next_option_value( args: &mut impl Iterator, flag: &str, @@ -7807,6 +8326,7 @@ fn print_usage(binary: &str) { eprintln!(" i18n unset Clear one translated workbench entry"); eprintln!(" i18n validate Validate workbench against the current official release"); eprintln!(" i18n proofread Mark localized workflow as manual proofreading"); + eprintln!(" i18n worker run Run translation provider worker once or repeatedly"); eprintln!( " i18n tasks / i18n task list / i18n task status Query current offline TextUnit translation task status" ); @@ -7854,6 +8374,9 @@ fn print_usage(binary: &str) { " {binary} i18n unset --translation-file /tmp/bat-workbench.json --translation-id unit-1" ); eprintln!(" {binary} i18n proofread --json"); + eprintln!( + " {binary} i18n worker run --provider mock --worker-concurrency 8 --run-count 2 --interval 30s" + ); eprintln!(" {binary} i18n tasks --json"); eprintln!(" {binary} i18n handoff --json"); eprintln!(" {binary} i18n status --json"); @@ -7908,6 +8431,17 @@ fn print_usage(binary: &str) { eprintln!(" --translated-file UTF-8 translation file for i18n set"); eprintln!(" --failure-reason Provider failure reason for i18n task update"); eprintln!(" --provider-run-id Provider run ID for i18n task update"); + eprintln!(" --translation-provider / --provider Provider for i18n worker run (mock/crowdin)"); + eprintln!(" --translation-fixture Mock/provider fixture for i18n worker run"); + eprintln!( + " --worker-concurrency Translation worker concurrency (default: 8, range 1..=256)" + ); + eprintln!(" --worker-max-attempts Maximum claims per translation task"); + eprintln!(" --worker-lease-seconds Lease seconds for one claimed task"); + eprintln!(" --worker-retry-backoff Retry backoff after retryable failure"); + eprintln!(" --worker-retry-backoff-seconds Retry backoff seconds"); + eprintln!(" --worker-max-tasks Maximum tasks claimed in one worker run"); + eprintln!(" --worker-id Worker ID prefix for lease diagnostics"); eprintln!(" --localized-release-id Explicit localized publication ID"); eprintln!(" --repack-spec UnityFS batch repack JSON spec"); eprintln!(); diff --git a/infrastructure/src/bin/bat/app_tests.rs b/infrastructure/src/bin/bat/app_tests.rs index ba771b1..7679498 100644 --- a/infrastructure/src/bin/bat/app_tests.rs +++ b/infrastructure/src/bin/bat/app_tests.rs @@ -140,6 +140,165 @@ fn env_invalid_values_error() { assert!(parse(&["bat", "--download-concurrency", "257"]).is_err()); } +#[test] +fn translation_worker_command_options_are_validated() { + let options = parse(&[ + "bat", + "i18n", + "worker", + "run", + "--provider", + "mock", + "--translation-fixture", + "/tmp/mock-translation.json", + "--worker-concurrency", + "8", + "--worker-max-attempts", + "4", + "--worker-lease-seconds", + "60", + "--worker-retry-backoff", + "0s", + "--worker-max-tasks", + "2", + "--worker-id", + "manual-run", + "--run-count", + "2", + "--interval", + "30s", + ]) + .unwrap(); + assert_eq!(options.command, CliCommand::TranslationWorker); + assert_eq!(options.translation_provider.as_deref(), Some("mock")); + assert_eq!( + options.translation_fixture, + Some(PathBuf::from("/tmp/mock-translation.json")) + ); + assert_eq!(options.worker_concurrency, 8); + assert_eq!(options.worker_max_attempts, 4); + assert_eq!(options.worker_lease_seconds, 60); + assert_eq!(options.worker_retry_backoff, Duration::ZERO); + assert_eq!(options.worker_max_tasks, Some(2)); + assert_eq!(options.worker_id.as_deref(), Some("manual-run")); + assert_eq!(options.run_count, Some(2)); + + assert!(parse(&["bat", "i18n", "worker", "run"]).is_ok()); + assert!(parse(&[ + "bat", + "translation", + "worker", + "run", + "--translation-concurrency", + "256" + ]) + .is_ok()); + assert!(parse(&["bat", "i18n", "worker", "run", "--worker-concurrency", "0"]).is_err()); + assert!(parse(&[ + "bat", + "i18n", + "worker", + "run", + "--worker-concurrency", + "257" + ]) + .is_err()); + assert!(parse(&["bat", "i18n", "worker", "run", "--worker-max-attempts", "0"]).is_err()); + assert!(parse(&[ + "bat", + "i18n", + "worker", + "run", + "--worker-lease-seconds", + "0" + ]) + .is_err()); + assert!(parse(&["bat", "i18n", "worker", "run", "--worker-max-tasks", "0"]).is_err()); + assert!(parse(&["bat", "i18n", "run", "--worker-concurrency", "8"]).is_err()); + assert!(parse(&["bat", "i18n", "worker", "run", "--provider", "unknown"]).is_err()); +} + +#[test] +fn translation_worker_env_defaults_apply() { + let options = parse_with_env( + &["bat", "i18n", "worker", "run"], + &[ + ("BAT_TRANSLATION_PROVIDER", "mock"), + ("BAT_TRANSLATION_FIXTURE", "/tmp/fixture.json"), + ("BAT_TRANSLATION_CONCURRENCY", "16"), + ("BAT_TRANSLATION_MAX_ATTEMPTS", "5"), + ("BAT_TRANSLATION_LEASE_SECONDS", "120"), + ("BAT_TRANSLATION_RETRY_BACKOFF_SECONDS", "0"), + ("BAT_TRANSLATION_MAX_TASKS", "7"), + ("BAT_TRANSLATION_WORKER_ID", "env-worker"), + ], + ) + .unwrap(); + assert_eq!(options.command, CliCommand::TranslationWorker); + assert_eq!(options.translation_provider.as_deref(), Some("mock")); + assert_eq!( + options.translation_fixture, + Some(PathBuf::from("/tmp/fixture.json")) + ); + assert_eq!(options.worker_concurrency, 16); + assert_eq!(options.worker_max_attempts, 5); + assert_eq!(options.worker_lease_seconds, 120); + assert_eq!(options.worker_retry_backoff, Duration::ZERO); + assert_eq!(options.worker_max_tasks, Some(7)); + assert_eq!(options.worker_id.as_deref(), Some("env-worker")); + + assert!(parse_with_env( + &["bat", "i18n", "worker", "run"], + &[("BAT_TRANSLATION_CONCURRENCY", "257")] + ) + .is_err()); + assert!(parse_with_env( + &["bat", "i18n", "worker", "run"], + &[("BAT_TRANSLATION_MAX_ATTEMPTS", "0")] + ) + .is_err()); +} + +#[test] +fn translation_worker_rpc_params_are_strict_and_accept_aliases() { + let default_config = rpc_translation_worker_config(None).unwrap(); + assert_eq!(default_config.provider, TranslationProviderKind::Mock); + assert_eq!(default_config.concurrency, DEFAULT_TRANSLATION_CONCURRENCY); + + let config = rpc_translation_worker_config(Some(&serde_json::json!({ + "translation_provider": "mock", + "provider_fixture": "/tmp/mock-provider.json", + "worker_concurrency": 8, + "worker_max_attempts": 4, + "worker_lease_seconds": 60, + "worker_retry_backoff_seconds": 0, + "worker_max_tasks": 2, + "translation_worker_id": "rpc-worker" + }))) + .unwrap(); + assert_eq!(config.provider, TranslationProviderKind::Mock); + assert_eq!( + config.fixture_path, + Some(PathBuf::from("/tmp/mock-provider.json")) + ); + assert_eq!(config.concurrency, 8); + assert_eq!(config.max_attempts, 4); + assert_eq!(config.lease_seconds, 60); + assert_eq!(config.retry_backoff, Duration::ZERO); + assert_eq!(config.max_tasks, Some(2)); + assert_eq!(config.worker_id, "rpc-worker"); + + assert!(rpc_translation_worker_config(Some(&serde_json::json!([]))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"provider": "bad"}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": "8"}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": -1}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": 0}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": 257}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"max_attempts": 0}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"lease_seconds": 0}))).is_err()); + assert!(rpc_translation_worker_config(Some(&serde_json::json!({"max_tasks": 0}))).is_err()); +} + #[test] fn grouped_workflow_commands_use_short_top_level_aliases() { let options = parse(&[ diff --git a/infrastructure/src/bin/bat/task_registry.rs b/infrastructure/src/bin/bat/task_registry.rs index 50cc467..3b3e64f 100644 --- a/infrastructure/src/bin/bat/task_registry.rs +++ b/infrastructure/src/bin/bat/task_registry.rs @@ -4,12 +4,14 @@ pub(super) const MAX_RETAINED_TASKS: usize = 64; /// 每个任务保留的进度日志行数上限。 pub(super) const MAX_TASK_LOG_LINES: usize = 200; -/// 任务类型：目前覆盖官方同步、校验与 catalog 更新检查。 -#[derive(Debug, Clone, Copy)] +/// 任务类型：覆盖资源同步、校验、修复、翻译 worker 与 catalog 更新检查。 +#[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(super) enum TaskKind { Sync, Verify, Repair, + /// 翻译 provider worker 轮次。 + TranslationWorker, /// catalog 更新检查：只做发现 + 拉取计划（dry-run），不下载不审计。 Refresh, } @@ -20,6 +22,7 @@ impl TaskKind { Self::Sync => RPC_METHOD_RESOURCE_SYNC, Self::Verify => RPC_METHOD_RESOURCE_VERIFY, Self::Repair => RPC_METHOD_RESOURCE_REPAIR, + Self::TranslationWorker => RPC_METHOD_TRANSLATION_WORKER_RUN, Self::Refresh => RPC_METHOD_CATALOG_REFRESH, } } @@ -56,6 +59,10 @@ impl TaskKind { config.repair = false; config.force = force; } + Self::TranslationWorker => { + config.dry_run = false; + config.force = false; + } } config } @@ -155,6 +162,7 @@ fn task_kind_static(kind: &str) -> Option<&'static str> { RPC_METHOD_RESOURCE_SYNC => Some(RPC_METHOD_RESOURCE_SYNC), RPC_METHOD_RESOURCE_VERIFY => Some(RPC_METHOD_RESOURCE_VERIFY), RPC_METHOD_RESOURCE_REPAIR => Some(RPC_METHOD_RESOURCE_REPAIR), + RPC_METHOD_TRANSLATION_WORKER_RUN => Some(RPC_METHOD_TRANSLATION_WORKER_RUN), RPC_METHOD_CATALOG_REFRESH => Some(RPC_METHOD_CATALOG_REFRESH), _ => None, } @@ -494,7 +502,9 @@ impl TaskStore { /// 提交给任务 worker 的作业（配置已按任务类型派生完毕）。 pub(super) struct TaskJob { pub(super) id: String, + pub(super) kind: TaskKind, pub(super) config: OfficialUpdateConfig, + pub(super) translation_worker_config: Option, /// 与任务记录共享的取消标志。 pub(super) cancel: Arc, } @@ -528,36 +538,68 @@ pub(super) fn run_task_worker( }); let cancel = Arc::clone(&job.cancel); - let run_result = { - let _sync_guard = sync_lock - .lock() - .unwrap_or_else(|poison| poison.into_inner()); - let progress_registry = registry.clone(); - let progress_id = job.id.clone(); - let cancel_check = Arc::clone(&cancel); - let stop_control = Arc::clone(&control); - service.run_with_progress_and_cancellation( - &job.config, - |event| { - progress_registry - .append_log(&progress_id, format!("[{}] {}", event.stage, event.message)); - progress_registry.update(&progress_id, |record| { - record.stage = Some(event.stage.to_string()); - record.message = Some(event.message.clone()); - }); - }, - || { - cancel_check.load(Ordering::Relaxed) - || daemon_control_stop_requested(Some(&stop_control)) - }, - ) + let run_result = if job.kind == TaskKind::TranslationWorker { + let worker_config = job + .translation_worker_config + .as_ref() + .ok_or_else(|| anyhow::anyhow!("translation worker 任务缺少运行配置")); + worker_config.and_then(|worker_config| { + let _sync_guard = sync_lock + .lock() + .unwrap_or_else(|poison| poison.into_inner()); + registry.append_log(&job.id, "[translation-worker] 开始执行".to_string()); + registry.update(&job.id, |record| { + record.stage = Some("translation-worker".to_string()); + record.message = Some("翻译 provider worker 正在执行".to_string()); + }); + let resource_root = active_official_resource_root(&job.config.output_root)?; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build()?; + runtime + .block_on(bat_infrastructure::run_translation_worker_at( + &resource_root, + worker_config, + )) + .and_then(|report| serde_json::to_value(report).map_err(anyhow::Error::from)) + }) + } else { + let run_result = { + let _sync_guard = sync_lock + .lock() + .unwrap_or_else(|poison| poison.into_inner()); + let progress_registry = registry.clone(); + let progress_id = job.id.clone(); + let cancel_check = Arc::clone(&cancel); + let stop_control = Arc::clone(&control); + service.run_with_progress_and_cancellation( + &job.config, + |event| { + progress_registry.append_log( + &progress_id, + format!("[{}] {}", event.stage, event.message), + ); + progress_registry.update(&progress_id, |record| { + record.stage = Some(event.stage.to_string()); + record.message = Some(event.message.clone()); + }); + }, + || { + cancel_check.load(Ordering::Relaxed) + || daemon_control_stop_requested(Some(&stop_control)) + }, + ) + }; + run_result + .map(|report| serde_json::to_value(&report).map_err(anyhow::Error::from)) + .and_then(|result| result) }; match run_result { Ok(report) => registry.update(&job.id, |record| { record.status = "succeeded"; record.finished_at = Some(unix_seconds_now()); - record.result = serde_json::to_value(&report).ok(); + record.result = Some(report); }), Err(error) => { let cancelled = cancel.load(Ordering::Relaxed); diff --git a/infrastructure/src/bin/bat/workflow_commands.rs b/infrastructure/src/bin/bat/workflow_commands.rs index 9de8496..a9f03ea 100644 --- a/infrastructure/src/bin/bat/workflow_commands.rs +++ b/infrastructure/src/bin/bat/workflow_commands.rs @@ -253,6 +253,43 @@ pub(super) fn run_translation_proofread(options: &CliOptions) -> anyhow::Result< print_report(options.output_format, &report) } +pub(super) fn run_translation_worker(options: &CliOptions) -> anyhow::Result<()> { + let resource_root = options + .resource_root + .clone() + .map(|path| lexical_absolute(&path).map_err(anyhow::Error::msg)) + .transpose()? + .unwrap_or(active_official_resource_root(&options.config.output_root)?); + let provider = options + .translation_provider + .as_deref() + .unwrap_or(TranslationProviderKind::Mock.as_str()); + let provider = TranslationProviderKind::parse(provider) + .ok_or_else(|| anyhow::anyhow!("i18n worker run 的 provider 无效：{provider}"))?; + let config = TranslationWorkerConfig { + provider, + fixture_path: options.translation_fixture.clone(), + concurrency: options.worker_concurrency, + max_attempts: options.worker_max_attempts, + lease_seconds: options.worker_lease_seconds, + retry_backoff: options.worker_retry_backoff, + max_tasks: options.worker_max_tasks, + worker_id: options + .worker_id + .clone() + .unwrap_or_else(|| format!("bat-worker-{}", std::process::id())), + }; + config.validate()?; + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build()?; + let report = runtime.block_on(bat_infrastructure::run_translation_worker_at( + &resource_root, + &config, + ))?; + print_report(options.output_format, &report) +} + pub(super) fn run_repack(options: &CliOptions) -> anyhow::Result<()> { let spec = options .repack_spec diff --git a/infrastructure/src/downloader.rs b/infrastructure/src/downloader.rs index aa0edeb..525639a 100644 --- a/infrastructure/src/downloader.rs +++ b/infrastructure/src/downloader.rs @@ -156,7 +156,7 @@ mod tests { use super::*; use std::sync::atomic::{AtomicUsize, Ordering}; use std::thread; - use std::time::Duration; + use std::time::{Duration, Instant}; struct TestBackend { active: AtomicUsize, @@ -222,7 +222,15 @@ mod tests { fn download(&self, task: usize) -> Result { if task == 0 { self.active.fetch_add(1, Ordering::SeqCst); - thread::sleep(Duration::from_millis(20)); + let deadline = Instant::now() + Duration::from_millis(500); + while self + .task_two_started_while_task_zero_active + .load(Ordering::SeqCst) + == 0 + && Instant::now() < deadline + { + thread::sleep(Duration::from_millis(1)); + } self.active.fetch_sub(1, Ordering::SeqCst); } else { if task == 1 { diff --git a/infrastructure/src/lib.rs b/infrastructure/src/lib.rs index 9e83e6b..2c7d66c 100644 --- a/infrastructure/src/lib.rs +++ b/infrastructure/src/lib.rs @@ -30,6 +30,7 @@ pub mod path_security; pub mod release_flow; pub mod resources; pub mod translation_tasks; +pub mod translation_worker; pub mod translation_workflow; mod zip_validation; @@ -138,11 +139,21 @@ pub use translation_tasks::{ build_translation_handoff, read_translation_handoff_at, sync_translation_task_repository_at, write_translation_handoff_at, PersistedTranslationTask, PersistedTranslationTaskState, ProviderRun, ProviderRunStatus, SqliteTranslationTaskRepository, TranslationHandoff, - TranslationJob, TranslationJobStatus, TranslationTaskStatus, TranslationTaskSyncReport, - TranslationUnit, TranslationUnitStatus, TRANSLATION_HANDOFF_FILE, - TRANSLATION_HANDOFF_SCHEMA_VERSION, TRANSLATION_TASK_REPOSITORY_FILE, + TranslationJob, TranslationJobStatus, TranslationTaskFailure, TranslationTaskStatus, + TranslationTaskSyncReport, TranslationTaskUnitResult, TranslationUnit, TranslationUnitStatus, + TRANSLATION_HANDOFF_FILE, TRANSLATION_HANDOFF_SCHEMA_VERSION, TRANSLATION_TASK_REPOSITORY_FILE, TRANSLATION_TASK_SCHEMA_VERSION, }; +pub use translation_worker::{ + run_translation_worker_at, run_translation_worker_with_provider, CrowdinProvider, + MockTranslationProvider, TranslationProvider, TranslationProviderFailureClass, + TranslationProviderKind, TranslationProviderRequest, TranslationProviderResponse, + TranslationProviderUnit, TranslationProviderUnitResult, TranslationWorkerConfig, + TranslationWorkerFailure, TranslationWorkerReport, DEFAULT_TRANSLATION_CONCURRENCY, + DEFAULT_TRANSLATION_LEASE_SECONDS, DEFAULT_TRANSLATION_MAX_ATTEMPTS, + DEFAULT_TRANSLATION_RETRY_BACKOFF, MAX_TRANSLATION_CONCURRENCY, MIN_TRANSLATION_CONCURRENCY, + MOCK_TRANSLATION_FIXTURE_VERSION, +}; pub use translation_workflow::{ export_translation_workbench, get_translation_entry, localized_text_asset_patches, read_translation_workbench, repack_bundle, set_translation, unset_translation, diff --git a/infrastructure/src/translation_tasks.rs b/infrastructure/src/translation_tasks.rs index 0ac0361..99c91e6 100644 --- a/infrastructure/src/translation_tasks.rs +++ b/infrastructure/src/translation_tasks.rs @@ -14,15 +14,15 @@ use crate::path_security::{ }; use bat_core::Result; use serde::{Deserialize, Serialize}; -use sqlx::sqlite::{SqliteConnectOptions, SqlitePoolOptions}; +use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePoolOptions}; use sqlx::{QueryBuilder, Sqlite, SqlitePool}; use std::collections::{BTreeMap, BTreeSet}; use std::path::Path; use std::str::FromStr; -use std::time::{SystemTime, UNIX_EPOCH}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; /// SQLite schema version for durable translation task state. -pub const TRANSLATION_TASK_SCHEMA_VERSION: u32 = 1; +pub const TRANSLATION_TASK_SCHEMA_VERSION: u32 = 2; const TRANSLATION_TASK_SCHEMA_COMPONENT: &str = "translation_tasks"; /// SQLite file name stored under a published official release root. pub const TRANSLATION_TASK_REPOSITORY_FILE: &str = "translation-tasks.sqlite"; @@ -147,6 +147,26 @@ pub struct TranslationUnit { pub status: TranslationUnitStatus, /// Provider failure diagnostic, when present. pub failure_reason: Option, + /// TextUnit-level provider results associated with this resource task. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub translation_results: Vec, +} + +/// One provider result linked to an immutable TextUnit. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TranslationTaskUnitResult { + /// Stable TextUnit ID from `official-textunit-index.json`. + pub unit_id: String, + /// Source text submitted to the provider. + pub source_text: String, + /// Provider-produced or human-supplied translation. + pub translated_text: String, + /// Provider identifier. + pub provider: String, + /// Provider run that produced this result. + pub provider_run_id: String, + /// Result persistence time. + pub translated_unix_seconds: u64, } /// One provider execution associated with one or more translation units. @@ -203,7 +223,16 @@ pub fn build_translation_handoff( let state = persisted .get(task.task_id.as_str()) .copied() - .map(|task| (task.task_status, task.failure_reason.clone(), task.attempt_count, task.provider_run_id.clone())) + .map(|task| { + ( + task.task_status, + task.failure_reason.clone(), + task.attempt_count, + task.provider_run_id.clone(), + task.provider.clone(), + task.translation_results.clone(), + ) + }) .unwrap_or_else(|| { ( initial_task_status(task), @@ -214,6 +243,8 @@ pub fn build_translation_handoff( }, 0, None, + None, + Vec::new(), ) }); let unit_status = match state.0 { @@ -234,6 +265,7 @@ pub fn build_translation_handoff( text_unit_formats: task.text_unit_formats.clone(), status: unit_status, failure_reason: state.1.clone(), + translation_results: state.5.clone(), }; if let Some(provider_run_id) = state.3 { let run = provider_runs @@ -241,7 +273,7 @@ pub fn build_translation_handoff( .or_insert_with(|| ProviderRun { provider_run_id, job_id: job_id.clone(), - provider: "worker".to_string(), + provider: state.4.unwrap_or_else(|| "worker".to_string()), status: ProviderRunStatus::Queued, unit_ids: Vec::new(), attempt_count: 0, @@ -403,6 +435,26 @@ pub struct PersistedTranslationTask { /// Provider-side run identifier, if known. #[serde(skip_serializing_if = "Option::is_none")] pub provider_run_id: Option, + /// Provider-produced results keyed by TextUnit ID. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub translation_results: Vec, + /// Provider identifier used by the latest run. + #[serde(skip_serializing_if = "Option::is_none")] + pub provider: Option, + /// Worker currently holding the lease. + #[serde(skip_serializing_if = "Option::is_none")] + pub lease_owner: Option, + /// Lease expiry as Unix seconds. + #[serde(skip_serializing_if = "Option::is_none")] + pub lease_expires_unix_seconds: Option, + /// Stable failure classification. + #[serde(skip_serializing_if = "Option::is_none")] + pub failure_class: Option, + /// Whether the latest failure can be retried. + pub failure_retryable: bool, + /// Earliest Unix time at which a failed task may be claimed again. + #[serde(skip_serializing_if = "Option::is_none")] + pub next_attempt_unix_seconds: Option, } /// Mutable persistence metadata associated with an official TextUnit task. @@ -422,6 +474,39 @@ pub struct PersistedTranslationTaskState { pub completed_unix_seconds: Option, /// Provider-side run identifier, if known. pub provider_run_id: Option, + /// Provider-produced results keyed by TextUnit ID. + pub translation_results: Vec, + /// Provider identifier used by the latest run. + pub provider: Option, + /// Worker currently holding the lease. + pub lease_owner: Option, + /// Lease expiry as Unix seconds. + pub lease_expires_unix_seconds: Option, + /// Stable failure classification. + pub failure_class: Option, + /// Whether the latest failure can be retried. + pub failure_retryable: bool, + /// Earliest Unix time at which a failed task may be claimed again. + pub next_attempt_unix_seconds: Option, +} + +/// Provider failure data used to atomically release a task lease. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TranslationTaskFailure { + /// Task being failed. + pub task_id: String, + /// Worker that owns the lease. + pub worker_id: String, + /// Provider run associated with the lease. + pub provider_run_id: String, + /// Stable failure classification. + pub failure_class: String, + /// Redacted diagnostic message. + pub failure_reason: String, + /// Whether a future attempt may retry the task. + pub retryable: bool, + /// Earliest retry time, when retryable. + pub next_attempt_unix_seconds: Option, } /// Result of synchronizing an immutable release queue into SQLite. @@ -463,7 +548,9 @@ impl SqliteTranslationTaskRepository { let options = SqliteConnectOptions::from_str(&format!("sqlite://{}", path.display())) .map_err(|error| bat_core::Error::Other(error.into()))? - .create_if_missing(create_if_missing); + .create_if_missing(create_if_missing) + .journal_mode(SqliteJournalMode::Wal) + .busy_timeout(Duration::from_secs(30)); let pool = SqlitePoolOptions::new() .max_connections(1) .connect_with(options) @@ -510,7 +597,14 @@ impl SqliteTranslationTaskRepository { created_unix_seconds INTEGER NOT NULL, updated_unix_seconds INTEGER NOT NULL, completed_unix_seconds INTEGER, - provider_run_id TEXT + provider_run_id TEXT, + translation_results_json TEXT NOT NULL DEFAULT '[]', + provider TEXT, + lease_owner TEXT, + lease_expires_unix_seconds INTEGER, + failure_class TEXT, + failure_retryable INTEGER NOT NULL DEFAULT 0, + next_attempt_unix_seconds INTEGER ) "#, ) @@ -573,6 +667,37 @@ impl SqliteTranslationTaskRepository { ) .await?; ensure_column(&self.pool, "translation_tasks", "provider_run_id", "TEXT").await?; + ensure_column( + &self.pool, + "translation_tasks", + "translation_results_json", + "TEXT NOT NULL DEFAULT '[]'", + ) + .await?; + ensure_column(&self.pool, "translation_tasks", "provider", "TEXT").await?; + ensure_column(&self.pool, "translation_tasks", "lease_owner", "TEXT").await?; + ensure_column( + &self.pool, + "translation_tasks", + "lease_expires_unix_seconds", + "INTEGER", + ) + .await?; + ensure_column(&self.pool, "translation_tasks", "failure_class", "TEXT").await?; + ensure_column( + &self.pool, + "translation_tasks", + "failure_retryable", + "INTEGER NOT NULL DEFAULT 0", + ) + .await?; + ensure_column( + &self.pool, + "translation_tasks", + "next_attempt_unix_seconds", + "INTEGER", + ) + .await?; let current: Option = sqlx::query_scalar("SELECT version FROM schema_migrations WHERE component = ?1") @@ -615,7 +740,10 @@ impl SqliteTranslationTaskRepository { let existing: Option = sqlx::query_as( r#" SELECT task_json, worker_status, failure_reason, attempt_count, - created_unix_seconds, completed_unix_seconds, provider_run_id + created_unix_seconds, completed_unix_seconds, provider_run_id, + translation_results_json, provider, lease_owner, + lease_expires_unix_seconds, failure_class, failure_retryable, + next_attempt_unix_seconds FROM translation_tasks WHERE task_id = ?1 "#, @@ -642,30 +770,64 @@ impl SqliteTranslationTaskRepository { created, completed, provider_run_id, + translation_results_json, + provider, + lease_owner, + lease_expires, + failure_class, + failure_retryable, + next_attempt, )) = existing { let immutable_unchanged = previous_task_json == task_json; - let (status, failure_reason, attempt_count, created, completed, provider_run_id) = - if immutable_unchanged { - ( - worker_status, - failure_reason, - attempt_count, - created, - completed, - provider_run_id, - ) - } else { - report.refreshed_count += 1; - ( - initial_status.as_str().to_string(), - initial_failure_reason.clone(), - 0_i64, - now, - None, - None, - ) - }; + let ( + status, + failure_reason, + attempt_count, + created, + completed, + provider_run_id, + translation_results_json, + provider, + lease_owner, + lease_expires, + failure_class, + failure_retryable, + next_attempt, + ) = if immutable_unchanged { + ( + worker_status, + failure_reason, + attempt_count, + created, + completed, + provider_run_id, + translation_results_json, + provider, + lease_owner, + lease_expires, + failure_class, + failure_retryable, + next_attempt, + ) + } else { + report.refreshed_count += 1; + ( + initial_status.as_str().to_string(), + initial_failure_reason.clone(), + 0_i64, + now, + None, + None, + "[]".to_string(), + None, + None, + None, + None, + 0_i64, + None, + ) + }; if immutable_unchanged { report.preserved_state_count += 1; } @@ -677,7 +839,11 @@ impl SqliteTranslationTaskRepository { text_unit_formats_json = ?8, task_json = ?9, worker_status = ?10, failure_reason = ?11, attempt_count = ?12, created_unix_seconds = ?13, updated_unix_seconds = ?14, - completed_unix_seconds = ?15, provider_run_id = ?16 + completed_unix_seconds = ?15, provider_run_id = ?16, + translation_results_json = ?17, provider = ?18, + lease_owner = ?19, lease_expires_unix_seconds = ?20, + failure_class = ?21, failure_retryable = ?22, + next_attempt_unix_seconds = ?23 WHERE task_id = ?1 "#, ) @@ -697,6 +863,13 @@ impl SqliteTranslationTaskRepository { .bind(now) .bind(completed) .bind(provider_run_id) + .bind(translation_results_json) + .bind(provider) + .bind(lease_owner) + .bind(lease_expires) + .bind(failure_class) + .bind(failure_retryable) + .bind(next_attempt) .execute(&mut *transaction) .await .map_err(db_error)?; @@ -770,7 +943,9 @@ impl SqliteTranslationTaskRepository { r#" SELECT task_json, worker_status, failure_reason, attempt_count, created_unix_seconds, updated_unix_seconds, - completed_unix_seconds, provider_run_id + completed_unix_seconds, provider_run_id, translation_results_json, + provider, lease_owner, lease_expires_unix_seconds, failure_class, + failure_retryable, next_attempt_unix_seconds FROM translation_tasks ORDER BY task_id "#, @@ -794,6 +969,247 @@ impl SqliteTranslationTaskRepository { Ok(self.list(query).await?.len() as u64) } + /// Requeues tasks whose worker lease expired. + pub async fn recover_expired_leases(&self, now_unix_seconds: u64) -> Result { + let now = i64::try_from(now_unix_seconds).unwrap_or(i64::MAX); + let result = sqlx::query( + r#" + UPDATE translation_tasks + SET worker_status = CASE + WHEN attempt_count > 0 THEN 'failed' + ELSE 'queued' + END, + failure_reason = 'worker lease expired; task recovered', + failure_class = 'lease_expired', + failure_retryable = 1, + next_attempt_unix_seconds = ?1, + updated_unix_seconds = ?1, + lease_owner = NULL, + lease_expires_unix_seconds = NULL + WHERE worker_status = 'running' + AND lease_expires_unix_seconds IS NOT NULL + AND lease_expires_unix_seconds <= ?1 + "#, + ) + .bind(now) + .execute(&self.pool) + .await + .map_err(db_error)?; + Ok(result.rows_affected()) + } + + /// Claims one queued or retryable failed task and assigns an exclusive + /// lease to one worker. + pub async fn claim_next( + &self, + worker_id: &str, + provider: &str, + lease_seconds: u64, + max_attempts: u32, + ) -> Result> { + if worker_id.trim().is_empty() || provider.trim().is_empty() { + return Err(bat_core::Error::InvalidArgument( + "translation worker_id/provider 不能为空".to_string(), + )); + } + if max_attempts == 0 { + return Err(bat_core::Error::InvalidArgument( + "translation worker max_attempts 必须大于 0".to_string(), + )); + } + if lease_seconds == 0 { + return Err(bat_core::Error::InvalidArgument( + "translation worker lease_seconds 必须大于 0".to_string(), + )); + } + let now = unix_seconds_now_i64(); + let lease_expires = now.saturating_add(i64::try_from(lease_seconds).unwrap_or(i64::MAX)); + let mut transaction = self.pool.begin().await.map_err(db_error)?; + let row: Option = sqlx::query_as( + r#" + SELECT task_json, worker_status, failure_reason, attempt_count, + created_unix_seconds, updated_unix_seconds, + completed_unix_seconds, provider_run_id, translation_results_json, + provider, lease_owner, lease_expires_unix_seconds, failure_class, + failure_retryable, next_attempt_unix_seconds + FROM translation_tasks + WHERE queue_status = 'queued_offline' + AND ( + worker_status = 'queued' + OR ( + worker_status = 'failed' + AND failure_retryable = 1 + AND attempt_count < ?1 + AND (next_attempt_unix_seconds IS NULL + OR next_attempt_unix_seconds <= ?2) + ) + ) + AND (lease_expires_unix_seconds IS NULL OR lease_expires_unix_seconds <= ?2) + ORDER BY task_id + LIMIT 1 + "#, + ) + .bind(i64::from(max_attempts)) + .bind(now) + .fetch_optional(&mut *transaction) + .await + .map_err(db_error)?; + let Some(row) = row else { + transaction.commit().await.map_err(db_error)?; + return Ok(None); + }; + let current = PersistedTranslationTask::from_row(row)?; + let attempt_count = current.attempt_count.saturating_add(1); + let provider_run_id = format!( + "{provider}:{}:attempt-{attempt_count}", + current.task.task_id + ); + let result = sqlx::query( + r#" + UPDATE translation_tasks + SET worker_status = 'running', + failure_reason = NULL, + attempt_count = ?2, + updated_unix_seconds = ?3, + completed_unix_seconds = NULL, + provider_run_id = ?4, + provider = ?5, + lease_owner = ?6, + lease_expires_unix_seconds = ?7, + failure_class = NULL, + failure_retryable = 0, + next_attempt_unix_seconds = NULL + WHERE task_id = ?1 + AND ( + worker_status = 'queued' + OR ( + worker_status = 'failed' + AND failure_retryable = 1 + AND attempt_count < ?8 + AND (next_attempt_unix_seconds IS NULL + OR next_attempt_unix_seconds <= ?3) + ) + ) + AND queue_status = 'queued_offline' + AND (lease_expires_unix_seconds IS NULL OR lease_expires_unix_seconds <= ?3) + "#, + ) + .bind(¤t.task.task_id) + .bind(i64::from(attempt_count)) + .bind(now) + .bind(&provider_run_id) + .bind(provider) + .bind(worker_id) + .bind(lease_expires) + .bind(i64::from(max_attempts)) + .execute(&mut *transaction) + .await + .map_err(db_error)?; + if result.rows_affected() != 1 { + transaction.rollback().await.map_err(db_error)?; + return Ok(None); + } + transaction.commit().await.map_err(db_error)?; + self.find(¤t.task.task_id).await.map(Some) + } + + /// Stores an idempotent provider result while the worker still owns its lease. + pub async fn complete_claim( + &self, + task_id: &str, + worker_id: &str, + provider_run_id: &str, + provider: &str, + translation_results: &[TranslationTaskUnitResult], + ) -> Result { + let results_json = serde_json::to_string(translation_results) + .map_err(|error| bat_core::Error::Serialization(error.to_string()))?; + let now = unix_seconds_now_i64(); + let result = sqlx::query( + r#" + UPDATE translation_tasks + SET worker_status = 'completed', + failure_reason = NULL, + updated_unix_seconds = ?2, + completed_unix_seconds = ?2, + provider_run_id = ?3, + provider = ?4, + translation_results_json = ?5, + lease_owner = NULL, + lease_expires_unix_seconds = NULL, + failure_class = NULL, + failure_retryable = 0, + next_attempt_unix_seconds = NULL + WHERE task_id = ?1 + AND worker_status = 'running' + AND lease_owner = ?6 + AND provider_run_id = ?3 + "#, + ) + .bind(task_id) + .bind(now) + .bind(provider_run_id) + .bind(provider) + .bind(results_json) + .bind(worker_id) + .execute(&self.pool) + .await + .map_err(db_error)?; + if result.rows_affected() != 1 { + return Err(bat_core::Error::InvalidArgument(format!( + "翻译任务 {} 的 lease 已失效，拒绝写入 provider 结果", + task_id + ))); + } + self.find(task_id).await + } + + /// Records one provider failure and releases the worker lease. + pub async fn fail_claim( + &self, + failure: TranslationTaskFailure, + ) -> Result { + let now = unix_seconds_now_i64(); + let next_attempt = failure + .next_attempt_unix_seconds + .map(|value| i64::try_from(value).unwrap_or(i64::MAX)); + let result = sqlx::query( + r#" + UPDATE translation_tasks + SET worker_status = 'failed', + failure_reason = ?2, + updated_unix_seconds = ?3, + lease_owner = NULL, + lease_expires_unix_seconds = NULL, + failure_class = ?4, + failure_retryable = ?5, + next_attempt_unix_seconds = ?6 + WHERE task_id = ?1 + AND worker_status = 'running' + AND lease_owner = ?7 + AND provider_run_id = ?8 + "#, + ) + .bind(&failure.task_id) + .bind(&failure.failure_reason) + .bind(now) + .bind(&failure.failure_class) + .bind(if failure.retryable { 1_i64 } else { 0_i64 }) + .bind(next_attempt) + .bind(&failure.worker_id) + .bind(&failure.provider_run_id) + .execute(&self.pool) + .await + .map_err(db_error)?; + if result.rows_affected() != 1 { + return Err(bat_core::Error::InvalidArgument(format!( + "翻译任务 {} 的 lease 已失效，拒绝写入 provider 失败状态", + failure.task_id + ))); + } + self.find(&failure.task_id).await + } + /// Updates provider state and returns the durable task record. pub async fn update_status( &self, @@ -818,7 +1234,10 @@ impl SqliteTranslationTaskRepository { UPDATE translation_tasks SET worker_status = ?2, failure_reason = ?3, attempt_count = ?4, updated_unix_seconds = ?5, completed_unix_seconds = ?6, - provider_run_id = COALESCE(?7, provider_run_id) + provider_run_id = COALESCE(?7, provider_run_id), + lease_owner = NULL, lease_expires_unix_seconds = NULL, + failure_class = NULL, failure_retryable = 0, + next_attempt_unix_seconds = NULL WHERE task_id = ?1 "#, ) @@ -841,7 +1260,9 @@ impl SqliteTranslationTaskRepository { r#" SELECT task_json, worker_status, failure_reason, attempt_count, created_unix_seconds, updated_unix_seconds, - completed_unix_seconds, provider_run_id + completed_unix_seconds, provider_run_id, translation_results_json, + provider, lease_owner, lease_expires_unix_seconds, failure_class, + failure_retryable, next_attempt_unix_seconds FROM translation_tasks WHERE task_id = ?1 "#, @@ -864,6 +1285,13 @@ type ExistingTaskRow = ( i64, Option, Option, + String, + Option, + Option, + Option, + Option, + i64, + Option, ); type TranslationTaskRow = ( @@ -875,6 +1303,13 @@ type TranslationTaskRow = ( i64, Option, Option, + String, + Option, + Option, + Option, + Option, + i64, + Option, ); impl PersistedTranslationTask { @@ -889,6 +1324,13 @@ impl PersistedTranslationTask { updated_unix_seconds: state.updated_unix_seconds, completed_unix_seconds: state.completed_unix_seconds, provider_run_id: state.provider_run_id, + translation_results: state.translation_results, + provider: state.provider, + lease_owner: state.lease_owner, + lease_expires_unix_seconds: state.lease_expires_unix_seconds, + failure_class: state.failure_class, + failure_retryable: state.failure_retryable, + next_attempt_unix_seconds: state.next_attempt_unix_seconds, } } @@ -910,6 +1352,13 @@ impl PersistedTranslationTask { updated_unix_seconds: generated_unix_seconds, completed_unix_seconds: None, provider_run_id: None, + translation_results: Vec::new(), + provider: None, + lease_owner: None, + lease_expires_unix_seconds: None, + failure_class: None, + failure_retryable: false, + next_attempt_unix_seconds: None, }, ) } @@ -924,12 +1373,21 @@ impl PersistedTranslationTask { updated_unix_seconds, completed_unix_seconds, provider_run_id, + translation_results_json, + provider, + lease_owner, + lease_expires_unix_seconds, + failure_class, + failure_retryable, + next_attempt_unix_seconds, ) = row; let task = serde_json::from_str(&task_json) .map_err(|error| bat_core::Error::Serialization(error.to_string()))?; let task_status = TranslationTaskStatus::parse(&worker_status).ok_or_else(|| { bat_core::Error::Serialization(format!("未知翻译任务 worker 状态：{worker_status}")) })?; + let translation_results = serde_json::from_str(&translation_results_json) + .map_err(|error| bat_core::Error::Serialization(error.to_string()))?; Ok(Self { task, task_status, @@ -951,6 +1409,25 @@ impl PersistedTranslationTask { }) .transpose()?, provider_run_id, + translation_results, + provider, + lease_owner, + lease_expires_unix_seconds: lease_expires_unix_seconds + .map(|value| { + u64::try_from(value).map_err(|_| { + bat_core::Error::Serialization("翻译任务 lease 时间无效".to_string()) + }) + }) + .transpose()?, + failure_class, + failure_retryable: failure_retryable != 0, + next_attempt_unix_seconds: next_attempt_unix_seconds + .map(|value| { + u64::try_from(value).map_err(|_| { + bat_core::Error::Serialization("翻译任务 next_attempt 时间无效".to_string()) + }) + }) + .transpose()?, }) } } @@ -1219,6 +1696,140 @@ mod tests { assert_eq!(retrievable[0].attempt_count, 1); } + #[tokio::test] + async fn sqlite_translation_tasks_recover_expired_leases_for_retry() { + let temp = tempfile::TempDir::new().unwrap(); + let repository = + SqliteTranslationTaskRepository::new(temp.path().join("translation-tasks.sqlite")) + .await + .unwrap(); + let queue = queue(vec![task( + "task-a", + "Bundles/a.bundle", + OfficialTextUnitTaskStatus::QueuedOffline, + Some(OfficialParseStatus::Parsed), + None, + )]); + repository.sync_queue(&queue).await.unwrap(); + + let claimed = repository + .claim_next("worker-a", "mock", 1, 3) + .await + .unwrap() + .unwrap(); + assert_eq!(claimed.task_status, TranslationTaskStatus::Running); + assert_eq!(claimed.lease_owner.as_deref(), Some("worker-a")); + force_expire_lease(&repository, "task-a").await; + + let recovered = repository + .recover_expired_leases(unix_seconds_now()) + .await + .unwrap(); + assert_eq!(recovered, 1); + let recovered_task = repository.find("task-a").await.unwrap(); + assert_eq!(recovered_task.task_status, TranslationTaskStatus::Failed); + assert_eq!( + recovered_task.failure_reason.as_deref(), + Some("worker lease expired; task recovered") + ); + assert_eq!( + recovered_task.failure_class.as_deref(), + Some("lease_expired") + ); + assert!(recovered_task.failure_retryable); + assert_eq!(recovered_task.lease_owner, None); + + let reclaimed = repository + .claim_next("worker-b", "mock", 30, 3) + .await + .unwrap() + .unwrap(); + assert_eq!(reclaimed.task_status, TranslationTaskStatus::Running); + assert_eq!(reclaimed.attempt_count, 2); + assert_eq!(reclaimed.lease_owner.as_deref(), Some("worker-b")); + } + + #[tokio::test] + async fn sqlite_translation_tasks_reject_stale_worker_writes() { + let temp = tempfile::TempDir::new().unwrap(); + let repository = + SqliteTranslationTaskRepository::new(temp.path().join("translation-tasks.sqlite")) + .await + .unwrap(); + let queue = queue(vec![task( + "task-a", + "Bundles/a.bundle", + OfficialTextUnitTaskStatus::QueuedOffline, + Some(OfficialParseStatus::Parsed), + None, + )]); + repository.sync_queue(&queue).await.unwrap(); + + let first_claim = repository + .claim_next("worker-a", "mock", 1, 3) + .await + .unwrap() + .unwrap(); + let first_run = first_claim.provider_run_id.clone().unwrap(); + force_expire_lease(&repository, "task-a").await; + repository + .recover_expired_leases(unix_seconds_now()) + .await + .unwrap(); + let second_claim = repository + .claim_next("worker-b", "mock", 30, 3) + .await + .unwrap() + .unwrap(); + let second_run = second_claim.provider_run_id.clone().unwrap(); + let result = TranslationTaskUnitResult { + unit_id: "unit-a".to_string(), + source_text: "source".to_string(), + translated_text: "translated".to_string(), + provider: "mock".to_string(), + provider_run_id: second_run.clone(), + translated_unix_seconds: 1, + }; + + assert!(repository + .complete_claim( + "task-a", + "worker-a", + &first_run, + "mock", + std::slice::from_ref(&result) + ) + .await + .is_err()); + let completed = repository + .complete_claim( + "task-a", + "worker-b", + &second_run, + "mock", + std::slice::from_ref(&result), + ) + .await + .unwrap(); + assert_eq!(completed.task_status, TranslationTaskStatus::Completed); + assert_eq!(completed.translation_results, vec![result.clone()]); + assert!(repository + .fail_claim(TranslationTaskFailure { + task_id: "task-a".to_string(), + worker_id: "worker-a".to_string(), + provider_run_id: first_run, + failure_class: "network".to_string(), + failure_reason: "late failure".to_string(), + retryable: true, + next_attempt_unix_seconds: Some(2), + }) + .await + .is_err()); + let task = repository.find("task-a").await.unwrap(); + assert_eq!(task.task_status, TranslationTaskStatus::Completed); + assert_eq!(task.translation_results, vec![result]); + } + #[test] fn translation_handoff_maps_worker_state_and_preserves_provider_progress() { let queue = queue(vec![ @@ -1254,6 +1865,13 @@ mod tests { updated_unix_seconds: 124, completed_unix_seconds: None, provider_run_id: Some("provider-run-1".to_string()), + translation_results: Vec::new(), + provider: Some("fixture".to_string()), + lease_owner: None, + lease_expires_unix_seconds: None, + failure_class: None, + failure_retryable: false, + next_attempt_unix_seconds: None, }, PersistedTranslationTask { task: queue.tasks[1].clone(), @@ -1264,6 +1882,13 @@ mod tests { updated_unix_seconds: 125, completed_unix_seconds: Some(125), provider_run_id: Some("provider-run-1".to_string()), + translation_results: Vec::new(), + provider: Some("fixture".to_string()), + lease_owner: None, + lease_expires_unix_seconds: None, + failure_class: None, + failure_retryable: false, + next_attempt_unix_seconds: None, }, ]; @@ -1286,6 +1911,20 @@ mod tests { assert_eq!(handoff.provider_runs[0].attempt_count, 2); } + async fn force_expire_lease(repository: &SqliteTranslationTaskRepository, task_id: &str) { + sqlx::query( + r#" + UPDATE translation_tasks + SET lease_expires_unix_seconds = 0 + WHERE task_id = ?1 + "#, + ) + .bind(task_id) + .execute(&repository.pool) + .await + .unwrap(); + } + #[test] fn translation_handoff_file_round_trips_with_version_check() { let temp = tempfile::TempDir::new().unwrap(); diff --git a/infrastructure/src/translation_worker.rs b/infrastructure/src/translation_worker.rs new file mode 100644 index 0000000..4aad1e7 --- /dev/null +++ b/infrastructure/src/translation_worker.rs @@ -0,0 +1,1374 @@ +//! 可恢复的 TextUnit provider worker。 +//! +//! worker 只消费已发布 release 中的 TextUnit 索引和 SQLite 任务状态，不 +//! 修改官方资源。provider 的输入、输出和错误分类是稳定 contract；状态、 +//! 租约和译文结果始终写入 `translation-tasks.sqlite`。 + +use crate::official_parse::{read_textunit_index_at, OfficialTextUnitIndexUnit}; +use crate::official_textunit_queue::read_textunit_task_queue_at; +use crate::translation_tasks::{ + PersistedTranslationTask, SqliteTranslationTaskRepository, TranslationTaskFailure, + TranslationTaskUnitResult, +}; +use async_trait::async_trait; +use serde::{Deserialize, Serialize}; +use std::collections::{BTreeMap, BTreeSet}; +use std::env; +use std::path::{Path, PathBuf}; +use std::process::Stdio; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, SystemTime, UNIX_EPOCH}; +use tokio::io::AsyncWriteExt; +use tokio::process::Command; + +/// Provider worker 的并发下限。 +pub const MIN_TRANSLATION_CONCURRENCY: usize = 1; +/// Provider worker 的默认并发。 +pub const DEFAULT_TRANSLATION_CONCURRENCY: usize = 8; +/// Provider worker 的并发上限。 +pub const MAX_TRANSLATION_CONCURRENCY: usize = 256; +/// Provider worker 默认最大尝试次数。 +pub const DEFAULT_TRANSLATION_MAX_ATTEMPTS: u32 = 3; +/// Provider worker 默认租约时长。 +pub const DEFAULT_TRANSLATION_LEASE_SECONDS: u64 = 300; +/// Provider worker 默认重试退避。 +pub const DEFAULT_TRANSLATION_RETRY_BACKOFF: Duration = Duration::from_secs(5); +/// 本地 mock fixture schema 版本。 +pub const MOCK_TRANSLATION_FIXTURE_VERSION: u32 = 1; + +/// Provider 类型。 +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TranslationProviderKind { + /// 本地、无网络 fixture provider。 + Mock, + /// Crowdin API v2 provider。 + Crowdin, +} + +impl TranslationProviderKind { + /// 返回稳定 provider 标签。 + pub fn as_str(self) -> &'static str { + match self { + Self::Mock => "mock", + Self::Crowdin => "crowdin", + } + } + + /// 解析 CLI/RPC provider 标签。 + pub fn parse(value: &str) -> Option { + match value { + "mock" | "fixture" => Some(Self::Mock), + "crowdin" => Some(Self::Crowdin), + _ => None, + } + } +} + +/// Provider worker 运行配置。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct TranslationWorkerConfig { + /// Provider 类型。 + pub provider: TranslationProviderKind, + /// mock provider 使用的 JSON fixture。 + pub fixture_path: Option, + /// 并发 worker 数量。 + pub concurrency: usize, + /// 单个任务允许的最大 claim 次数。 + pub max_attempts: u32, + /// claim 租约时长。 + pub lease_seconds: u64, + /// 可重试 provider 失败的退避时长。 + #[serde( + rename = "retry_backoff_seconds", + serialize_with = "serialize_duration_seconds" + )] + pub retry_backoff: Duration, + /// 本轮最多 claim 的任务数；省略表示处理全部可用任务。 + pub max_tasks: Option, + /// worker 实例前缀，用于 lease 诊断。 + pub worker_id: String, +} + +impl Default for TranslationWorkerConfig { + fn default() -> Self { + Self { + provider: TranslationProviderKind::Mock, + fixture_path: None, + concurrency: DEFAULT_TRANSLATION_CONCURRENCY, + max_attempts: DEFAULT_TRANSLATION_MAX_ATTEMPTS, + lease_seconds: DEFAULT_TRANSLATION_LEASE_SECONDS, + retry_backoff: DEFAULT_TRANSLATION_RETRY_BACKOFF, + max_tasks: None, + worker_id: format!("bat-worker-{}", std::process::id()), + } + } +} + +impl TranslationWorkerConfig { + /// 校验 worker 配置。 + pub fn validate(&self) -> anyhow::Result<()> { + if !(MIN_TRANSLATION_CONCURRENCY..=MAX_TRANSLATION_CONCURRENCY).contains(&self.concurrency) + { + return Err(anyhow::anyhow!( + "翻译 worker 并发必须在 {MIN_TRANSLATION_CONCURRENCY}..={MAX_TRANSLATION_CONCURRENCY} 范围内" + )); + } + if self.max_attempts == 0 { + return Err(anyhow::anyhow!("翻译 worker 最大尝试次数必须大于 0")); + } + if self.lease_seconds == 0 { + return Err(anyhow::anyhow!("翻译 worker lease 时长必须大于 0")); + } + if self.max_tasks.is_some_and(|count| count == 0) { + return Err(anyhow::anyhow!("翻译 worker 最大任务数必须大于 0")); + } + if self.worker_id.trim().is_empty() { + return Err(anyhow::anyhow!("翻译 worker ID 不能为空")); + } + Ok(()) + } +} + +fn serialize_duration_seconds(duration: &Duration, serializer: S) -> Result +where + S: serde::Serializer, +{ + serializer.serialize_u64(duration.as_secs()) +} + +/// Provider 单个 TextUnit 输入。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TranslationProviderUnit { + /// 稳定 TextUnit ID。 + pub unit_id: String, + /// 所属资源任务 ID。 + pub task_id: String, + /// 官方 release ID。 + pub official_release_id: String, + /// 官方资源 destination。 + pub destination: String, + /// ZIP 内 entry。 + pub archive_entry: Option, + /// 原文。 + pub source_text: String, + /// Unity serialized file。 + pub serialized_file: Option, + /// Unity object path ID。 + pub path_id: Option, + /// Unity class ID。 + pub class_id: Option, + /// TypeTree 字段路径。 + pub field_path: Option, + /// TextUnit format。 + pub format: Option, + /// TextUnit 来源类型。 + pub text_source_kind: Option, + /// TextAsset 名称。 + pub asset_name: Option, + /// 解析器保留的上下文，包括可选 `crowdin_string_id`。 + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub context: BTreeMap, +} + +/// 一次 provider 批处理请求。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TranslationProviderRequest { + /// provider run ID。 + pub provider_run_id: String, + /// 官方 release ID。 + pub official_release_id: String, + /// 资源任务 ID。 + pub task_id: String, + /// 资源 destination。 + pub destination: String, + /// ZIP 内 entry。 + pub archive_entry: Option, + /// 本次提交的 TextUnit。 + pub units: Vec, +} + +/// provider 返回的单个译文。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TranslationProviderUnitResult { + /// TextUnit ID。 + pub unit_id: String, + /// provider 看到的原文，用于防止 stale result。 + pub source_text: String, + /// provider 产生的译文。 + pub translated_text: String, +} + +/// 一次 provider 批处理响应。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct TranslationProviderResponse { + /// 对应的 provider run ID。 + pub provider_run_id: String, + /// provider 结果。 + pub units: Vec, +} + +/// provider 失败分类。 +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TranslationProviderFailureClass { + /// DNS、连接、超时等网络错误。 + Network, + /// provider 限流。 + RateLimited, + /// provider 服务端错误。 + Server, + /// 认证失败。 + Authentication, + /// provider 找不到对应资源。 + NotFound, + /// 请求或 TextUnit 映射无效。 + InvalidRequest, + /// 其他不可重试错误。 + Unknown, +} + +impl TranslationProviderFailureClass { + /// 返回稳定失败分类标签。 + pub fn as_str(self) -> &'static str { + match self { + Self::Network => "network", + Self::RateLimited => "rate_limited", + Self::Server => "server", + Self::Authentication => "authentication", + Self::NotFound => "not_found", + Self::InvalidRequest => "invalid_request", + Self::Unknown => "unknown", + } + } + + fn default_retryable(self) -> bool { + matches!(self, Self::Network | Self::RateLimited | Self::Server) + } +} + +/// provider 调用失败，包含可诊断分类但不包含凭据。 +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TranslationProviderError { + /// 失败分类。 + pub class: TranslationProviderFailureClass, + /// 脱敏后的诊断信息。 + pub message: String, + /// 是否允许 worker 重试。 + pub retryable: bool, +} + +impl TranslationProviderError { + fn new(class: TranslationProviderFailureClass, message: impl Into) -> Self { + Self { + retryable: class.default_retryable(), + class, + message: message.into(), + } + } +} + +/// Provider 扩展接口。 +#[async_trait] +pub trait TranslationProvider: Send + Sync { + /// 返回稳定 provider 名称。 + fn name(&self) -> &'static str; + + /// 翻译一个资源任务中的所有 TextUnit。 + async fn translate( + &self, + request: TranslationProviderRequest, + ) -> Result; +} + +#[derive(Debug, Deserialize)] +struct MockTranslationFixture { + #[serde(default = "default_mock_fixture_version")] + schema_version: u32, + #[serde(default)] + translations: BTreeMap, + #[serde(default)] + failures: BTreeMap, +} + +#[derive(Debug, Clone, Deserialize)] +struct MockFailureFixture { + class: TranslationProviderFailureClass, + message: String, + #[serde(default)] + retryable: Option, +} + +/// 本地 fixture provider。 +#[derive(Debug, Clone)] +pub struct MockTranslationProvider { + translations: BTreeMap, + failures: BTreeMap, +} + +impl MockTranslationProvider { + /// 从 fixture 文件创建 mock provider。 + pub fn from_file(path: Option<&Path>) -> anyhow::Result { + let fixture = if let Some(path) = path { + let bytes = std::fs::read(path) + .map_err(|error| anyhow::anyhow!("读取 mock translation fixture 失败：{error}"))?; + serde_json::from_slice::(&bytes).map_err(|error| { + anyhow::anyhow!( + "解析 mock translation fixture 失败 {}：{error}", + path.display() + ) + })? + } else { + MockTranslationFixture { + schema_version: MOCK_TRANSLATION_FIXTURE_VERSION, + translations: BTreeMap::new(), + failures: BTreeMap::new(), + } + }; + if fixture.schema_version != MOCK_TRANSLATION_FIXTURE_VERSION { + return Err(anyhow::anyhow!( + "不支持的 mock translation fixture schema：{}，当前版本={}", + fixture.schema_version, + MOCK_TRANSLATION_FIXTURE_VERSION + )); + } + Ok(Self { + translations: fixture.translations, + failures: fixture.failures, + }) + } +} + +#[async_trait] +impl TranslationProvider for MockTranslationProvider { + fn name(&self) -> &'static str { + "mock" + } + + async fn translate( + &self, + request: TranslationProviderRequest, + ) -> Result { + let mut units = Vec::with_capacity(request.units.len()); + for unit in &request.units { + let key = unit.unit_id.as_str(); + if let Some(failure) = self + .failures + .get(key) + .or_else(|| self.failures.get(&request.task_id)) + { + return Err(TranslationProviderError { + class: failure.class, + message: failure.message.clone(), + retryable: failure + .retryable + .unwrap_or_else(|| failure.class.default_retryable()), + }); + } + let translated_text = self + .translations + .get(key) + .or_else(|| self.translations.get(&request.task_id)) + .cloned() + .unwrap_or_else(|| format!("【mock】{}", unit.source_text)); + units.push(TranslationProviderUnitResult { + unit_id: unit.unit_id.clone(), + source_text: unit.source_text.clone(), + translated_text, + }); + } + Ok(TranslationProviderResponse { + provider_run_id: request.provider_run_id, + units, + }) + } +} + +/// Crowdin provider 的脱敏配置。 +pub struct CrowdinProvider { + project_id: String, + language_id: String, + token: String, + api_base_url: String, + curl_command: PathBuf, +} + +impl std::fmt::Debug for CrowdinProvider { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter + .debug_struct("CrowdinProvider") + .field("project_id", &self.project_id) + .field("language_id", &self.language_id) + .field("api_base_url", &self.api_base_url) + .field("curl_command", &self.curl_command) + .finish_non_exhaustive() + } +} + +impl CrowdinProvider { + /// 从环境变量读取 Crowdin 凭据和项目配置。 + /// + /// 支持 `CROWDIN_API_TOKEN`、`CROWDIN_PROJECT_ID`、`CROWDIN_LANGUAGE_ID`， + /// 可选 `CROWDIN_API_BASE_URL` 和 `BAT_CURL`。token 永不进入 report、 + /// fixture 或日志。 + pub fn from_env() -> anyhow::Result { + let required = |name: &str| { + env::var(name) + .ok() + .filter(|value| !value.trim().is_empty()) + .ok_or_else(|| anyhow::anyhow!("缺少 Crowdin 环境变量 {name}")) + }; + let project_id = required("CROWDIN_PROJECT_ID")?; + let language_id = required("CROWDIN_LANGUAGE_ID")?; + if !project_id + .chars() + .all(|character| character.is_ascii_digit()) + { + return Err(anyhow::anyhow!("CROWDIN_PROJECT_ID 必须是数字")); + } + if !language_id + .chars() + .all(|character| character.is_ascii_alphanumeric() || matches!(character, '-' | '_')) + { + return Err(anyhow::anyhow!( + "CROWDIN_LANGUAGE_ID 只允许 ASCII 字母、数字、- 和 _" + )); + } + Ok(Self { + project_id, + language_id, + token: required("CROWDIN_API_TOKEN")?, + api_base_url: env::var("CROWDIN_API_BASE_URL") + .ok() + .filter(|value| !value.trim().is_empty()) + .unwrap_or_else(|| "https://api.crowdin.com/api/v2".to_string()), + curl_command: env::var_os("BAT_CURL") + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from("curl")), + }) + } + + async fn fetch_translation(&self, string_id: &str) -> Result { + if !string_id + .chars() + .all(|character| character.is_ascii_digit()) + { + return Err(TranslationProviderError::new( + TranslationProviderFailureClass::InvalidRequest, + "TextUnit 缺少有效的 numeric crowdin_string_id", + )); + } + let url = format!( + "{}/projects/{}/languages/{}/translations?stringIds[]={string_id}", + self.api_base_url.trim_end_matches('/'), + self.project_id, + self.language_id + ); + let config = format!( + "url = \"{}\"\nrequest = \"GET\"\nheader = \"Authorization: Bearer {}\"\nheader = \"Accept: application/json\"\n", + curl_config_escape(&url), + curl_config_escape(&self.token) + ); + let mut child = Command::new(&self.curl_command) + .arg("--silent") + .arg("--show-error") + .arg("--fail-with-body") + .arg("--max-time") + .arg("60") + .arg("--config") + .arg("-") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .map_err(|error| { + TranslationProviderError::new( + TranslationProviderFailureClass::Network, + format!("启动 Crowdin curl 失败：{error}"), + ) + })?; + if let Some(mut stdin) = child.stdin.take() { + stdin.write_all(config.as_bytes()).await.map_err(|error| { + TranslationProviderError::new( + TranslationProviderFailureClass::Network, + format!("写入 Crowdin curl 请求失败：{error}"), + ) + })?; + } + let output = child.wait_with_output().await.map_err(|error| { + TranslationProviderError::new( + TranslationProviderFailureClass::Network, + format!("等待 Crowdin curl 失败：{error}"), + ) + })?; + if !output.status.success() { + return Err(crowdin_curl_error(&output.stderr)); + } + let body: serde_json::Value = serde_json::from_slice(&output.stdout).map_err(|error| { + TranslationProviderError::new( + TranslationProviderFailureClass::Unknown, + format!("Crowdin 返回不是 JSON：{error}"), + ) + })?; + find_translation_text(&body).ok_or_else(|| { + TranslationProviderError::new( + TranslationProviderFailureClass::NotFound, + format!("Crowdin 没有返回 string_id={string_id} 的译文"), + ) + }) + } +} + +#[async_trait] +impl TranslationProvider for CrowdinProvider { + fn name(&self) -> &'static str { + "crowdin" + } + + async fn translate( + &self, + request: TranslationProviderRequest, + ) -> Result { + let mut units = Vec::with_capacity(request.units.len()); + for unit in &request.units { + let string_id = unit.context.get("crowdin_string_id").ok_or_else(|| { + TranslationProviderError::new( + TranslationProviderFailureClass::InvalidRequest, + format!("TextUnit {} 缺少 context.crowdin_string_id", unit.unit_id), + ) + })?; + units.push(TranslationProviderUnitResult { + unit_id: unit.unit_id.clone(), + source_text: unit.source_text.clone(), + translated_text: self.fetch_translation(string_id).await?, + }); + } + Ok(TranslationProviderResponse { + provider_run_id: request.provider_run_id, + units, + }) + } +} + +/// worker 单轮运行报告。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct TranslationWorkerReport { + /// 稳定命令名。 + pub command: &'static str, + /// 运行状态。 + pub status: &'static str, + /// 官方 release ID。 + pub official_release_id: String, + /// provider 名称。 + pub provider: String, + /// 本轮回收的过期租约数。 + pub recovered_lease_count: u64, + /// 本轮 claim 数。 + pub claimed_count: usize, + /// 成功完成的任务数。 + pub completed_count: usize, + /// 失败任务数，包含仍可重试的失败。 + pub failed_count: usize, + /// 已安排重试的失败数。 + pub retry_scheduled_count: usize, + /// 仍未消费的任务数。 + pub remaining_count: usize, + /// 失败诊断。 + pub failures: Vec, +} + +/// worker 失败诊断。 +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct TranslationWorkerFailure { + /// 任务 ID。 + pub task_id: String, + /// provider run ID。 + pub provider_run_id: String, + /// 失败分类。 + pub failure_class: String, + /// 脱敏失败原因。 + pub failure_reason: String, + /// 是否安排后续重试。 + pub retryable: bool, +} + +#[derive(Debug, Default)] +struct WorkerStats { + claimed_count: AtomicUsize, + completed_count: AtomicUsize, + failed_count: AtomicUsize, + retry_scheduled_count: AtomicUsize, + failures: Mutex>, +} + +struct WorkerTaskContext<'a> { + repository: &'a SqliteTranslationTaskRepository, + index: &'a crate::official_parse::OfficialTextUnitIndex, + provider: &'a dyn TranslationProvider, + provider_name: &'a str, + worker_id: &'a str, + max_attempts: u32, + retry_backoff: Duration, + stats: &'a WorkerStats, +} + +/// 运行一个 provider worker 轮次。 +pub async fn run_translation_worker_at( + resource_root: &Path, + config: &TranslationWorkerConfig, +) -> anyhow::Result { + config.validate()?; + let provider: Arc = match config.provider { + TranslationProviderKind::Mock => Arc::new(MockTranslationProvider::from_file( + config.fixture_path.as_deref(), + )?), + TranslationProviderKind::Crowdin => Arc::new(CrowdinProvider::from_env()?), + }; + run_translation_worker_with_provider(resource_root, config, provider).await +} + +/// 使用指定 provider 运行 worker，供测试和插件宿主使用。 +pub async fn run_translation_worker_with_provider( + resource_root: &Path, + config: &TranslationWorkerConfig, + provider: Arc, +) -> anyhow::Result { + config.validate()?; + let queue = read_textunit_task_queue_at(resource_root) + .map_err(anyhow::Error::msg)? + .ok_or_else(|| anyhow::anyhow!("缺少官方 TextUnit 任务队列"))?; + let index = Arc::new( + read_textunit_index_at(resource_root) + .map_err(anyhow::Error::msg)? + .ok_or_else(|| anyhow::anyhow!("缺少官方 TextUnit 明细索引"))?, + ); + let repository = Arc::new( + SqliteTranslationTaskRepository::new(SqliteTranslationTaskRepository::repository_path( + resource_root, + )) + .await + .map_err(|error| anyhow::anyhow!("打开翻译任务状态库失败：{error}"))?, + ); + repository + .sync_queue(&queue) + .await + .map_err(|error| anyhow::anyhow!("同步翻译任务队列失败：{error}"))?; + let recovered_lease_count = repository + .recover_expired_leases(unix_seconds_now()) + .await + .map_err(|error| anyhow::anyhow!("回收翻译 worker lease 失败：{error}"))?; + let stats = Arc::new(WorkerStats::default()); + let claimed_limit = Arc::new(AtomicUsize::new(0)); + let mut handles = Vec::with_capacity(config.concurrency); + + for worker_index in 0..config.concurrency { + let repository = Arc::clone(&repository); + let index = Arc::clone(&index); + let provider = Arc::clone(&provider); + let stats = Arc::clone(&stats); + let claimed_limit = Arc::clone(&claimed_limit); + let worker_id = format!("{}-{worker_index}", config.worker_id); + let provider_name = provider.name().to_string(); + let max_tasks = config.max_tasks; + let max_attempts = config.max_attempts; + let lease_seconds = config.lease_seconds; + let retry_backoff = config.retry_backoff; + handles.push(tokio::spawn(async move { + loop { + if let Some(max_tasks) = max_tasks { + let reservation = claimed_limit.fetch_add(1, Ordering::AcqRel); + if reservation >= max_tasks { + claimed_limit.fetch_sub(1, Ordering::AcqRel); + break; + } + } + let claimed = repository + .claim_next(&worker_id, &provider_name, lease_seconds, max_attempts) + .await + .map_err(|error| anyhow::anyhow!("claim 翻译任务失败：{error}"))?; + let Some(task) = claimed else { + if max_tasks.is_some() { + claimed_limit.fetch_sub(1, Ordering::AcqRel); + } + break; + }; + stats.claimed_count.fetch_add(1, Ordering::Relaxed); + process_claimed_task( + &WorkerTaskContext { + repository: &repository, + index: &index, + provider: provider.as_ref(), + provider_name: &provider_name, + worker_id: &worker_id, + max_attempts, + retry_backoff, + stats: &stats, + }, + &task, + ) + .await?; + } + Ok::<(), anyhow::Error>(()) + })); + } + for handle in handles { + handle + .await + .map_err(|error| anyhow::anyhow!("等待翻译 worker 失败：{error}"))??; + } + + let remaining_count = repository + .list(&Default::default()) + .await + .map_err(|error| anyhow::anyhow!("读取翻译 worker 剩余任务失败：{error}"))? + .into_iter() + .filter(|task| { + task.task.status + == crate::official_textunit_queue::OfficialTextUnitTaskStatus::QueuedOffline + && task.task_status != crate::translation_tasks::TranslationTaskStatus::Completed + && task.task_status != crate::translation_tasks::TranslationTaskStatus::Skipped + }) + .count(); + let failures = stats + .failures + .lock() + .map_err(|_| anyhow::anyhow!("读取翻译 worker 失败列表时 mutex poisoned"))? + .clone(); + let failed_count = stats.failed_count.load(Ordering::Relaxed); + Ok(TranslationWorkerReport { + command: "translation-worker", + status: if failed_count == 0 { + "completed" + } else { + "completed_with_failures" + }, + official_release_id: queue.official_release_id, + provider: provider.name().to_string(), + recovered_lease_count, + claimed_count: stats.claimed_count.load(Ordering::Relaxed), + completed_count: stats.completed_count.load(Ordering::Relaxed), + failed_count, + retry_scheduled_count: stats.retry_scheduled_count.load(Ordering::Relaxed), + remaining_count, + failures, + }) +} + +async fn process_claimed_task( + context: &WorkerTaskContext<'_>, + task: &PersistedTranslationTask, +) -> anyhow::Result<()> { + let request = match provider_request(task, context.index) { + Ok(request) => request, + Err(error) => { + record_provider_failure( + context, + task, + TranslationProviderError::new( + TranslationProviderFailureClass::InvalidRequest, + error.to_string(), + ), + ) + .await?; + return Ok(()); + } + }; + match context.provider.translate(request.clone()).await { + Ok(response) => { + let results = + match validate_provider_response(&request, response, context.provider_name) { + Ok(results) => results, + Err(error) => { + record_provider_failure( + context, + task, + TranslationProviderError::new( + TranslationProviderFailureClass::InvalidRequest, + error.to_string(), + ), + ) + .await?; + return Ok(()); + } + }; + context + .repository + .complete_claim( + &task.task.task_id, + context.worker_id, + &task.provider_run_id.clone().unwrap_or_default(), + context.provider_name, + &results, + ) + .await + .map_err(|error| anyhow::anyhow!("写入翻译任务完成结果失败：{error}"))?; + context + .stats + .completed_count + .fetch_add(1, Ordering::Relaxed); + } + Err(error) => { + record_provider_failure(context, task, error).await?; + } + } + Ok(()) +} + +fn provider_request( + task: &PersistedTranslationTask, + index: &crate::official_parse::OfficialTextUnitIndex, +) -> anyhow::Result { + let parse_entry_key = task + .task + .parse_entry_key + .as_deref() + .ok_or_else(|| anyhow::anyhow!("任务没有 parse_entry_key"))?; + let units = index + .units + .iter() + .filter(|unit| { + unit.parse_entry_key == parse_entry_key + && unit.destination == task.task.destination + && unit.archive_entry == task.task.archive_entry + }) + .map(|unit| provider_unit(task, unit)) + .collect::>(); + if units.is_empty() { + return Err(anyhow::anyhow!( + "任务 {} 在 TextUnit 索引中没有匹配条目", + task.task.task_id + )); + } + let provider_run_id = task + .provider_run_id + .clone() + .ok_or_else(|| anyhow::anyhow!("任务没有 provider_run_id"))?; + Ok(TranslationProviderRequest { + provider_run_id, + official_release_id: task.task.official_release_id.clone(), + task_id: task.task.task_id.clone(), + destination: task.task.destination.clone(), + archive_entry: task.task.archive_entry.clone(), + units, + }) +} + +fn provider_unit( + task: &PersistedTranslationTask, + unit: &OfficialTextUnitIndexUnit, +) -> TranslationProviderUnit { + TranslationProviderUnit { + unit_id: unit.id.clone(), + task_id: task.task.task_id.clone(), + official_release_id: task.task.official_release_id.clone(), + destination: unit.destination.clone(), + archive_entry: unit.archive_entry.clone(), + source_text: unit.source_text.clone(), + serialized_file: unit.serialized_file.clone(), + path_id: unit.path_id, + class_id: unit.class_id, + field_path: unit.field_path.clone(), + format: unit.format.clone(), + text_source_kind: unit.text_source_kind.clone(), + asset_name: unit.asset_name.clone(), + context: unit.context.clone(), + } +} + +fn validate_provider_response( + request: &TranslationProviderRequest, + response: TranslationProviderResponse, + provider_name: &str, +) -> anyhow::Result> { + if response.provider_run_id != request.provider_run_id { + return Err(anyhow::anyhow!( + "provider run ID 不一致：expected={} actual={}", + request.provider_run_id, + response.provider_run_id + )); + } + let expected = request + .units + .iter() + .map(|unit| (unit.unit_id.as_str(), unit)) + .collect::>(); + let mut seen = BTreeSet::new(); + let mut results = Vec::with_capacity(response.units.len()); + for result in response.units { + let Some(unit) = expected.get(result.unit_id.as_str()) else { + return Err(anyhow::anyhow!( + "provider 返回未知 TextUnit：{}", + result.unit_id + )); + }; + if !seen.insert(result.unit_id.clone()) { + return Err(anyhow::anyhow!( + "provider 重复返回 TextUnit：{}", + result.unit_id + )); + } + if result.source_text != unit.source_text { + return Err(anyhow::anyhow!( + "provider 返回的 source_text 与当前索引不一致：{}", + result.unit_id + )); + } + results.push(TranslationTaskUnitResult { + unit_id: result.unit_id, + source_text: result.source_text, + translated_text: result.translated_text, + provider: provider_name.to_string(), + provider_run_id: request.provider_run_id.clone(), + translated_unix_seconds: unix_seconds_now(), + }); + } + if seen.len() != expected.len() { + return Err(anyhow::anyhow!( + "provider 结果不完整：expected={} actual={}", + expected.len(), + seen.len() + )); + } + Ok(results) +} + +async fn record_provider_failure( + context: &WorkerTaskContext<'_>, + task: &PersistedTranslationTask, + error: TranslationProviderError, +) -> anyhow::Result<()> { + let retryable = error.retryable && task.attempt_count < context.max_attempts; + let next_attempt = + retryable.then(|| unix_seconds_now().saturating_add(context.retry_backoff.as_secs())); + context + .repository + .fail_claim(TranslationTaskFailure { + task_id: task.task.task_id.clone(), + worker_id: context.worker_id.to_string(), + provider_run_id: task.provider_run_id.clone().unwrap_or_default(), + failure_class: error.class.as_str().to_string(), + failure_reason: error.message.clone(), + retryable, + next_attempt_unix_seconds: next_attempt, + }) + .await + .map_err(|failure| anyhow::anyhow!("写入翻译任务失败状态失败：{failure}"))?; + context.stats.failed_count.fetch_add(1, Ordering::Relaxed); + if retryable { + context + .stats + .retry_scheduled_count + .fetch_add(1, Ordering::Relaxed); + } + context + .stats + .failures + .lock() + .map_err(|_| anyhow::anyhow!("写入翻译 worker 失败列表时 mutex poisoned"))? + .push(TranslationWorkerFailure { + task_id: task.task.task_id.clone(), + provider_run_id: task.provider_run_id.clone().unwrap_or_default(), + failure_class: error.class.as_str().to_string(), + failure_reason: error.message, + retryable, + }); + Ok(()) +} + +fn crowdin_curl_error(stderr: &[u8]) -> TranslationProviderError { + let message = String::from_utf8_lossy(stderr); + let status = (100..=599).find(|code| message.contains(&code.to_string())); + match status { + Some(401 | 403) => TranslationProviderError::new( + TranslationProviderFailureClass::Authentication, + "Crowdin authentication failed", + ), + Some(404) => TranslationProviderError::new( + TranslationProviderFailureClass::NotFound, + "Crowdin resource was not found", + ), + Some(429) => TranslationProviderError::new( + TranslationProviderFailureClass::RateLimited, + "Crowdin rate limit exceeded", + ), + Some(500..=599) => TranslationProviderError::new( + TranslationProviderFailureClass::Server, + format!("Crowdin server request failed with HTTP {status:?}"), + ), + _ => TranslationProviderError::new( + TranslationProviderFailureClass::Network, + "Crowdin request failed", + ), + } +} + +fn find_translation_text(value: &serde_json::Value) -> Option { + match value { + serde_json::Value::Object(object) => { + for key in ["translation", "text"] { + if let Some(serde_json::Value::String(text)) = object.get(key) { + return Some(text.clone()); + } + } + object.values().find_map(find_translation_text) + } + serde_json::Value::Array(values) => values.iter().find_map(find_translation_text), + _ => None, + } +} + +fn curl_config_escape(value: &str) -> String { + value.replace('\\', "\\\\").replace('"', "\\\"") +} + +fn default_mock_fixture_version() -> u32 { + MOCK_TRANSLATION_FIXTURE_VERSION +} + +fn unix_seconds_now() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::official_changes::OfficialResourceChangeKind; + use crate::official_parse::{ + OfficialParseSourceKind, OfficialParseStatus, OfficialTextUnitIndex, + OfficialTextUnitIndexSummary, + }; + use crate::official_textunit_queue::{ + OfficialTextUnitTask, OfficialTextUnitTaskQueue, OfficialTextUnitTaskStatus, + OfficialTextUnitTaskSummary, OFFICIAL_TEXTUNIT_TASK_QUEUE_VERSION, + }; + + fn fixture_root() -> (tempfile::TempDir, OfficialTextUnitTaskQueue) { + let temp = tempfile::TempDir::new().unwrap(); + let task = OfficialTextUnitTask { + task_id: "textunit/release-1/bundle".to_string(), + official_release_id: "release-1".to_string(), + destination: "bundle".to_string(), + change_kind: OfficialResourceChangeKind::Added, + url: "https://example.invalid/bundle".to_string(), + bytes: 10, + blake3: "hash".to_string(), + parse_entry_key: Some("direct:bundle".to_string()), + archive_entry: None, + source_kind: Some(OfficialParseSourceKind::DirectBundle), + parse_status: Some(OfficialParseStatus::Parsed), + text_asset_count: 1, + text_assets: vec!["Scenario".to_string()], + text_unit_count: 2, + text_unit_formats: vec!["plain".to_string()], + text_unit_error_count: 0, + status: OfficialTextUnitTaskStatus::QueuedOffline, + reason: None, + }; + let queue = OfficialTextUnitTaskQueue { + queue_version: OFFICIAL_TEXTUNIT_TASK_QUEUE_VERSION, + official_release_id: "release-1".to_string(), + previous_release_id: None, + generated_unix_seconds: 1, + current_resource_root: temp.path().to_path_buf(), + summary: OfficialTextUnitTaskSummary { + resource_candidate_count: 1, + parse_entry_count: 1, + queued_task_count: 1, + text_unit_count: 2, + ..OfficialTextUnitTaskSummary::default() + }, + tasks: vec![task], + }; + (temp, queue) + } + + fn index(root: &Path) -> crate::official_parse::OfficialTextUnitIndex { + OfficialTextUnitIndex { + version: crate::official_parse::OFFICIAL_TEXTUNIT_INDEX_VERSION, + generated_unix_seconds: 1, + resource_root: root.to_path_buf(), + summary: OfficialTextUnitIndexSummary { + unit_count: 2, + ..OfficialTextUnitIndexSummary::default() + }, + units: (0..2) + .map(|index| crate::official_parse::OfficialTextUnitIndexUnit { + id: format!("direct:bundle#unit:{index}"), + parse_entry_key: "direct:bundle".to_string(), + source_url: "https://example.invalid/bundle".to_string(), + destination: "bundle".to_string(), + archive_entry: None, + source_kind: OfficialParseSourceKind::DirectBundle, + unity_version: None, + source_text: format!("source-{index}"), + serialized_file: Some("CAB-bundle".to_string()), + path_id: Some(index), + class_id: Some(49), + field_path: Some(format!("text.{index}")), + field_offset: None, + field_byte_size: None, + format: Some("plain".to_string()), + text_source_kind: Some("text_asset".to_string()), + asset_name: Some("Scenario".to_string()), + context: BTreeMap::new(), + }) + .collect(), + errors: Vec::new(), + } + } + + #[tokio::test] + async fn mock_worker_claims_tasks_and_persists_textunit_results() { + let (temp, queue) = fixture_root(); + crate::official_textunit_queue::write_textunit_task_queue_at(temp.path(), &queue).unwrap(); + crate::official_parse::write_textunit_index_at(temp.path(), &index(temp.path())).unwrap(); + let fixture = temp.path().join("mock.json"); + std::fs::write( + &fixture, + serde_json::to_vec(&serde_json::json!({ + "schema_version": 1, + "translations": { + "direct:bundle#unit:0": "translated-0", + "direct:bundle#unit:1": "translated-1" + } + })) + .unwrap(), + ) + .unwrap(); + let config = TranslationWorkerConfig { + fixture_path: Some(fixture), + concurrency: 2, + retry_backoff: Duration::ZERO, + ..TranslationWorkerConfig::default() + }; + let report = run_translation_worker_at(temp.path(), &config) + .await + .unwrap(); + assert_eq!(report.completed_count, 1); + assert_eq!(report.failed_count, 0); + let repository = SqliteTranslationTaskRepository::open( + SqliteTranslationTaskRepository::repository_path(temp.path()), + ) + .await + .unwrap(); + let task = repository.find(&queue.tasks[0].task_id).await.unwrap(); + assert_eq!(task.task_status.as_str(), "completed"); + assert_eq!(task.translation_results.len(), 2); + assert_eq!(task.translation_results[0].translated_text, "translated-0"); + } + + #[tokio::test] + async fn mock_worker_retries_retryable_failures_and_keeps_diagnostic() { + let (temp, queue) = fixture_root(); + crate::official_textunit_queue::write_textunit_task_queue_at(temp.path(), &queue).unwrap(); + crate::official_parse::write_textunit_index_at(temp.path(), &index(temp.path())).unwrap(); + let fixture = temp.path().join("mock-failure.json"); + std::fs::write( + &fixture, + serde_json::to_vec(&serde_json::json!({ + "schema_version": 1, + "failures": { + "textunit/release-1/bundle": { + "class": "rate_limited", + "message": "fixture throttled", + "retryable": true + } + } + })) + .unwrap(), + ) + .unwrap(); + let config = TranslationWorkerConfig { + fixture_path: Some(fixture), + concurrency: 1, + max_attempts: 2, + retry_backoff: Duration::ZERO, + ..TranslationWorkerConfig::default() + }; + let report = run_translation_worker_at(temp.path(), &config) + .await + .unwrap(); + assert_eq!(report.failed_count, 2); + assert_eq!(report.retry_scheduled_count, 1); + assert_eq!(report.failures.len(), 2); + let repository = SqliteTranslationTaskRepository::open( + SqliteTranslationTaskRepository::repository_path(temp.path()), + ) + .await + .unwrap(); + let task = repository.find(&queue.tasks[0].task_id).await.unwrap(); + assert_eq!(task.attempt_count, 2); + assert_eq!(task.failure_class.as_deref(), Some("rate_limited")); + assert!(!task.failure_retryable); + } + + #[tokio::test] + async fn worker_claims_next_task_when_one_worker_finishes_before_slow_peer() { + let temp = tempfile::TempDir::new().unwrap(); + let tasks = ["task-a", "task-b", "task-c"] + .into_iter() + .map(|task_id| OfficialTextUnitTask { + task_id: task_id.to_string(), + official_release_id: "release-1".to_string(), + destination: format!("{task_id}.bundle"), + change_kind: OfficialResourceChangeKind::Added, + url: format!("https://example.invalid/{task_id}.bundle"), + bytes: 10, + blake3: format!("{task_id}-hash"), + parse_entry_key: Some(format!("direct:{task_id}.bundle")), + archive_entry: None, + source_kind: Some(OfficialParseSourceKind::DirectBundle), + parse_status: Some(OfficialParseStatus::Parsed), + text_asset_count: 1, + text_assets: vec!["Scenario".to_string()], + text_unit_count: 1, + text_unit_formats: vec!["plain".to_string()], + text_unit_error_count: 0, + status: OfficialTextUnitTaskStatus::QueuedOffline, + reason: None, + }) + .collect::>(); + let queue = OfficialTextUnitTaskQueue { + queue_version: OFFICIAL_TEXTUNIT_TASK_QUEUE_VERSION, + official_release_id: "release-1".to_string(), + previous_release_id: None, + generated_unix_seconds: 1, + current_resource_root: temp.path().to_path_buf(), + summary: OfficialTextUnitTaskSummary { + resource_candidate_count: 3, + parse_entry_count: 3, + queued_task_count: 3, + text_unit_count: 3, + ..OfficialTextUnitTaskSummary::default() + }, + tasks, + }; + let index = OfficialTextUnitIndex { + version: crate::official_parse::OFFICIAL_TEXTUNIT_INDEX_VERSION, + generated_unix_seconds: 1, + resource_root: temp.path().to_path_buf(), + summary: OfficialTextUnitIndexSummary { + unit_count: 3, + ..OfficialTextUnitIndexSummary::default() + }, + units: queue + .tasks + .iter() + .map(|task| crate::official_parse::OfficialTextUnitIndexUnit { + id: format!("{}#unit:0", task.parse_entry_key.as_deref().unwrap()), + parse_entry_key: task.parse_entry_key.clone().unwrap(), + source_url: task.url.clone(), + destination: task.destination.clone(), + archive_entry: None, + source_kind: OfficialParseSourceKind::DirectBundle, + unity_version: None, + source_text: format!("source-{}", task.task_id), + serialized_file: Some(format!("CAB-{}", task.task_id)), + path_id: Some(1), + class_id: Some(49), + field_path: Some("text.0".to_string()), + field_offset: None, + field_byte_size: None, + format: Some("plain".to_string()), + text_source_kind: Some("text_asset".to_string()), + asset_name: Some("Scenario".to_string()), + context: BTreeMap::new(), + }) + .collect(), + errors: Vec::new(), + }; + crate::official_textunit_queue::write_textunit_task_queue_at(temp.path(), &queue).unwrap(); + crate::official_parse::write_textunit_index_at(temp.path(), &index).unwrap(); + + #[derive(Debug)] + struct DelayProvider { + events: Arc>>, + } + + #[async_trait::async_trait] + impl TranslationProvider for DelayProvider { + fn name(&self) -> &'static str { + "delay" + } + + async fn translate( + &self, + request: TranslationProviderRequest, + ) -> Result { + self.events + .lock() + .unwrap() + .push(format!("start:{}", request.task_id)); + if request.task_id == "task-a" { + tokio::time::sleep(Duration::from_millis(150)).await; + } else { + tokio::time::sleep(Duration::from_millis(10)).await; + } + self.events + .lock() + .unwrap() + .push(format!("finish:{}", request.task_id)); + Ok(TranslationProviderResponse { + provider_run_id: request.provider_run_id, + units: request + .units + .into_iter() + .map(|unit| { + let source_text = unit.source_text; + TranslationProviderUnitResult { + unit_id: unit.unit_id, + translated_text: format!("translated-{source_text}"), + source_text, + } + }) + .collect(), + }) + } + } + + let events = Arc::new(Mutex::new(Vec::new())); + let provider = Arc::new(DelayProvider { + events: Arc::clone(&events), + }); + let config = TranslationWorkerConfig { + concurrency: 2, + retry_backoff: Duration::ZERO, + ..TranslationWorkerConfig::default() + }; + + let report = run_translation_worker_with_provider(temp.path(), &config, provider) + .await + .unwrap(); + + assert_eq!(report.completed_count, 3); + let events = events.lock().unwrap().clone(); + let start_c = events + .iter() + .position(|event| event == "start:task-c") + .unwrap(); + let finish_a = events + .iter() + .position(|event| event == "finish:task-a") + .unwrap(); + assert!( + start_c < finish_a, + "worker 应独立领取新任务，实际事件顺序：{events:?}" + ); + } + + #[test] + fn crowdin_provider_does_not_expose_token_in_debug() { + let provider = CrowdinProvider { + project_id: "1".to_string(), + language_id: "zh-CN".to_string(), + token: "secret-token".to_string(), + api_base_url: "https://api.example.invalid".to_string(), + curl_command: PathBuf::from("curl"), + }; + assert!(!format!("{provider:?}").contains("secret-token")); + } +} diff --git a/internal/api/admin.go b/internal/api/admin.go index 31c3d75..b912c2e 100644 --- a/internal/api/admin.go +++ b/internal/api/admin.go @@ -6,6 +6,7 @@ import ( "errors" "io" "net/http" + "net/url" "strconv" "strings" @@ -36,6 +37,8 @@ func (s *Server) handleAdminIndex(w http.ResponseWriter, r *http.Request) { "/v1/resources", "/openapi.yaml", "/admin/schedules", + "/admin/translation/tasks", + "/admin/translation/handoff", }, Controls: []string{ "/admin/control/reload", @@ -50,6 +53,7 @@ func (s *Server) handleAdminIndex(w http.ResponseWriter, r *http.Request) { "/admin/control/schedule-remove", "/admin/control/schedule-run", "/admin/control/translation-task-update", + "/admin/control/translation-worker-run", "/admin/control/translation-proofread", }, } @@ -82,6 +86,10 @@ func (s *Server) handleAdminControl(w http.ResponseWriter, r *http.Request) { s.handleAdminTranslationTaskUpdate(w, r) return } + if action == "translation-worker-run" { + s.handleAdminTranslationWorkerRun(w, r) + return + } if action == "translation-proofread" { s.handleAdminTranslationProofread(w, r) return @@ -187,6 +195,34 @@ func (s *Server) handleAdminTranslationTaskUpdate(w http.ResponseWriter, r *http }) } +func (s *Server) handleAdminTranslationWorkerRun(w http.ResponseWriter, r *http.Request) { + backend, ok := s.backend.(TranslationBackend) + if !ok || backend == nil { + writeErrorJSON(w, http.StatusServiceUnavailable, "translation_backend_unavailable", "Rust bat translation backend is unavailable") + return + } + var params backendrpc.TranslationWorkerRunParams + if !decodeAdminTranslationJSON(w, r, ¶ms) { + return + } + if err := validateTranslationWorkerRunParams(params); err != nil { + writeErrorJSON(w, http.StatusBadRequest, "invalid_translation_params", err.Error()) + return + } + result, err := backend.TranslationWorkerRun(r.Context(), params) + if err != nil { + s.writeControlBackendError(w, "translation-worker-run", err) + return + } + writeNoStoreJSON(w, http.StatusAccepted, AdminControlResponse{ + Service: "bat-api", + Action: "translation-worker-run", + RPCMethod: "translation.worker.run", + Status: "accepted", + Result: result, + }) +} + func (s *Server) handleAdminTranslationProofread(w http.ResponseWriter, r *http.Request) { backend, ok := s.backend.(TranslationBackend) if !ok || backend == nil { @@ -207,6 +243,63 @@ func (s *Server) handleAdminTranslationProofread(w http.ResponseWriter, r *http. }) } +func (s *Server) handleAdminTranslationTasks(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet && r.Method != http.MethodHead { + writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed") + return + } + if !s.requireAdminToken(w, r) { + return + } + backend, ok := s.backend.(TranslationBackend) + if !ok || backend == nil { + writeErrorJSON(w, http.StatusServiceUnavailable, "translation_backend_unavailable", "Rust bat translation backend is unavailable") + return + } + params, err := translationTaskListParams(r) + if err != nil { + writeErrorJSON(w, http.StatusBadRequest, "invalid_translation_query", err.Error()) + return + } + result, err := backend.TranslationTasks(r.Context(), params) + if err != nil { + s.writeControlBackendError(w, "translation-tasks", err) + return + } + if r.Method == http.MethodHead { + w.Header().Set("Cache-Control", "no-store") + w.WriteHeader(http.StatusOK) + return + } + writeNoStoreJSON(w, http.StatusOK, result) +} + +func (s *Server) handleAdminTranslationHandoff(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodGet && r.Method != http.MethodHead { + writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed") + return + } + if !s.requireAdminToken(w, r) { + return + } + backend, ok := s.backend.(TranslationBackend) + if !ok || backend == nil { + writeErrorJSON(w, http.StatusServiceUnavailable, "translation_backend_unavailable", "Rust bat translation backend is unavailable") + return + } + result, err := backend.TranslationHandoff(r.Context()) + if err != nil { + s.writeControlBackendError(w, "translation-handoff", err) + return + } + if r.Method == http.MethodHead { + w.Header().Set("Cache-Control", "no-store") + w.WriteHeader(http.StatusOK) + return + } + writeNoStoreJSON(w, http.StatusOK, result) +} + func (s *Server) handleAdminSchedules(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodGet && r.Method != http.MethodHead { writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed") @@ -254,6 +347,63 @@ func scheduleListParams(r *http.Request) (backendrpc.ScheduleListParams, error) return params, nil } +func translationTaskListParams(r *http.Request) (backendrpc.TranslationTaskListParams, error) { + query := r.URL.Query() + params := backendrpc.TranslationTaskListParams{ + TaskID: strings.TrimSpace(query.Get("task_id")), + ReleaseID: firstTrimmedQuery(query, "release_id", "official_release_id"), + Destination: strings.TrimSpace(query.Get("destination")), + PathPattern: strings.TrimSpace(query.Get("path_pattern")), + ArchiveEntry: strings.TrimSpace(query.Get("archive_entry")), + Status: firstTrimmedQuery(query, "status", "task_status"), + WorkerStatus: strings.TrimSpace(query.Get("worker_status")), + ParseStatus: strings.TrimSpace(query.Get("parse_status")), + Format: firstTrimmedQuery(query, "format", "text_unit_format"), + } + if raw := strings.TrimSpace(query.Get("offset")); raw != "" { + offset, err := strconv.ParseUint(raw, 10, 64) + if err != nil { + return backendrpc.TranslationTaskListParams{}, errors.New("offset must be a non-negative integer") + } + params.Offset = &offset + } + if raw := strings.TrimSpace(query.Get("limit")); raw != "" { + limit, err := strconv.ParseUint(raw, 10, 64) + if err != nil || limit == 0 || limit > 1000 { + return backendrpc.TranslationTaskListParams{}, errors.New("limit must be in 1..=1000") + } + params.Limit = &limit + } + if raw := strings.TrimSpace(query.Get("has_reason")); raw != "" { + hasReason, err := strconv.ParseBool(raw) + if err != nil { + return backendrpc.TranslationTaskListParams{}, errors.New("has_reason must be a boolean") + } + params.HasReason = &hasReason + } + if raw := strings.TrimSpace(query.Get("has_failure_reason")); raw != "" { + hasFailureReason, err := strconv.ParseBool(raw) + if err != nil { + return backendrpc.TranslationTaskListParams{}, errors.New("has_failure_reason must be a boolean") + } + params.HasFailureReason = &hasFailureReason + } + return params, nil +} + +func firstTrimmedQuery(query url.Values, keys ...string) string { + for _, key := range keys { + values := query[key] + if len(values) == 0 { + continue + } + if value := strings.TrimSpace(values[0]); value != "" { + return value + } + } + return "" +} + func (s *Server) handleAdminScheduleControl(w http.ResponseWriter, r *http.Request, action string) { backend, ok := s.backend.(ScheduleBackend) if !ok || backend == nil { @@ -325,7 +475,23 @@ func decodeAdminScheduleJSON(w http.ResponseWriter, r *http.Request, target any) } func decodeAdminTranslationJSON(w http.ResponseWriter, r *http.Request, target any) bool { - return decodeAdminJSON(w, r, target, "invalid_translation_params", "translation task request") + return decodeAdminJSON(w, r, target, "invalid_translation_params", "translation request") +} + +func validateTranslationWorkerRunParams(params backendrpc.TranslationWorkerRunParams) error { + if params.Concurrency != nil && (*params.Concurrency < 1 || *params.Concurrency > 256) { + return errors.New("translation worker concurrency must be in 1..=256") + } + if params.MaxAttempts != nil && *params.MaxAttempts == 0 { + return errors.New("translation worker max_attempts must be greater than 0") + } + if params.LeaseSeconds != nil && *params.LeaseSeconds == 0 { + return errors.New("translation worker lease_seconds must be greater than 0") + } + if params.MaxTasks != nil && *params.MaxTasks == 0 { + return errors.New("translation worker max_tasks must be greater than 0") + } + return nil } func decodeAdminJSON(w http.ResponseWriter, r *http.Request, target any, errorCode string, subject string) bool { diff --git a/internal/api/api_test.go b/internal/api/api_test.go index d9fa8bb..8e0feb4 100644 --- a/internal/api/api_test.go +++ b/internal/api/api_test.go @@ -588,7 +588,8 @@ func (f *fakeBackend) ResourceManifest(ctx context.Context, offset int, limit in type controlBackend struct { *fakeBackend - calls []string + calls []string + translationTaskListParams []backendrpc.TranslationTaskListParams } func (b *controlBackend) DaemonReload(ctx context.Context) (*backendrpc.Ack, error) { @@ -631,6 +632,38 @@ func (b *controlBackend) TranslationTaskUpdate(ctx context.Context, params backe return json.RawMessage(`{"task_status":"` + params.Status + `"}`), nil } +func (b *controlBackend) TranslationTasks(ctx context.Context, params backendrpc.TranslationTaskListParams) (json.RawMessage, error) { + b.calls = append(b.calls, "translation.tasks") + b.translationTaskListParams = append(b.translationTaskListParams, params) + return json.RawMessage(`{"tasks":[{"task_id":"textunit/v-current/Scenario","worker_status":"failed","failure_reason":"provider rejected payload"}]}`), nil +} + +func (b *controlBackend) TranslationHandoff(ctx context.Context) (json.RawMessage, error) { + b.calls = append(b.calls, "translation.handoff") + return json.RawMessage(`{"jobs":[],"provider_runs":[]}`), nil +} + +func (b *controlBackend) TranslationWorkerRun(ctx context.Context, params backendrpc.TranslationWorkerRunParams) (*backendrpc.TranslationWorkerRunResult, error) { + b.calls = append(b.calls, "translation.worker.run") + concurrency := uint64(8) + if params.Concurrency != nil { + concurrency = *params.Concurrency + } + return &backendrpc.TranslationWorkerRunResult{ + TaskID: "task-translation-worker-1", + Kind: "translation.worker.run", + Worker: backendrpc.TranslationWorkerConfig{ + Provider: params.Provider, + FixturePath: params.FixturePath, + Concurrency: concurrency, + MaxAttempts: 3, + LeaseSeconds: 300, + RetryBackoffSeconds: 5, + WorkerID: params.WorkerID, + }, + }, nil +} + func (b *controlBackend) TranslationProofread(ctx context.Context) (json.RawMessage, error) { b.calls = append(b.calls, "translation.proofread") return json.RawMessage(`{"translation_workflow_status":"manual_proofreading"}`), nil @@ -743,6 +776,81 @@ func TestAdminScheduleEndpointsProxyAuthenticatedRequests(t *testing.T) { } } +func TestAdminTranslationQueryEndpointsProxyAuthenticatedRequests(t *testing.T) { + cfg := DefaultConfig() + cfg.AuthToken = "translation-token" + if err := cfg.Normalize(); err != nil { + t.Fatal(err) + } + backend := &controlBackend{fakeBackend: &fakeBackend{}} + s := NewServer(cfg, backend, nil) + + request := httptest.NewRequest( + http.MethodGet, + "/admin/translation/tasks?offset=10&limit=25&task_id=textunit%2Fv-current%2FScenario&release_id=v-current&destination=MediaResources%2FGameData%2FScenario.zip&path_pattern=Scenario&archive_entry=ScenarioExcelTable.json&status=queued_offline&worker_status=failed&parse_status=ok&format=json&has_reason=true&has_failure_reason=false", + nil, + ) + request.Header.Set("Authorization", "Bearer translation-token") + recorder := httptest.NewRecorder() + s.Handler().ServeHTTP(recorder, request) + if recorder.Code != http.StatusOK { + t.Fatalf("tasks status=%d body=%s", recorder.Code, recorder.Body.String()) + } + if !json.Valid(recorder.Body.Bytes()) || !strings.Contains(recorder.Body.String(), "failure_reason") { + t.Fatalf("tasks body=%s", recorder.Body.String()) + } + if len(backend.translationTaskListParams) != 1 { + t.Fatalf("translation task query params=%#v", backend.translationTaskListParams) + } + params := backend.translationTaskListParams[0] + if params.Offset == nil || *params.Offset != 10 || + params.Limit == nil || *params.Limit != 25 || + params.TaskID != "textunit/v-current/Scenario" || + params.ReleaseID != "v-current" || + params.Destination != "MediaResources/GameData/Scenario.zip" || + params.PathPattern != "Scenario" || + params.ArchiveEntry != "ScenarioExcelTable.json" || + params.Status != "queued_offline" || + params.WorkerStatus != "failed" || + params.ParseStatus != "ok" || + params.Format != "json" || + params.HasReason == nil || !*params.HasReason || + params.HasFailureReason == nil || *params.HasFailureReason { + t.Fatalf("params=%#v", params) + } + + request = httptest.NewRequest(http.MethodGet, "/admin/translation/handoff", nil) + request.Header.Set("Authorization", "Bearer translation-token") + recorder = httptest.NewRecorder() + s.Handler().ServeHTTP(recorder, request) + if recorder.Code != http.StatusOK { + t.Fatalf("handoff status=%d body=%s", recorder.Code, recorder.Body.String()) + } + if !strings.Contains(recorder.Body.String(), "provider_runs") { + t.Fatalf("handoff body=%s", recorder.Body.String()) + } + if len(backend.calls) < 2 || + backend.calls[len(backend.calls)-2] != "translation.tasks" || + backend.calls[len(backend.calls)-1] != "translation.handoff" { + t.Fatalf("calls=%v", backend.calls) + } + + request = httptest.NewRequest(http.MethodGet, "/admin/translation/tasks?limit=0", nil) + request.Header.Set("Authorization", "Bearer translation-token") + recorder = httptest.NewRecorder() + s.Handler().ServeHTTP(recorder, request) + if recorder.Code != http.StatusBadRequest { + t.Fatalf("invalid task query status=%d body=%s", recorder.Code, recorder.Body.String()) + } + + request = httptest.NewRequest(http.MethodGet, "/admin/translation/tasks", nil) + recorder = httptest.NewRecorder() + s.Handler().ServeHTTP(recorder, request) + if recorder.Code != http.StatusUnauthorized { + t.Fatalf("unauthenticated tasks status=%d body=%s", recorder.Code, recorder.Body.String()) + } +} + func TestDiscoverCallsStatusBeforeDoctor(t *testing.T) { root := fixtureRoot(t) bytes := uint64(20) @@ -1281,6 +1389,11 @@ func TestOpenAPIAndAdminReservedEndpoints(t *testing.T) { if len(admin.Controls) == 0 || admin.Controls[0] != "/admin/control/reload" { t.Fatalf("admin controls=%v", admin.Controls) } + links := strings.Join(admin.Links, "\n") + if !strings.Contains(links, "/admin/translation/tasks") || + !strings.Contains(links, "/admin/translation/handoff") { + t.Fatalf("admin links=%v", admin.Links) + } } func TestAdminControlForwardsAllowlistedActions(t *testing.T) { @@ -1305,6 +1418,7 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) { {name: "repair", action: "repair", rpcMethod: "resource.repair", call: "resource.repair"}, {name: "catalog refresh", action: "catalog-refresh", rpcMethod: "catalog.refresh", call: "catalog.refresh"}, {name: "translation task update", action: "translation-task-update", body: `{"task_id":"textunit/v-current/Scenario","status":"failed","failure_reason":"provider rejected payload","provider_run_id":"provider-run-1"}`, rpcMethod: "translation.task.update", call: "translation.task.update"}, + {name: "translation worker run", action: "translation-worker-run", body: `{"provider":"mock","concurrency":8,"max_tasks":2,"retry_backoff_seconds":0,"worker_id":"dashboard-worker"}`, rpcMethod: "translation.worker.run", call: "translation.worker.run"}, {name: "translation proofread", action: "translation-proofread", rpcMethod: "translation.proofread", call: "translation.proofread"}, } for _, tc := range tests { @@ -1336,6 +1450,14 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) { if recorder.Code != http.StatusBadRequest { t.Fatalf("invalid translation update status=%d body=%s", recorder.Code, recorder.Body.String()) } + + request = httptest.NewRequest(http.MethodPost, "/admin/control/translation-worker-run", strings.NewReader(`{"concurrency":0}`)) + request.Header.Set("Authorization", "Bearer control-token") + recorder = httptest.NewRecorder() + s.Handler().ServeHTTP(recorder, request) + if recorder.Code != http.StatusBadRequest { + t.Fatalf("invalid translation worker status=%d body=%s", recorder.Code, recorder.Body.String()) + } } func TestAdminControlRejectsUnauthenticatedDangerousAndUnsupportedActions(t *testing.T) { diff --git a/internal/api/openapi.go b/internal/api/openapi.go index 1021bce..b3cf30a 100644 --- a/internal/api/openapi.go +++ b/internal/api/openapi.go @@ -139,6 +139,80 @@ paths: description: Missing or invalid admin token. "503": description: Rust bat schedule backend is unavailable. + /admin/translation/tasks: + get: + summary: List Rust-owned translation task worker status + parameters: + - name: offset + in: query + schema: + type: integer + minimum: 0 + - name: limit + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + - name: task_id + in: query + schema: + type: string + - name: release_id + in: query + schema: + type: string + - name: destination + in: query + schema: + type: string + - name: archive_entry + in: query + schema: + type: string + - name: status + in: query + schema: + type: string + - name: worker_status + in: query + schema: + type: string + - name: parse_status + in: query + schema: + type: string + - name: format + in: query + schema: + type: string + - name: has_reason + in: query + schema: + type: boolean + - name: has_failure_reason + in: query + schema: + type: boolean + responses: + "200": + description: Current translation task JSON report from Rust bat. + "400": + description: Invalid translation task query. + "401": + description: Missing or invalid admin token. + "503": + description: Rust bat translation backend is unavailable. + /admin/translation/handoff: + get: + summary: Read Rust-owned translation handoff state + responses: + "200": + description: Current translation handoff JSON report from Rust bat. + "401": + description: Missing or invalid admin token. + "503": + description: Rust bat translation backend is unavailable. /admin/control/{action}: post: summary: Forward an allowlisted control or schedule action to Rust bat @@ -148,7 +222,7 @@ paths: required: true schema: type: string - enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, translation-task-update, translation-proofread] + enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, translation-task-update, translation-worker-run, translation-proofread] requestBody: required: false content: @@ -199,6 +273,34 @@ paths: type: string provider_run_id: type: string + provider: + type: string + enum: [mock, crowdin] + fixture_path: + type: string + concurrency: + type: integer + format: int64 + minimum: 1 + maximum: 256 + max_attempts: + type: integer + format: int64 + minimum: 1 + lease_seconds: + type: integer + format: int64 + minimum: 1 + retry_backoff_seconds: + type: integer + format: int64 + minimum: 0 + max_tasks: + type: integer + format: int64 + minimum: 1 + worker_id: + type: string responses: "202": description: Rust bat accepted the control request. diff --git a/internal/api/rpc_release.go b/internal/api/rpc_release.go index 10e89c8..9d2529a 100644 --- a/internal/api/rpc_release.go +++ b/internal/api/rpc_release.go @@ -54,6 +54,9 @@ type ScheduleBackend interface { // by the dashboard. It does not create arbitrary translation jobs. type TranslationBackend interface { TranslationTaskUpdate(ctx context.Context, params backendrpc.TranslationTaskUpdateParams) (json.RawMessage, error) + TranslationTasks(ctx context.Context, params backendrpc.TranslationTaskListParams) (json.RawMessage, error) + TranslationHandoff(ctx context.Context) (json.RawMessage, error) + TranslationWorkerRun(ctx context.Context, params backendrpc.TranslationWorkerRunParams) (*backendrpc.TranslationWorkerRunResult, error) TranslationProofread(ctx context.Context) (json.RawMessage, error) } @@ -117,6 +120,18 @@ func (r RPCClient) TranslationTaskUpdate(ctx context.Context, params backendrpc. return r.Client.TranslationTaskUpdate(ctx, params) } +func (r RPCClient) TranslationTasks(ctx context.Context, params backendrpc.TranslationTaskListParams) (json.RawMessage, error) { + return r.Client.TranslationTasks(ctx, params) +} + +func (r RPCClient) TranslationHandoff(ctx context.Context) (json.RawMessage, error) { + return r.Client.TranslationHandoff(ctx) +} + +func (r RPCClient) TranslationWorkerRun(ctx context.Context, params backendrpc.TranslationWorkerRunParams) (*backendrpc.TranslationWorkerRunResult, error) { + return r.Client.TranslationWorkerRun(ctx, params) +} + func (r RPCClient) TranslationProofread(ctx context.Context) (json.RawMessage, error) { return r.Client.TranslationProofread(ctx) } diff --git a/internal/api/server.go b/internal/api/server.go index 65aebf6..07ba4e9 100644 --- a/internal/api/server.go +++ b/internal/api/server.go @@ -62,6 +62,8 @@ func (s *Server) Handler() http.Handler { mux.HandleFunc(launcherHostPath("/api/launcher/resource/bootstrap.json"), s.handleLauncherBootstrap) mux.HandleFunc("/openapi.yaml", s.handleOpenAPI) mux.HandleFunc("/admin/schedules", s.handleAdminSchedules) + mux.HandleFunc("/admin/translation/tasks", s.handleAdminTranslationTasks) + mux.HandleFunc("/admin/translation/handoff", s.handleAdminTranslationHandoff) mux.HandleFunc("/admin/control/", s.handleAdminControl) mux.HandleFunc("/admin/", s.handleAdminIndex) mux.HandleFunc("/"+ServerInfoHost+"/", s.handleServerInfoCDN) @@ -127,6 +129,8 @@ func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) { "/openapi.yaml", "/admin/", "/admin/schedules", + "/admin/translation/tasks", + "/admin/translation/handoff", "/admin/control/{action}", }, }) diff --git a/internal/backendrpc/client.go b/internal/backendrpc/client.go index ff5971a..217e2d8 100644 --- a/internal/backendrpc/client.go +++ b/internal/backendrpc/client.go @@ -273,6 +273,58 @@ type TranslationTaskUpdateParams struct { ProviderRunID string `json:"provider_run_id,omitempty"` } +// TranslationTaskListParams filters the Rust-owned translation task queue. +// The result shape is intentionally left as Rust JSON so Go does not duplicate +// the translation task schema. +type TranslationTaskListParams struct { + Offset *uint64 `json:"offset,omitempty"` + Limit *uint64 `json:"limit,omitempty"` + TaskID string `json:"task_id,omitempty"` + ReleaseID string `json:"release_id,omitempty"` + Destination string `json:"destination,omitempty"` + PathPattern string `json:"path_pattern,omitempty"` + ArchiveEntry string `json:"archive_entry,omitempty"` + Status string `json:"status,omitempty"` + WorkerStatus string `json:"worker_status,omitempty"` + ParseStatus string `json:"parse_status,omitempty"` + Format string `json:"format,omitempty"` + HasReason *bool `json:"has_reason,omitempty"` + HasFailureReason *bool `json:"has_failure_reason,omitempty"` +} + +// TranslationWorkerRunParams starts one Rust-owned provider worker task. +// Pointer numeric fields preserve explicit zeroes so Rust can reject invalid +// dashboard input instead of receiving omitted defaults. +type TranslationWorkerRunParams struct { + Provider string `json:"provider,omitempty"` + FixturePath string `json:"fixture_path,omitempty"` + Concurrency *uint64 `json:"concurrency,omitempty"` + MaxAttempts *uint64 `json:"max_attempts,omitempty"` + LeaseSeconds *uint64 `json:"lease_seconds,omitempty"` + RetryBackoffSeconds *uint64 `json:"retry_backoff_seconds,omitempty"` + MaxTasks *uint64 `json:"max_tasks,omitempty"` + WorkerID string `json:"worker_id,omitempty"` +} + +// TranslationWorkerConfig mirrors the accepted worker config returned by Rust. +type TranslationWorkerConfig struct { + Provider string `json:"provider"` + FixturePath string `json:"fixture_path,omitempty"` + Concurrency uint64 `json:"concurrency"` + MaxAttempts uint64 `json:"max_attempts"` + LeaseSeconds uint64 `json:"lease_seconds"` + RetryBackoffSeconds uint64 `json:"retry_backoff_seconds"` + MaxTasks *uint64 `json:"max_tasks,omitempty"` + WorkerID string `json:"worker_id"` +} + +// TranslationWorkerRunResult is returned when translation.worker.run is queued. +type TranslationWorkerRunResult struct { + TaskID string `json:"task_id"` + Kind string `json:"kind"` + Worker TranslationWorkerConfig `json:"worker"` +} + // TranslationProofread marks the current localized workflow as manual // proofreading. Rust owns the persisted localized state. func (c *Client) TranslationProofread(ctx context.Context) (json.RawMessage, error) { @@ -502,6 +554,20 @@ func (c *Client) TranslationTaskUpdate(ctx context.Context, params TranslationTa return c.rawData(ctx, "translation.task.update", params) } +func (c *Client) TranslationTasks(ctx context.Context, params TranslationTaskListParams) (json.RawMessage, error) { + return c.rawData(ctx, "translation.tasks", params) +} + +func (c *Client) TranslationHandoff(ctx context.Context) (json.RawMessage, error) { + return c.rawData(ctx, "translation.handoff", nil) +} + +func (c *Client) TranslationWorkerRun(ctx context.Context, params TranslationWorkerRunParams) (*TranslationWorkerRunResult, error) { + var out TranslationWorkerRunResult + _, err := c.Call(ctx, "translation.worker.run", params, &out) + return &out, err +} + func (c *Client) CatalogStatus(ctx context.Context) (json.RawMessage, error) { return c.rawData(ctx, "catalog.status", nil) } diff --git a/internal/backendrpc/client_test.go b/internal/backendrpc/client_test.go index d955f50..2bbc060 100644 --- a/internal/backendrpc/client_test.go +++ b/internal/backendrpc/client_test.go @@ -336,6 +336,94 @@ func TestScheduleRunSendsScopeAndMaxRuns(t *testing.T) { } } +func TestTranslationTasksSendsQueryParams(t *testing.T) { + offset := uint64(10) + limit := uint64(25) + hasReason := true + hasFailureReason := false + client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { + if req.Method != "translation.tasks" { + t.Fatalf("method = %s", req.Method) + } + var params TranslationTaskListParams + if err := json.Unmarshal(req.Params, ¶ms); err != nil { + t.Fatalf("decode params: %v", err) + } + if params.Offset == nil || *params.Offset != offset || + params.Limit == nil || *params.Limit != limit || + params.TaskID != "textunit/v-current/Scenario" || + params.ReleaseID != "v-current" || + params.Destination != "MediaResources/GameData/Scenario.zip" || + params.PathPattern != "Scenario" || + params.ArchiveEntry != "ScenarioExcelTable.json" || + params.Status != "queued_offline" || + params.WorkerStatus != "failed" || + params.ParseStatus != "ok" || + params.Format != "json" || + params.HasReason == nil || *params.HasReason != hasReason || + params.HasFailureReason == nil || *params.HasFailureReason != hasFailureReason { + t.Fatalf("params = %#v", params) + } + return testResponse{ + Result: testEnvelope{ + OK: true, + Status: "ok", + RequestID: "req-test-translation-tasks", + Data: map[string]any{"tasks": []any{}}, + }, + } + }) + + raw, err := client.TranslationTasks(context.Background(), TranslationTaskListParams{ + Offset: &offset, + Limit: &limit, + TaskID: "textunit/v-current/Scenario", + ReleaseID: "v-current", + Destination: "MediaResources/GameData/Scenario.zip", + PathPattern: "Scenario", + ArchiveEntry: "ScenarioExcelTable.json", + Status: "queued_offline", + WorkerStatus: "failed", + ParseStatus: "ok", + Format: "json", + HasReason: &hasReason, + HasFailureReason: &hasFailureReason, + }) + if err != nil { + t.Fatalf("TranslationTasks error: %v", err) + } + if !json.Valid(raw) { + t.Fatalf("invalid raw JSON: %s", string(raw)) + } +} + +func TestTranslationHandoffUsesRustMethod(t *testing.T) { + client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { + if req.Method != "translation.handoff" { + t.Fatalf("method = %s", req.Method) + } + if string(req.Params) != "null" { + t.Fatalf("params = %s, want null", req.Params) + } + return testResponse{ + Result: testEnvelope{ + OK: true, + Status: "ok", + RequestID: "req-test-translation-handoff", + Data: map[string]any{"jobs": []any{}}, + }, + } + }) + + raw, err := client.TranslationHandoff(context.Background()) + if err != nil { + t.Fatalf("TranslationHandoff error: %v", err) + } + if !json.Valid(raw) { + t.Fatalf("invalid raw JSON: %s", string(raw)) + } +} + func TestTranslationTaskUpdateSendsWorkerParams(t *testing.T) { client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { if req.Method != "translation.task.update" { @@ -375,6 +463,75 @@ func TestTranslationTaskUpdateSendsWorkerParams(t *testing.T) { } } +func TestTranslationWorkerRunSendsProviderConfig(t *testing.T) { + var concurrency uint64 = 8 + var maxAttempts uint64 = 4 + var leaseSeconds uint64 = 60 + var retryBackoff uint64 = 0 + var maxTasks uint64 = 2 + client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { + if req.Method != "translation.worker.run" { + t.Fatalf("method = %s", req.Method) + } + var params TranslationWorkerRunParams + if err := json.Unmarshal(req.Params, ¶ms); err != nil { + t.Fatalf("decode params: %v", err) + } + if params.Provider != "mock" || + params.FixturePath != "/tmp/mock-provider.json" || + params.Concurrency == nil || *params.Concurrency != concurrency || + params.MaxAttempts == nil || *params.MaxAttempts != maxAttempts || + params.LeaseSeconds == nil || *params.LeaseSeconds != leaseSeconds || + params.RetryBackoffSeconds == nil || *params.RetryBackoffSeconds != retryBackoff || + params.MaxTasks == nil || *params.MaxTasks != maxTasks || + params.WorkerID != "dashboard-worker" { + t.Fatalf("params = %#v", params) + } + return testResponse{ + Result: testEnvelope{ + OK: true, + Status: "accepted", + RequestID: "req-test-translation-worker", + Data: map[string]any{ + "task_id": "task-worker-1", + "kind": "translation.worker.run", + "worker": map[string]any{ + "provider": "mock", + "fixture_path": "/tmp/mock-provider.json", + "concurrency": concurrency, + "max_attempts": maxAttempts, + "lease_seconds": leaseSeconds, + "retry_backoff_seconds": retryBackoff, + "max_tasks": maxTasks, + "worker_id": "dashboard-worker", + }, + }, + }, + } + }) + + result, err := client.TranslationWorkerRun(context.Background(), TranslationWorkerRunParams{ + Provider: "mock", + FixturePath: "/tmp/mock-provider.json", + Concurrency: &concurrency, + MaxAttempts: &maxAttempts, + LeaseSeconds: &leaseSeconds, + RetryBackoffSeconds: &retryBackoff, + MaxTasks: &maxTasks, + WorkerID: "dashboard-worker", + }) + if err != nil { + t.Fatalf("TranslationWorkerRun error: %v", err) + } + if result.TaskID != "task-worker-1" || + result.Kind != "translation.worker.run" || + result.Worker.Concurrency != concurrency || + result.Worker.RetryBackoffSeconds != retryBackoff || + result.Worker.MaxTasks == nil || *result.Worker.MaxTasks != maxTasks { + t.Fatalf("unexpected result: %#v", result) + } +} + func TestTranslationProofreadUsesRustMethod(t *testing.T) { client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { if req.Method != "translation.proofread" { diff --git a/web/admin/README.md b/web/admin/README.md index 42cc010..a11c123 100644 --- a/web/admin/README.md +++ b/web/admin/README.md @@ -1,16 +1,26 @@ # bat-api Admin Panel -This directory is reserved for the future player-facing `bat-api` management -panel. The current backend exposes a non-mutating index at `GET /admin/` that -returns JSON links for health, readiness, bootstrap, release, resources, and -OpenAPI. `GET /admin/schedules` and mutating -`/admin/control/schedule-{add,update,remove,run}` routes are protected by the -configured admin token and forward the Rust-owned schedule contract. -`GET /admin/schedules` accepts optional `id`, `group`, and `enabled` filters. -`POST /admin/control/translation-task-update` uses the same token and forwards -the narrow `translation.task.update` provider-worker status contract. +本目录预留给后续 `bat-api` 管理面板。当前后端已提供只读入口 +`GET /admin/`，返回 health、ready、bootstrap、release、resources 和 OpenAPI +链接。 -The production panel must reuse the same HTTP authentication, rate limiting, -access logging, reverse-proxy handling, and no-store dynamic response policy as -the resource API. Static frontend assets are not implemented yet. See -`docs/reports/GO_STATUS.md` for the current component boundary and status. +需要配置 `BAT_API_AUTH_TOKEN` 的接口： + +- `GET /admin/schedules`：转发 Rust 持有的 schedule 查询，支持 `id`、`group` + 和 `enabled` 过滤。 +- `POST /admin/control/schedule-{add,update,remove,run}`：转发 + `schedule.*` 计划控制。 +- `POST /admin/control/translation-task-update`：转发 + `translation.task.update`，供外部 provider 流程回写任务状态。 +- `POST /admin/control/translation-worker-run`：转发 + `translation.worker.run`，触发 Rust provider worker。 +- `POST /admin/control/translation-proofread`：转发 + `translation.proofread`，把汉化 workflow 标记为人工校对中。 +- `GET /admin/translation/tasks`：转发 `translation.tasks`，按分页、release、 + TextUnit、状态和失败原因过滤查询 worker 状态。 +- `GET /admin/translation/handoff`：转发 `translation.handoff`，读取当前 + release 的完整翻译交接视图。 + +正式前端必须复用资源 API 的 HTTP 鉴权、限流、访问日志、反代处理和动态响应 +`Cache-Control: no-store` 策略。静态前端资产仍为 not implemented，尚未作为 +本目录的稳定交付物。当前组件边界见 `docs/reports/GO_STATUS.md`。