feat(i18n): 接入翻译 provider worker
bat-rust / Build and test Rust (push) Canceled after 0s
bat-rust / Build and test Go API (push) Canceled after 0s

Closes #44
This commit is contained in:
2026-08-30 21:13:31 +08:00
parent 7f465523e1
commit f441f1810e
29 changed files with 3815 additions and 130 deletions
+7 -7
View File
@@ -28,9 +28,9 @@ Rust 侧官方日服资源链路已经从实验验证推进到正式入口:
11. 官方同步会拒绝危险输出目录、路径逃逸和现有 symlink 路径组件;下载目标、`.part`、manifest、snapshot、PID、status、log 和控制锁文件不会跟随 symlink,daemon 状态类文件默认以 `0600` 权限创建。 11. 官方同步会拒绝危险输出目录、路径逃逸和现有 symlink 路径组件;下载目标、`.part`、manifest、snapshot、PID、status、log 和控制锁文件不会跟随 symlink,daemon 状态类文件默认以 `0600` 权限创建。
12. `<output>/official-version-state.json` 会明确保存当前已完成版本、正在拉取版本、上一个可用版本和失败版本;同一 app version、bundle version 和 Addressables root 的失败只保留最新一条,同一版本开始重新拉取或后续发布成功时会清理对应失败记录;`bat status` 会显示最后成功时间、下次检查时间、最后错误摘要、当前阶段、当前下载 URL 进度、版本状态摘要、最近历史失败版本和原因、结构化日志路径和轮转日志路径,人类可读输出不会把完整版本状态 JSON 内联打印。 12. `<output>/official-version-state.json` 会明确保存当前已完成版本、正在拉取版本、上一个可用版本和失败版本;同一 app version、bundle version 和 Addressables root 的失败只保留最新一条,同一版本开始重新拉取或后续发布成功时会清理对应失败记录;`bat status` 会显示最后成功时间、下次检查时间、最后错误摘要、当前阶段、当前下载 URL 进度、版本状态摘要、最近历史失败版本和原因、结构化日志路径和轮转日志路径,人类可读输出不会把完整版本状态 JSON 内联打印。
13. 资源导入链路已支持 CAS + `ResourceRepository` 索引写入,官方同步可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后触发导入,默认 CAS 路径为 `<output>/.cas`、SQLite 索引为 `<output>/resources.sqlite`,也可通过 `--import-cas-root``--import-resource-db``BAT_IMPORT_CAS_ROOT``BAT_IMPORT_RESOURCE_DB` 覆盖;`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 分页查询现有索引,数据库不存在时返回 `available=false` 且不会创建空库。`Resource` metadata 已通过 `metadata_json` 兼容迁移保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式;当前/上一个/结构变化 catalog、失败 staging 复用、403/404、hash mismatch、CRC、metadata 迁移与 UnityFS 边界校验均有离线回归 fixture 或单测覆盖。 13. 资源导入链路已支持 CAS + `ResourceRepository` 索引写入,官方同步可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后触发导入,默认 CAS 路径为 `<output>/.cas`、SQLite 索引为 `<output>/resources.sqlite`,也可通过 `--import-cas-root``--import-resource-db``BAT_IMPORT_CAS_ROOT``BAT_IMPORT_RESOURCE_DB` 覆盖;`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、bundle path、archive entry、parse status 和 TextUnit format 分页查询现有索引,数据库不存在时返回 `available=false` 且不会创建空库。`Resource` metadata 已通过 `metadata_json` 兼容迁移保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式;当前/上一个/结构变化 catalog、失败 staging 复用、403/404、hash mismatch、CRC、metadata 迁移与 UnityFS 边界校验均有离线回归 fixture 或单测覆盖。
14. 非 dry-run 官方同步在校验完成并发布后,会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`,在当前 release 下写入 `official-resource-changes.json``crowdin-translation-handoff.json`;同一 destination 只有 size 或 BLAKE3 改变才算 modified,仅 URL/CDN 根变化但内容相同不会触发解析/翻译候选。随后刷新 `official-parse-cache.json``official-textunit-index.json`,并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json``crowdin-textunit-queue.json`;删除资源只进入差异记录,不进入 TextUnit/Crowdin 队列。`parse.text_units``parse.errors` RPC/CLI 可按 destination、archive entry、path id、class id、field path 和 format 查询当前 release 的 TextUnit 明细与解析错误;`translation.tasks` RPC/CLI 可按 release、destination、archive entry、任务状态、parse status、TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因;`translation.task.update` 可回写 provider worker 状态,`translation.proofread` 可把汉化 workflow 标记为人工校对中;TextUnit 已包含 class id、field path、字段 offset/byte size 等可追溯定位。Crowdin 当前仅预留本地离线队列,不发网络请求up-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要,不重复解析。官方同步报告默认 `localized_release_status=not_localized`,含义是原版资源已经发布、汉化资源未发布;UnityFS TextAsset patch 发布成功后会写 `localized-patch-manifest.json`,校验 hash/size/diff/rollback 后才允许 `localized.status` 返回 `status=published``status_code=localized.published``localized_release_status=localized``bat` 首次启动会在二进制所在目录释放 `.env` 配置模板(`0600`),之后每次启动自动加载(不覆盖已存在的环境变量),支持 `BAT_OUTPUT`/`BAT_LOCALIZED_OUTPUT`/`BAT_IMPORT_REPOSITORY`/`BAT_IMPORT_CAS_ROOT`/`BAT_IMPORT_RESOURCE_DB`/`BAT_STATE_DIR`/`BAT_AUTO_DISCOVER`/`BAT_WATCH`/`BAT_DAEMON`/`BAT_PROXY` 等键,实现编辑 `.env` 后无参启动;优先级为命令行参数 > 进程环境变量 > `.env` > 内置默认值,`BAT_SKIP_ENV_FILE=1` 可整体禁用;Redis 键为预留。daemon 任务历史持久化在 `<state-dir>/bat-tasks.json`(版本化、`0600` 原子写),重启后任务经 `task.*` 仍可查,中断任务标记 `task_interrupted``BAT-ERR-700005`)。 14. 非 dry-run 官方同步在校验完成并发布后,会先对比上一完整 release 与当前 release 的 `official-download-manifest.json`,在当前 release 下写入 `official-resource-changes.json``crowdin-translation-handoff.json`;同一 destination 只有 size 或 BLAKE3 改变才算 modified,仅 URL/CDN 根变化但内容相同不会触发解析/翻译候选。随后刷新 `official-parse-cache.json``official-textunit-index.json`,并从 Added/Modified 资源、parse cache 与 TextUnit 明细索引派生 `official-textunit-tasks.json``crowdin-textunit-queue.json`;删除资源只进入差异记录,不进入 TextUnit/Crowdin 队列。`parse.text_units``parse.errors` RPC/CLI 可按 destination、archive entry、path id、class id、field path 和 format 查询当前 release 的 TextUnit 明细与解析错误;`translation.tasks` RPC/CLI 可按 release、destination、archive entry、任务状态、parse status、TextUnit format 和 reason presence 查询离线 TextUnit 翻译任务状态与跳过/失败原因;`translation.task.update` 可回写 provider worker 状态,`translation.worker.run` 可触发 Rust provider worker 独立 claim/lease/retry 并落库 TextUnit 级译文结果,`translation.proofread` 可把汉化 workflow 标记为人工校对中;TextUnit 已包含 class id、field path、字段 offset/byte size 等可追溯定位。Crowdin provider 通过 `CROWDIN_*` 环境变量接入,mock provider 支持本地 fixtureup-to-date 轮询在已有有效解析缓存、TextUnit 明细索引和 TextUnit 队列时只读取摘要,不重复解析。官方同步报告默认 `localized_release_status=not_localized`,含义是原版资源已经发布、汉化资源未发布;UnityFS TextAsset patch 发布成功后会写 `localized-patch-manifest.json`,校验 hash/size/diff/rollback 后才允许 `localized.status` 返回 `status=published``status_code=localized.published``localized_release_status=localized``bat` 首次启动会在二进制所在目录释放 `.env` 配置模板(`0600`),之后每次启动自动加载(不覆盖已存在的环境变量),支持 `BAT_OUTPUT`/`BAT_LOCALIZED_OUTPUT`/`BAT_IMPORT_REPOSITORY`/`BAT_IMPORT_CAS_ROOT`/`BAT_IMPORT_RESOURCE_DB`/`BAT_STATE_DIR`/`BAT_AUTO_DISCOVER`/`BAT_WATCH`/`BAT_DAEMON`/`BAT_PROXY``BAT_TRANSLATION_*` 等键,实现编辑 `.env` 后无参启动;优先级为命令行参数 > 进程环境变量 > `.env` > 内置默认值,`BAT_SKIP_ENV_FILE=1` 可整体禁用;Redis 键为预留。daemon 任务历史持久化在 `<state-dir>/bat-tasks.json`(版本化、`0600` 原子写),重启后任务经 `task.*` 仍可查,中断任务标记 `task_interrupted``BAT-ERR-700005`)。
15. issue 43 已补齐 Rust `bat``res` / `parse` / `i18n` 工作流入口:支持资源拉取、解析刷新、可再生解析缓存清理、离线翻译工作台、人工文本查看/修改/清空、工作台发布前校验、有限 TextAsset 汉化发布、人工校对状态标记、既有 patch 能力的批量重打包、单次/限定次数/周期执行和版本化 schedule CRUD。schedule 查询现在按一级工作流过滤,删除/执行会校验作用域,单轮执行可限制计划数;schedule CRUD、翻译任务状态回写`translation.proofread` 状态标记已通过 `bat.sock` 的 RPC 以及 `bat-api` 的鉴权管理接口暴露,dashboard 不维护第二套状态。该例外只编排已有解析和 patch 能力,不扩大解析器覆盖;完整 AssetBundle 重打包、真实 provider worker 和 Web 前端仍是后续工作。G-008(产品级 Go 同步 CLI)已决策关闭。真实官方网络全量拉取 smoke 已固化(G-018 已关闭);真实大文件与运行报告默认在 `/tmp` 隔离目录,不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md` 15. issue 43 已补齐 Rust `bat``res` / `parse` / `i18n` 工作流入口:支持资源拉取、解析刷新、可再生解析缓存清理、离线翻译工作台、人工文本查看/修改/清空、工作台发布前校验、有限 TextAsset 汉化发布、人工校对状态标记、既有 patch 能力的批量重打包、单次/限定次数/周期执行和版本化 schedule CRUD。issue 44 已接入 `translation.worker.run` provider worker:默认并发 8、范围 `1..=256`,每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果。schedule 查询现在按一级工作流过滤,删除/执行会校验作用域,单轮执行可限制计划数;schedule CRUD、翻译任务查询/交接视图、翻译任务状态回写、provider worker 触发`translation.proofread` 状态标记已通过 `bat.sock` 的 RPC 以及 `bat-api` 的鉴权管理接口暴露,dashboard 不维护第二套状态。该例外只编排已有解析和 patch 能力,不扩大解析器覆盖;完整 AssetBundle 重打包和 Web 前端仍是后续工作。G-008(产品级 Go 同步 CLI)已决策关闭。真实官方网络全量拉取 smoke 已固化(G-018 已关闭);真实大文件与运行报告默认在 `/tmp` 隔离目录,不纳入 Git。Go 细节见 `docs/reports/GO_STATUS.md`
当前翻译交接还包括 `translation-tasks.sqlite` 和版本化 `translation-handoff.json` 当前翻译交接还包括 `translation-tasks.sqlite` 和版本化 `translation-handoff.json`
`translation.tasks` 查询单项 worker 状态,`translation.handoff` 查询完整 `translation.tasks` 查询单项 worker 状态,`translation.handoff` 查询完整
@@ -146,7 +146,7 @@ job/unit/provider run 状态。当前下载实现使用默认 8 个独立 worker
待完成: 待完成:
- `crowdin-textunit-queue.json` 接入真实 Crowdin/provider worker、翻译记忆 Patch 构建;围绕 `translation.task.update` 完成真实 worker 集成与失败原因落库验证,并继续扩展 CAS 诊断查询面。 - 基于已接入的 `translation.worker.run` 继续推进翻译记忆、完整 Patch 构建/rollback继续扩展 CAS 诊断查询面。
- 真实线上全量下载 smoke 已固化为 `scripts/official-full-pull-smoke.sh``make official-smoke`;实际运行报告由脚本写入隔离输出目录。 - 真实线上全量下载 smoke 已固化为 `scripts/official-full-pull-smoke.sh``make official-smoke`;实际运行报告由脚本写入隔离输出目录。
- 增加更多权限和极端文件系统场景测试。 - 增加更多权限和极端文件系统场景测试。
@@ -271,7 +271,7 @@ cargo run -p bat-infrastructure --bin bat -- \
--watch --watch
``` ```
资源 HTTP bootstrap / 只读分发入口是 Go `cmd/bat-api`。生产拓扑下它与 Rust `bat` 同环境运行,经 `bat.sock` RPC 获取当前 `resource_root`,不在配置里写死资源目录;本地开发不能全量跑 `bat` 时用 fixture 和 Go 门禁验证。`internal/api/testdata/contract/` 已固化来自 Rust 输出并经归一化的 `catalog.status``resource.manifest``official-sync-snapshot.json` contract fixtureGo mirror 测试会防止字段名、null 语义和 `game_main_config_bootstrap` 再次漂移。`bat-api` 已补 launcher 资源引导兼容端点、玩家-facing HTTP 控制面和鉴权调度管理接口(token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单;`reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh``schedule.*` 可经 Web 转发),响应只来自已发布 snapshot/RPC,不提供登录、网关、鉴权或完整 package update manifest。 资源 HTTP bootstrap / 只读分发入口是 Go `cmd/bat-api`。生产拓扑下它与 Rust `bat` 同环境运行,经 `bat.sock` RPC 获取当前 `resource_root`,不在配置里写死资源目录;本地开发不能全量跑 `bat` 时用 fixture 和 Go 门禁验证。`internal/api/testdata/contract/` 已固化来自 Rust 输出并经归一化的 `catalog.status``resource.manifest``official-sync-snapshot.json` contract fixtureGo mirror 测试会防止字段名、null 语义和 `game_main_config_bootstrap` 再次漂移。`bat-api` 已补 launcher 资源引导兼容端点、玩家-facing HTTP 控制面和鉴权调度/translation 管理接口(token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI、管理控制白名单;`reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh``schedule.*``translation.tasks` / `translation.handoff` 查询、`translation.task.update``translation.worker.run``translation.proofread` 可经 Web 转发),响应只来自已发布 snapshot/RPC,不提供登录、网关、鉴权或完整 package update manifest。
生产要求: 生产要求:
@@ -292,8 +292,8 @@ GitHub issue 状态:#1 已关闭;#17 的历史决定不代表当前下载实
下一阶段必须优先完成: 下一阶段必须优先完成:
1. Issue #1P0,主体已实现):`bat.sock` Unix socket JSON-RPC 已扩展为面向 Go 服务层的 Rust Resource Backend API。统一 envelope`ok``status``error``data``request_id`)与 `BAT-ERR` 错误码模型已落地;`daemon.*`status/logs/stop/restart/reload/refresh/doctor)、`resource.*`state/sync/verify/repair/manifest/list/index)、`schedule.*`list/add/update/remove/run)、`parse.*`status/text_units/errors)、`translation.*`tasks/handoff/task.update/proofread)、`localized.*`status)、`catalog.*`status/refresh/diff/versions)、`task.*`status/list/cancel/logs)、文件级 `patch.apply``unityfs.patch_text_asset` / `unityfs.patch_string_field` / `unityfs.patch_field` 已实现,长任务返回 `task_id` 可轮询(任务执行器单 worker FIFO,与 watch 循环互斥;任务历史持久化于 `<state-dir>/bat-tasks.json`,daemon 重启后仍可查,中断任务标记 `task_interrupted`);错误码已接入下载、launcher/metadata、server-info/marker 与配置校验路径。剩余:发布级 `patch build` / `patch rollback`、复杂 `unityfs.*` 语义编辑、`task.create`(按设计由语义方法创建)、`daemon.clean-stable`(由 CLI 侧按进程生命周期显式执行,live RPC 内不做在线清理)、Redis 任务后端(`.env` 已预留配置键,接入时机另议)。Go 层通过 RPC 调用 Rust backend,不走 FFIFFI 降级说明见 `docs/architecture/official-resource-backend.md` §7)。 1. Issue #1P0,主体已实现):`bat.sock` Unix socket JSON-RPC 已扩展为面向 Go 服务层的 Rust Resource Backend API。统一 envelope`ok``status``error``data``request_id`)与 `BAT-ERR` 错误码模型已落地;`daemon.*`status/logs/stop/restart/reload/refresh/doctor)、`resource.*`state/sync/verify/repair/manifest/list/index)、`schedule.*`list/add/update/remove/run)、`parse.*`status/text_units/errors)、`translation.*`tasks/handoff/task.update/proofread)、`localized.*`status)、`catalog.*`status/refresh/diff/versions)、`task.*`status/list/cancel/logs)、文件级 `patch.apply``unityfs.patch_text_asset` / `unityfs.patch_string_field` / `unityfs.patch_field` 已实现,长任务返回 `task_id` 可轮询(任务执行器单 worker FIFO,与 watch 循环互斥;任务历史持久化于 `<state-dir>/bat-tasks.json`,daemon 重启后仍可查,中断任务标记 `task_interrupted`);错误码已接入下载、launcher/metadata、server-info/marker 与配置校验路径。剩余:发布级 `patch build` / `patch rollback`、复杂 `unityfs.*` 语义编辑、`task.create`(按设计由语义方法创建)、`daemon.clean-stable`(由 CLI 侧按进程生命周期显式执行,live RPC 内不做在线清理)、Redis 任务后端(`.env` 已预留配置键,接入时机另议)。Go 层通过 RPC 调用 Rust backend,不走 FFIFFI 降级说明见 `docs/architecture/official-resource-backend.md` §7)。
2. Go 侧:进度见 `docs/reports/GO_STATUS.md`。G-008 已关闭;`bat-api` 资源 bootstrap/分发 MVP 已落地,已含 `/v1/bootstrap``/v1/launcher/bootstrap`、launcher 资源 metadata 兼容、HTTP 鉴权/限流/日志/反代适配、动态 JSON no-store、OpenAPI、管理控制白名单、CDN Range/缓存头、RPC 周期刷新和 USERGUIDE 基础章节;仓库内 Rust/Go snapshot contract fixture 与同机 live smoke 已落地。失效 `resource_root` 或 manifest 不完整时会清空旧索引并使 `/readyz` 返回 `503`;编码 dot-segment 会在路由规范化前拒绝。refresh mtime/size 增量缓存和可选持久化另议。 2. Go 侧:进度见 `docs/reports/GO_STATUS.md`。G-008 已关闭;`bat-api` 资源 bootstrap/分发 MVP 已落地,已含 `/v1/bootstrap``/v1/launcher/bootstrap`、launcher 资源 metadata 兼容、HTTP 鉴权/限流/日志/反代适配、动态 JSON no-store、OpenAPI、管理控制白名单、translation admin 查询/控制、CDN Range/缓存头、RPC 周期刷新和 USERGUIDE 基础章节;仓库内 Rust/Go snapshot contract fixture 与同机 live smoke 已落地。失效 `resource_root` 或 manifest 不完整时会清空旧索引并使 `/readyz` 返回 `503`;编码 dot-segment 会在路由规范化前拒绝。refresh mtime/size 增量缓存和可选持久化另议。
3. 文本提取 / 翻译队列 / Patch 输入:`official-textunit-index.json``official-textunit-tasks.json``crowdin-textunit-queue.json` 已生成;TextUnit 明细、解析错误和离线翻译任务状态/失败原因已可查询,`translation.task.update` 已提供 Rust CLI、Go typed helper 和 bat-api dashboard 状态回写入口,`translation.proofread` 已提供人工校对状态标记入口。剩余为真实 Crowdin/provider worker、翻译记忆 Patch 构建。 3. 文本提取 / 翻译队列 / Patch 输入:`official-textunit-index.json``official-textunit-tasks.json``crowdin-textunit-queue.json` 已生成;TextUnit 明细、解析错误和离线翻译任务状态/失败原因已可查询,`translation.tasks` / `translation.handoff` 已提供 Go typed helper 和 bat-api dashboard 查询入口,`translation.task.update` 已提供 Rust CLI、Go typed helper 和 bat-api dashboard 状态回写入口,`translation.worker.run` 已提供 mock/Crowdin provider worker、lease、重试和 TextUnit 译文结果落库,`translation.proofread` 已提供人工校对状态标记入口。剩余为翻译记忆、完整 Patch 构建/rollback 和复杂重打包
4. Issue #3(已完成当前目标):AssetBundle UnityFS 基础解析校验已覆盖 header、block、directory、LZ4/LZMA、alignment、总大小、计数、路径安全、重复 directory、边界和隔离 UnityPy 真实样本;复杂对象解析与发布级重打包继续跟踪 G-005。 4. Issue #3(已完成当前目标):AssetBundle UnityFS 基础解析校验已覆盖 header、block、directory、LZ4/LZMA、alignment、总大小、计数、路径安全、重复 directory、边界和隔离 UnityPy 真实样本;复杂对象解析与发布级重打包继续跟踪 G-005。
5. Issue #2(已完成当前目标):Addressables JSON/compact catalog 已提取 bundle provider、name、hash/size/CRC、资源类型和依赖关系,并贯通 ResourceEntry、SQLite 与 resource.index 输出;独立二进制格式仍明确拒绝。 5. Issue #2(已完成当前目标):Addressables JSON/compact catalog 已提取 bundle provider、name、hash/size/CRC、资源类型和依赖关系,并贯通 ResourceEntry、SQLite 与 resource.index 输出;独立二进制格式仍明确拒绝。
6. 通用 Binary/JSON/Text Patch 基础已落地;复杂 AssetBundle 重打包和真实翻译系统仍应后置,UnityFS TextAsset patch 发布前置已具备回归测试。 6. 通用 Binary/JSON/Text Patch 基础已落地;复杂 AssetBundle 重打包和真实翻译系统仍应后置,UnityFS TextAsset patch 发布前置已具备回归测试。
@@ -315,4 +315,4 @@ GitHub issue 状态:#1 已关闭;#17 的历史决定不代表当前下载实
- **当前总体完成度**:不再固定写单一百分比,以各模块状态、`GO_STATUS.md` 和 issue 为准。 - **当前总体完成度**:不再固定写单一百分比,以各模块状态、`GO_STATUS.md` 和 issue 为准。
- **当前基线状态**Rust `bat` 同步闭环可用;Go `bat-api` 资源 bootstrap/分发 MVP + 玩家-facing HTTP 控制面 + launcher 资源引导兼容 + RPC 周期刷新/诊断 + readiness + `backendrpc` 可用;CAS 用户级导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、通用 Binary/JSON/Text Patch 基础和 UnityFS TextAsset patch 发布前置可用;完整 AssetBundle 重打包未完成。 - **当前基线状态**Rust `bat` 同步闭环可用;Go `bat-api` 资源 bootstrap/分发 MVP + 玩家-facing HTTP 控制面 + launcher 资源引导兼容 + RPC 周期刷新/诊断 + readiness + `backendrpc` 可用;CAS 用户级导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、通用 Binary/JSON/Text Patch 基础和 UnityFS TextAsset patch 发布前置可用;完整 AssetBundle 重打包未完成。
- **下一工程里程碑**真实 Crowdin/provider worker / 翻译记忆、复杂 AssetBundle 解析和重打包;`bat-api` 同机 live 联调已完成。 - **下一工程里程碑**翻译记忆、完整 localized Patch 构建/rollback、复杂 AssetBundle 解析和重打包;`bat-api` 同机 live 联调已完成。
+8 -8
View File
@@ -36,7 +36,7 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付
8. `bat-ffi` 已提供 Manifest inspect 和官方 sync plan 的可选无状态粗粒度 JSON C ABI helper。 8. `bat-ffi` 已提供 Manifest inspect 和官方 sync plan 的可选无状态粗粒度 JSON C ABI helper。
9. 官方原版资源默认发布到 `./bat-resources`,汉化产物默认发布到独立的 `./bat-localized`;当前官方同步报告会标记 `localized_release_status=not_localized`,表示原版资源已发布、汉化资源未发布;`translation.proofread` 可把汉化 workflow 标记为人工校对中,但不会覆盖已发布的汉化 release。 9. 官方原版资源默认发布到 `./bat-resources`,汉化产物默认发布到独立的 `./bat-localized`;当前官方同步报告会标记 `localized_release_status=not_localized`,表示原版资源已发布、汉化资源未发布;`translation.proofread` 可把汉化 workflow 标记为人工校对中,但不会覆盖已发布的汉化 release。
10. 官方同步校验完成并发布新 release 后会生成 `official-resource-changes.json``crowdin-translation-handoff.json``official-parse-cache.json``official-textunit-index.json``official-textunit-tasks.json``crowdin-textunit-queue.json`,用 Added/Modified 资源驱动后续解析/翻译增量;up-to-date 轮询在已有有效缓存、TextUnit 明细索引和队列时只读取摘要,不重复解析。 10. 官方同步校验完成并发布新 release 后会生成 `official-resource-changes.json``crowdin-translation-handoff.json``official-parse-cache.json``official-textunit-index.json``official-textunit-tasks.json``crowdin-textunit-queue.json`,用 Added/Modified 资源驱动后续解析/翻译增量;up-to-date 轮询在已有有效缓存、TextUnit 明细索引和队列时只读取摘要,不重复解析。
11. issue 43 已补齐 Rust `bat``res` / `parse` / `i18n` 工作流:单次/限定次数/周期执行、版本化 schedule CRUD 与作用域过滤、解析缓存清理、翻译工作台校验、离线翻译工作台、人工文本查看/修改/清空、翻译任务状态回写、人工校对状态标记、既有 patch 能力的批量重打包和独立汉化 release 发布;schedule CRUD、`translation.task.update``translation.proofread` 已经通过 `bat.sock``bat-api` 管理接口暴露,dashboard 不维护第二套状态;该能力不扩大冻结期解析器覆盖。 11. issue 43 已补齐 Rust `bat``res` / `parse` / `i18n` 工作流:单次/限定次数/周期执行、版本化 schedule CRUD 与作用域过滤、解析缓存清理、翻译工作台校验、离线翻译工作台、人工文本查看/修改/清空、翻译任务状态回写、人工校对状态标记、既有 patch 能力的批量重打包和独立汉化 release 发布;issue 44 已接入 `translation.worker.run` provider worker,默认并发 8、范围 `1..=256`,每个 worker 独立 claim 下一项任务并落库 lease、失败分类、重试计划和 TextUnit 译文结果;schedule CRUD、`translation.tasks` / `translation.handoff``translation.task.update``translation.worker.run``translation.proofread` 已经通过 `bat.sock``bat-api` 管理接口暴露,dashboard 不维护第二套状态;该能力不扩大冻结期解析器覆盖。
12. 文档已整理:根目录保留入口文档,历史报告进入 `docs/reports/historical/`,误嵌套的 `docs/docs` 已合并。 12. 文档已整理:根目录保留入口文档,历史报告进入 `docs/reports/historical/`,误嵌套的 `docs/docs` 已合并。
### 仍是骨架或占位 ### 仍是骨架或占位
@@ -45,8 +45,8 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付
2. `bat-patch` 已具备确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、通用 Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据;文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放,当前发布级可用的是 `bat-assetbundle` + `LocalizedPatchService` 的 UnityFS TextAsset patch 前置链路。 2. `bat-patch` 已具备确定性 Binary hunk diff/apply、RFC 6902 JSON Patch apply、UTF-8 Text Patch、通用 Patch manifest、BLAKE3/size 完整性校验和 rollback 元数据;文件级 `patch.apply` RPC / `patch-apply` CLI 与 UnityFS TextAsset / TypeTree string / TypeTree 语义字段写入入口已开放,当前发布级可用的是 `bat-assetbundle` + `LocalizedPatchService` 的 UnityFS TextAsset patch 前置链路。
3. Go 侧边界已冻结(见 `docs/reports/GO_STATUS.md`):同步/运维命令行 = Rust `bat`;资源分发 = `cmd/bat-api` MVP`internal/backendrpc` 完成;`cmd/bat` 仅为试验(`bin/bat-go`)。完整游戏业务 API / Web / SDK 仍未完成。 3. Go 侧边界已冻结(见 `docs/reports/GO_STATUS.md`):同步/运维命令行 = Rust `bat`;资源分发 = `cmd/bat-api` MVP`internal/backendrpc` 完成;`cmd/bat` 仅为试验(`bin/bat-go`)。完整游戏业务 API / Web / SDK 仍未完成。
4. Addressables parser 已覆盖当前真实形态 fixture/golden,但还不是完整 Unity Addressables/SBP catalog 兼容层。 4. Addressables parser 已覆盖当前真实形态 fixture/golden,但还不是完整 Unity Addressables/SBP catalog 兼容层。
5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository,并通过 `resource.index` RPC/CLI 查询;Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要,资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format;单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`,可通过 `parse.text_units` / `parse.errors` 查询;离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询,`translation.task.update` 已提供 worker 状态回写 contract`translation.proofread` 已提供汉化 workflow 人工校对标记 contract,真实 provider worker 集成仍需继续推进 5. 官方同步结果可配置为发布后自动导入 CAS + ResourceRepository,并通过 `resource.index` RPC/CLI 查询;Resource metadata 已保存 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 摘要,资源级查询已覆盖 release、平台、destination、archive entry、parse status 和 TextUnit format;单条 TextUnit 明细和解析错误已持久化到 `official-textunit-index.json`,可通过 `parse.text_units` / `parse.errors` 查询;离线 TextUnit 翻译任务状态和跳过/失败原因可通过 `translation.tasks` 查询,`translation.task.update` 已提供 worker 状态回写 contract`translation.worker.run` 已提供真实 provider worker 触发、lease/retry 和结果落库 contract`translation.proofread` 已提供汉化 workflow 人工校对标记 contract。
6. 汉化 Patch 发布前置已具备 UnityFS TextAsset manifest/apply/diff/rollback/完整性校验和 `localized.status` 严格校验;真实 Crowdin worker翻译记忆到完整汉化文件集合的构建仍未完成。 6. 汉化 Patch 发布前置已具备 UnityFS TextAsset manifest/apply/diff/rollback/完整性校验和 `localized.status` 严格校验;真实 provider worker 已接入,翻译记忆到完整汉化文件集合的构建仍未完成。
7. 真实官方网络全量下载 smoke 已固化为可重复脚本和 runbook(G-018 已关闭);真实运行记录处于长期运行测试阶段,报告待后续提供。 7. 真实官方网络全量下载 smoke 已固化为可重复脚本和 runbook(G-018 已关闭);真实运行记录处于长期运行测试阶段,报告待后续提供。
8. 完整 Web、数据库迁移、插件加载机制尚未实现;`bat-api` 资源 bootstrap/分发 OpenAPI 已通过 `/openapi.yaml` 提供,完整游戏业务 API 的 OpenAPI 仍未完成。 8. 完整 Web、数据库迁移、插件加载机制尚未实现;`bat-api` 资源 bootstrap/分发 OpenAPI 已通过 `/openapi.yaml` 提供,完整游戏业务 API 的 OpenAPI 仍未完成。
9. 原 Git 历史未恢复;当前仓库以新初始化基线为准。 9. 原 Git 历史未恢复;当前仓库以新初始化基线为准。
@@ -179,7 +179,7 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付
4. Rust 自动更新入口:**已完成当前生产入口**。`bat` 支持 snapshot、marker diff、bootstrap cache、one-shot、`--watch``--daemon`、默认 1 小时间隔、北京时间固定强制刷新,以及 Unix socket JSON-RPC 后台运维命令返回。 4. Rust 自动更新入口:**已完成当前生产入口**。`bat` 支持 snapshot、marker diff、bootstrap cache、one-shot、`--watch``--daemon`、默认 1 小时间隔、北京时间固定强制刷新,以及 Unix socket JSON-RPC 后台运维命令返回。
5. Go 入口边界:**已冻结**。同步命令行 = Rust `bat`G-008 关闭);资源分发 = `bat-api` MVP(G-009 部分完成)。详见 `docs/reports/GO_STATUS.md` 5. Go 入口边界:**已冻结**。同步命令行 = Rust `bat`G-008 关闭);资源分发 = `bat-api` MVP(G-009 部分完成)。详见 `docs/reports/GO_STATUS.md`
6. 用户级 `sync``manifest inspect``cache status`**未完成**。Rust `bat --json` 是当前稳定进程边界;`bat-ffi` 只提供可选兼容用的 Manifest inspect 和 sync plan JSON helper。 6. 用户级 `sync``manifest inspect``cache status`**未完成**。Rust `bat --json` 是当前稳定进程边界;`bat-ffi` 只提供可选兼容用的 Manifest inspect 和 sync plan JSON helper。
7. 下载结果写入 CAS + ResourceRepository**部分完成**。CAS 和 SQLite ResourceRepository 已存在,官方同步入口可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后导入;`resource.index` 可按资源级 release、平台、destination、archive entry、parse status 和 TextUnit format 查询现有索引和资源 metadata;`parse.text_units` / `parse.errors` 可查询当前 release 的 TextUnit 明细与解析错误;`translation.tasks` 可查询离线 TextUnit 翻译任务状态跳过/失败原因。剩余工作是真实 provider worker 集成、CAS 诊断入口和面向大索引的查询优化。 7. 下载结果写入 CAS + ResourceRepository**部分完成**。CAS 和 SQLite ResourceRepository 已存在,官方同步入口可用 `--import-repository` / `BAT_IMPORT_REPOSITORY=1` 在已校验 release 发布后导入;`resource.index` 可按资源级 release、平台、destination、archive entry、parse status 和 TextUnit format 查询现有索引和资源 metadata;`parse.text_units` / `parse.errors` 可查询当前 release 的 TextUnit 明细与解析错误;`translation.tasks` 可查询离线 TextUnit 翻译任务状态跳过/失败原因和 worker 结果。剩余工作是 CAS 诊断入口和面向大索引的查询优化。
8. Linux 生产同步不依赖已安装官方启动器:**已完成当前 Rust 入口**。`--auto-discover` 只使用官方 HTTP metadata 和临时目录解析 `GameMainConfig` 8. Linux 生产同步不依赖已安装官方启动器:**已完成当前 Rust 入口**。`--auto-discover` 只使用官方 HTTP metadata 和临时目录解析 `GameMainConfig`
9. 真实官方网络全量下载 smoke test:**命令已固化(G-018 已关闭)**。`scripts/official-full-pull-smoke.sh` / `make official-smoke` 已固化 dry-run、首次下载、二次 up-to-date 和本地损坏 repair 的可重复流程;真实运行处于长期运行测试阶段,报告待后续提供。 9. 真实官方网络全量下载 smoke test:**命令已固化(G-018 已关闭)**。`scripts/official-full-pull-smoke.sh` / `make official-smoke` 已固化 dry-run、首次下载、二次 up-to-date 和本地损坏 repair 的可重复流程;真实运行处于长期运行测试阶段,报告待后续提供。
10. 官方发布后的增量 handoff 与解析缓存:**已完成基础入口**。新 release 发布后先生成 `official-resource-changes.json``crowdin-translation-handoff.json`,新增+变更资源进入解析/翻译候选;`official-parse-cache.json` 基于下载 manifest 覆盖直接 UnityFS bundle、zip 内 UnityFS 条目和非候选资源记录;随后生成 `official-textunit-index.json``official-textunit-tasks.json``crowdin-textunit-queue.json`,本地文件未变化且缓存/索引有效时跳过重复解析。 10. 官方发布后的增量 handoff 与解析缓存:**已完成基础入口**。新 release 发布后先生成 `official-resource-changes.json``crowdin-translation-handoff.json`,新增+变更资源进入解析/翻译候选;`official-parse-cache.json` 基于下载 manifest 覆盖直接 UnityFS bundle、zip 内 UnityFS 条目和非候选资源记录;随后生成 `official-textunit-index.json``official-textunit-tasks.json``crowdin-textunit-queue.json`,本地文件未变化且缓存/索引有效时跳过重复解析。
@@ -404,8 +404,8 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付
2. G-008 已决策关闭:同步/运维命令行 = 近乎全自动的 Rust `bat` 2. G-008 已决策关闭:同步/运维命令行 = 近乎全自动的 Rust `bat`
3. G-009 / issue #19`bat-api` 资源分发和同机 `bat.sock` live 联调已完成;拉取仍在 Rust `bat` 3. G-009 / issue #19`bat-api` 资源分发和同机 `bat.sock` live 联调已完成;拉取仍在 Rust `bat`
4. 继续 Addressables 结构变体与 UnityFS 复杂对象能力(issue #3 / G-005 的后续阶段)。 4. 继续 Addressables 结构变体与 UnityFS 复杂对象能力(issue #3 / G-005 的后续阶段)。
5. `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 5. 基于 `translation.worker.run` 继续推进翻译记忆和 Patch 构建。
6. 继续扩展 G-011 剩余查询面:真实 provider worker 集成与状态落库验证、`doctor cas` 诊断入口和面向大索引的查询优化。 6. 继续扩展 G-011 剩余查询面:`doctor cas` 诊断入口和面向大索引的查询优化。
--- ---
@@ -459,9 +459,9 @@ BlueArchiveToolkit 不是一次性脚本,也不是演示项目。最终交付
按最终目标计算,当前总体完成度不再固定写单一百分比,以模块状态和 issue 收敛情况为准。 按最终目标计算,当前总体完成度不再固定写单一百分比,以模块状态和 issue 收敛情况为准。
已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、通用 Binary/JSON/Text Patch 基础、UnityFS TextAsset patch 发布前置,以及 Go `bat-api` 资源分发和同机 live 联调。下一阶段的关键是真实 Crowdin worker、翻译记忆、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。 已完成的是稳定基线、架构骨架、部分接口、CAS V1、Rust 官方资源同步闭环、可配置 CAS/ResourceRepository 导入、TextUnit 明细索引/查询、增量 Crowdin 离线队列、provider worker、通用 Binary/JSON/Text Patch 基础、UnityFS TextAsset patch 发布前置,以及 Go `bat-api` 资源分发和同机 live 联调。下一阶段的关键是翻译记忆、完整 localized Patch 构建/rollback、复杂 AssetBundle 解析/重打包和官方资源长期运行报告。
--- ---
- **下一份应更新文档**:真实官方网络 smoke 记录 - **下一份应更新文档**:真实官方网络 smoke 记录
- **下一项工程任务**:推进真实 Crowdin worker / 翻译记忆、复杂 AssetBundle 解析,并持续执行官方资源长期运行 smoke。 - **下一项工程任务**:推进翻译记忆、完整 localized Patch 构建/rollback、复杂 AssetBundle 解析,并持续执行官方资源长期运行 smoke。
+2 -2
View File
@@ -15,7 +15,7 @@
- `bat-infrastructure` CAS 适配层、SQLite Resource Repository、资源导入服务、官方资源 pull/update 服务。 - `bat-infrastructure` CAS 适配层、SQLite Resource Repository、资源导入服务、官方资源 pull/update 服务。
- `bat`:官方资源自动发现、全量拉取、原子发布到 `current -> versions/<id>`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、默认并发 8(可配置 `1..=256`report 按 plan 顺序、进度按完成数单调上报)、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json``--watch` 常驻更新、`--daemon` 后台运行,以及 Unix socket JSON-RPC live control/backend 方法(`daemon.*``resource.*``parse.*``translation.tasks/handoff/task.update``localized.status``catalog.*``task.*``patch.apply``unityfs.patch_*`)。 - `bat`:官方资源自动发现、全量拉取、原子发布到 `current -> versions/<id>`、本地 manifest audit/repair、`.part` 断点续传、403/404/5xx 分类重试、指数退避、默认并发 8(可配置 `1..=256`report 按 plan 顺序、进度按完成数单调上报)、下载 quarantine 诊断、ZIP 结构校验、官方 seed `.hash` 校验、snapshot/cache、版本化 `official-launcher-bootstrap.json``--watch` 常驻更新、`--daemon` 后台运行,以及 Unix socket JSON-RPC live control/backend 方法(`daemon.*``resource.*``parse.*``translation.tasks/handoff/task.update``localized.status``catalog.*``task.*``patch.apply``unityfs.patch_*`)。
- `internal/backendrpc`Go 侧 typed Unix socket JSON-RPC client,是 `bat-api` 调用 Rust daemon 的默认路径。 - `internal/backendrpc`Go 侧 typed Unix socket JSON-RPC client,是 `bat-api` 调用 Rust daemon 的默认路径。
- `cmd/bat-api`:资源 bootstrap + 分发 HTTP MVPissue #19 / G-009);`/v1/bootstrap``/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口,launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导,`/healthz` 暴露 RPC refresh 诊断,`/readyz` 做 release readinessCDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头;玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI管理控制白名单;`.env` 配置端口/RPC socket/刷新周期;生产资源根来自 RPC,不负责自动拉取。 - `cmd/bat-api`:资源 bootstrap + 分发 HTTP MVPissue #19 / G-009);`/v1/bootstrap``/v1/launcher/bootstrap` 组织 `bat` 已发布 release 的启动前资源入口,launcher 形状兼容端点仅输出资源 metadata / GameMainConfig 引导,`/healthz` 暴露 RPC refresh 诊断,`/readyz` 做 release readinessCDN path 支持 `GET`/`HEAD`/`Range`、ETag、Last-Modified 和缓存头;玩家-facing 控制面已具备 token 鉴权、限流、访问日志、反代 IP 适配、动态 JSON no-store、OpenAPI管理控制白名单和 translation admin 查询/控制入口`.env` 配置端口/RPC socket/刷新周期;生产资源根来自 RPC,不负责自动拉取。
- Go 边界权威说明:[`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)G-008 已关闭:同步 CLI = Rust `bat`)。 - Go 边界权威说明:[`docs/reports/GO_STATUS.md`](docs/reports/GO_STATUS.md)G-008 已关闭:同步 CLI = Rust `bat`)。
- 官方同步会维护 `<output>/official-version-state.json`,明确记录当前已完成版本、正在拉取版本、上一个可用版本和失败版本。 - 官方同步会维护 `<output>/official-version-state.json`,明确记录当前已完成版本、正在拉取版本、上一个可用版本和失败版本。
- 资源导入链路可配置为在官方 release 发布后写入 CAS + `ResourceRepository`,资源 metadata 会记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式,TextAsset/Table/Media 会按类型分类索引;`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、archive entry、parse status 和 TextUnit format 分页查询索引。 - 资源导入链路可配置为在官方 release 发布后写入 CAS + `ResourceRepository`,资源 metadata 会记录 release、平台、bundle path、parse status、TextAsset 名称和 TextUnit 数量/格式,TextAsset/Table/Media 会按类型分类索引;`resource.index` RPC/CLI 可按类型、hash、路径模式、官方 release ID、平台、destination、archive entry、parse status 和 TextUnit format 分页查询索引。
@@ -186,7 +186,7 @@ BlueArchiveToolkit/
1. 维护并联调 Go `bat-api` 资源 bootstrap/分发入口;`cmd/bat` 仅有 `doctor``manifest inspect``sync plan` 试验能力,不应误写成完整产品 CLI。 1. 维护并联调 Go `bat-api` 资源 bootstrap/分发入口;`cmd/bat` 仅有 `doctor``manifest inspect``sync plan` 试验能力,不应误写成完整产品 CLI。
2. 补齐 AssetBundle UnityFS 引擎级解析。 2. 补齐 AssetBundle UnityFS 引擎级解析。
3. 扩展 Addressables catalog 解析覆盖,继续用真实形态 fixture/golden 锁定行为。 3. 扩展 Addressables catalog 解析覆盖,继续用真实形态 fixture/golden 锁定行为。
4. `official-textunit-tasks.json` / `crowdin-textunit-queue.json` 接入真实 Crowdin worker翻译记忆 Patch 构建。 4. 基于 `translation.worker.run` provider worker 推进翻译记忆、完整 Patch 构建和发布/回滚闭环
5. 按 smoke runbook 在具备网络和磁盘窗口的环境中执行真实官方全量拉取,并保留本地报告。 5. 按 smoke runbook 在具备网络和磁盘窗口的环境中执行真实官方全量拉取,并保留本地报告。
不建议在 Go 产品入口、资源解析和文本提取基础能力完成前优先开发 Web UI。 不建议在 Go 产品入口、资源解析和文本提取基础能力完成前优先开发 Web UI。
+7 -1
View File
@@ -58,6 +58,7 @@ HTTPS_PROXY=http://user:pass@127.0.0.1:7890 bat --auto-discover --daemon
| `i18n handoff` | 查询当前翻译交接视图 | | `i18n handoff` | 查询当前翻译交接视图 |
| `i18n status` | 显示当前汉化 release 状态 | | `i18n status` | 显示当前汉化 release 状态 |
| `i18n task update` | 回写 provider worker 任务状态 | | `i18n task update` | 回写 provider worker 任务状态 |
| `i18n worker run` | 运行真实 provider worker;支持单次、限定次数和周期执行 |
| `i18n publish` | 校验工作台并发布独立汉化 release;`--force` 使用新的手动 release ID | | `i18n publish` | 校验工作台并发布独立汉化 release;`--force` 使用新的手动 release ID |
| `i18n schedule` | 管理翻译和汉化发布计划 | | `i18n schedule` | 管理翻译和汉化发布计划 |
| `refresh` | 执行一次更新检查;若有 live daemon,则通过 RPC 请求其刷新 | | `refresh` | 执行一次更新检查;若有 live daemon,则通过 RPC 请求其刷新 |
@@ -137,13 +138,15 @@ BAT_API_SKIP_ENV_FILE=1 go run ./cmd/bat-api \
| `GET /openapi.yaml` | bat-api OpenAPI 文档 | | `GET /openapi.yaml` | bat-api OpenAPI 文档 |
| `GET /admin/` | 管理控制入口与允许操作列表 | | `GET /admin/` | 管理控制入口与允许操作列表 |
| `GET /admin/schedules?id=...&group=...&enabled=...` | 读取/过滤 Rust `bat` 调度计划;需要管理 token | | `GET /admin/schedules?id=...&group=...&enabled=...` | 读取/过滤 Rust `bat` 调度计划;需要管理 token |
| `GET /admin/translation/tasks?limit=100&worker_status=failed` | 读取/过滤 Rust 翻译任务和 provider worker 状态;需要管理 token |
| `GET /admin/translation/handoff` | 读取当前 release 的完整翻译交接视图;需要管理 token |
| `POST /admin/control/{action}` | 经白名单转发 Rust `bat` 控制请求;见下文 | | `POST /admin/control/{action}` | 经白名单转发 Rust `bat` 控制请求;见下文 |
launcher 兼容端点只服务启动前资源发现。它们复用 Rust `bat` snapshot 中的 `launcher_metadata``game_main_config_bootstrap`,显式标记 `scope=resource_bootstrap_only` / `package_update_manifest=false``bat-api` 不下载 launcher 包,不生成官方 PC package update manifest,也不仿造登录、账号、网关、鉴权或游戏业务协议。 launcher 兼容端点只服务启动前资源发现。它们复用 Rust `bat` snapshot 中的 `launcher_metadata``game_main_config_bootstrap`,显式标记 `scope=resource_bootstrap_only` / `package_update_manifest=false``bat-api` 不下载 launcher 包,不生成官方 PC package update manifest,也不仿造登录、账号、网关、鉴权或游戏业务协议。
生产面对玩家分发时,应启用 HTTP token 鉴权、限流和访问日志: 生产面对玩家分发时,应启用 HTTP token 鉴权、限流和访问日志:
- `BAT_API_AUTH_TOKEN`:启用 `Authorization: Bearer <token>``X-BAT-Token` 或 query fallback 鉴权;token 推荐由 secret manager 或进程环境提供,不建议写入提交文件。`/admin/control/*``/admin/schedules` 需要此 token。 - `BAT_API_AUTH_TOKEN`:启用 `Authorization: Bearer <token>``X-BAT-Token` 或 query fallback 鉴权;token 推荐由 secret manager 或进程环境提供,不建议写入提交文件。`/admin/control/*``/admin/schedules``/admin/translation/*` 需要此 token。
- `BAT_API_AUTH_QUERY_PARAM`query fallback 参数名,默认 `bat_token`;兼容不能写 header 的客户端,访问日志不会记录 query。 - `BAT_API_AUTH_QUERY_PARAM`query fallback 参数名,默认 `bat_token`;兼容不能写 header 的客户端,访问日志不会记录 query。
- `BAT_API_AUTH_EXEMPT_PATHS`:逗号分隔的免鉴权 path 或 slash-prefix,例如 `/healthz,/readyz` - `BAT_API_AUTH_EXEMPT_PATHS`:逗号分隔的免鉴权 path 或 slash-prefix,例如 `/healthz,/readyz`
- `BAT_API_RATE_LIMIT_RPS` / `BAT_API_RATE_LIMIT_BURST`:按客户端 IP 的进程内 token bucket 限流;边缘反代/CDN 仍应配置独立限流。 - `BAT_API_RATE_LIMIT_RPS` / `BAT_API_RATE_LIMIT_BURST`:按客户端 IP 的进程内 token bucket 限流;边缘反代/CDN 仍应配置独立限流。
@@ -167,6 +170,8 @@ launcher 兼容端点只服务启动前资源发现。它们复用 Rust `bat` sn
| `schedule-remove` | `schedule.remove` | `{ "id": "..." }` | `202` + Rust schedule report | | `schedule-remove` | `schedule.remove` | `{ "id": "..." }` | `202` + Rust schedule report |
| `schedule-run` | `schedule.run` | 可选 `{ "id": "...", "force": true }` | `202` + 执行报告 | | `schedule-run` | `schedule.run` | 可选 `{ "id": "...", "force": true }` | `202` + 执行报告 |
| `translation-task-update` | `translation.task.update` | `{ "task_id": "...", "status": "...", "failure_reason": "...", "provider_run_id": "..." }` | `202` + 当前任务记录 | | `translation-task-update` | `translation.task.update` | `{ "task_id": "...", "status": "...", "failure_reason": "...", "provider_run_id": "..." }` | `202` + 当前任务记录 |
| `translation-worker-run` | `translation.worker.run` | `{ "provider": "mock", "concurrency": 8, "max_tasks": 2 }` | `202` + worker task |
| `translation-proofread` | `translation.proofread` | 无 | `202` + 汉化状态 |
`stop``clean-stable`、patch 和 UnityFS 写入命令不会经 HTTP 暴露。 `stop``clean-stable`、patch 和 UnityFS 写入命令不会经 HTTP 暴露。
@@ -420,6 +425,7 @@ curl -i -H 'Range: bytes=0-1023' \
| `translation.tasks` | ✅ | 查询离线 TextUnit 翻译任务及 worker 状态 | | `translation.tasks` | ✅ | 查询离线 TextUnit 翻译任务及 worker 状态 |
| `translation.handoff` | ✅ | 查询完整 job/unit/provider run 交接视图 | | `translation.handoff` | ✅ | 查询完整 job/unit/provider run 交接视图 |
| `translation.task.update` | ✅ | 回写当前 release 的 provider worker 状态 | | `translation.task.update` | ✅ | 回写当前 release 的 provider worker 状态 |
| `translation.worker.run` | ✅ | 触发 Rust provider worker,落库 TextUnit 译文结果、lease、失败分类和重试状态 |
| `translation.proofread` | ✅ | 将当前汉化 workflow 标记为人工校对中 | | `translation.proofread` | ✅ | 将当前汉化 workflow 标记为人工校对中 |
| `localized.status` | ✅ | 查询汉化 release 与当前官方 release 的匹配状态 | | `localized.status` | ✅ | 查询汉化 release 与当前官方 release 的匹配状态 |
| `catalog.status` | ✅ | 当前已发布版本的 catalog 概览(app/bundle 版本、addressables 根、端点与 marker 计数、launcher 元数据) | | `catalog.status` | ✅ | 当前已发布版本的 catalog 概览(app/bundle 版本、addressables 根、端点与 marker 计数、launcher 元数据) |
+168 -2
View File
@@ -108,16 +108,114 @@ paths:
responses: responses:
"200": "200":
description: Admin links and allowlisted control actions. description: Admin links and allowlisted control actions.
/admin/schedules:
get:
summary: List Rust-owned resource workflow schedules
parameters:
- name: id
in: query
schema:
type: string
- name: group
in: query
schema:
type: string
enum: [res, parse, i18n]
- name: enabled
in: query
schema:
type: boolean
responses:
"200":
description: Current schedule JSON report.
"401":
description: Missing or invalid admin token.
"503":
description: Rust bat schedule backend is unavailable.
/admin/translation/tasks:
get:
summary: List Rust-owned translation task worker status
parameters:
- name: offset
in: query
schema:
type: integer
minimum: 0
- name: limit
in: query
schema:
type: integer
minimum: 1
maximum: 1000
- name: task_id
in: query
schema:
type: string
- name: release_id
in: query
schema:
type: string
- name: destination
in: query
schema:
type: string
- name: archive_entry
in: query
schema:
type: string
- name: status
in: query
schema:
type: string
- name: worker_status
in: query
schema:
type: string
- name: parse_status
in: query
schema:
type: string
- name: format
in: query
schema:
type: string
- name: has_reason
in: query
schema:
type: boolean
- name: has_failure_reason
in: query
schema:
type: boolean
responses:
"200":
description: Current translation task JSON report from Rust bat.
"400":
description: Invalid translation task query.
"401":
description: Missing or invalid admin token.
"503":
description: Rust bat translation backend is unavailable.
/admin/translation/handoff:
get:
summary: Read Rust-owned translation handoff state
responses:
"200":
description: Current translation handoff JSON report from Rust bat.
"401":
description: Missing or invalid admin token.
"503":
description: Rust bat translation backend is unavailable.
/admin/control/{action}: /admin/control/{action}:
post: post:
summary: Forward an allowlisted control action to Rust bat summary: Forward an allowlisted control or schedule action to Rust bat
parameters: parameters:
- name: action - name: action
in: path in: path
required: true required: true
schema: schema:
type: string type: string
enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh] enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, translation-task-update, translation-worker-run, translation-proofread]
requestBody: requestBody:
required: false required: false
content: content:
@@ -128,6 +226,74 @@ paths:
properties: properties:
force: force:
type: boolean type: boolean
id:
type: string
group:
type: string
action:
type: string
args:
type: array
items:
type: string
next_run_unix_seconds:
type: integer
format: int64
delay_seconds:
type: integer
format: int64
every_seconds:
type: integer
format: int64
count:
type: integer
format: int64
max_runs:
type: integer
format: int64
minimum: 1
clear_args:
type: boolean
clear_every:
type: boolean
enabled:
type: boolean
task_id:
type: string
status:
type: string
failure_reason:
type: string
provider_run_id:
type: string
provider:
type: string
enum: [mock, crowdin]
fixture_path:
type: string
concurrency:
type: integer
format: int64
minimum: 1
maximum: 256
max_attempts:
type: integer
format: int64
minimum: 1
lease_seconds:
type: integer
format: int64
minimum: 1
retry_backoff_seconds:
type: integer
format: int64
minimum: 0
max_tasks:
type: integer
format: int64
minimum: 1
worker_id:
type: string
responses: responses:
"202": "202":
description: Rust bat accepted the control request. description: Rust bat accepted the control request.
+2 -2
View File
@@ -208,6 +208,6 @@ catalog 仍按“明确不支持”处理,不把低保真路径伪装成完整
1. 完成 Addressables Windows/Android 当前版本 catalog 样本集合,关闭 G-007 当前阶段。 1. 完成 Addressables Windows/Android 当前版本 catalog 样本集合,关闭 G-007 当前阶段。
2. 完成 TypeTree 字段 reader 和 MonoBehaviour/ScriptableObject 遍历,推进 G-005。 2. 完成 TypeTree 字段 reader 和 MonoBehaviour/ScriptableObject 遍历,推进 G-005。
3. `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 3. 基于 `translation.worker.run` 推进翻译记忆和 Patch 构建。
4. 翻译任务状态接入 CAS/ResourceRepository 查询面,推进 G-011。 4. 扩展翻译任务结果在 CAS/ResourceRepository 查询面的索引,推进 G-011。
5. 在通用 Binary/JSON/Text Patch 基础上继续扩展复杂 AssetBundle 重打包和发布流程统一,保留当前 UnityFS TextAsset patch 发布前置链路。 5. 在通用 Binary/JSON/Text Patch 基础上继续扩展复杂 AssetBundle 重打包和发布流程统一,保留当前 UnityFS TextAsset patch 发布前置链路。
@@ -179,8 +179,9 @@
`resource.index` RPC / CLI 只读查询现有 SQLite 索引;索引不存在时返回 `resource.index` RPC / CLI 只读查询现有 SQLite 索引;索引不存在时返回
`available=false`,不会因为查询创建空库。发布后的 TextUnit 队列还会在当前 `available=false`,不会因为查询创建空库。发布后的 TextUnit 队列还会在当前
release 根目录写入 `translation-tasks.sqlite`,由版本化 `schema_migrations` release 根目录写入 `translation-tasks.sqlite`,由版本化 `schema_migrations`
管理 queued/running/failed/completed/skipped 和 failure reason。`translation.tasks` 管理 queued/running/failed/completed/skipped、provider run、lease、失败分类、
优先查询这份状态库,`translation.task.update` 供 provider worker 回写状态; 重试计划和 TextUnit 级译文结果。`translation.tasks` 优先查询这份状态库,
`translation.worker.run` 由 Rust worker 回写状态;`translation.task.update` 仍供外部 provider 流程回写状态;
没有状态库的旧 release 才回退到 immutable JSON 队列。G-011 剩余工作是 CAS 没有状态库的旧 release 才回退到 immutable JSON 队列。G-011 剩余工作是 CAS
诊断入口和面向大索引的查询优化。 诊断入口和面向大索引的查询优化。
+2 -2
View File
@@ -92,8 +92,8 @@ CAS V1 和 Rust 官方同步闭环完成后,下一阶段优先推进:
1. 继续联调 Go `bat-api` 与 Rust daemon 的资源分发路径;Go 同步 CLI 不再作为产品目标。 1. 继续联调 Go `bat-api` 与 Rust daemon 的资源分发路径;Go 同步 CLI 不再作为产品目标。
2.`docs/guides/official-full-pull-smoke.md` 在隔离目录执行真实官方网络全量下载 smoke,并保留运行报告。 2.`docs/guides/official-full-pull-smoke.md` 在隔离目录执行真实官方网络全量下载 smoke,并保留运行报告。
3. `crowdin-textunit-queue.json` 接入真实 Crowdin worker、翻译记忆和 Patch 构建。 3. 基于 `translation.worker.run` 推进翻译记忆和 Patch 构建。
4. 扩展 ResourceRepository 查询面:真实 provider worker 集成与状态落库验证、CAS 诊断入口和更丰富 TextUnit 查询。 4. 扩展 ResourceRepository 查询面:CAS 诊断入口、面向大索引的查询优化和更丰富 TextUnit 查询。
5. 继续完善 AssetBundle 复杂对象解析、复杂对象重打包和 Patch 发布流程统一;通用 Binary/JSON/Text Patch 基础与 UnityFS TextAsset patch 发布前置链路已可用。 5. 继续完善 AssetBundle 复杂对象解析、复杂对象重打包和 Patch 发布流程统一;通用 Binary/JSON/Text Patch 基础与 UnityFS TextAsset patch 发布前置链路已可用。
优先阅读: 优先阅读:
+36 -2
View File
@@ -133,9 +133,30 @@ bat i18n publish \
发布只接受当前实现支持的直接 TextAsset 条目;TypeTree 字段和 zip 内 bundle 使用 `parse repack` 的 spec 单独处理。`--force` 不覆盖已有目录,而是生成独立的 `<official-release>-manual-<unix-seconds>` 汉化 release ID;也可以用 `--localized-release-id` 显式指定新 ID。因此强制发布仍保留旧 release 和 rollback 信息。 发布只接受当前实现支持的直接 TextAsset 条目;TypeTree 字段和 zip 内 bundle 使用 `parse repack` 的 spec 单独处理。`--force` 不覆盖已有目录,而是生成独立的 `<official-release>-manual-<unix-seconds>` 汉化 release ID;也可以用 `--localized-release-id` 显式指定新 ID。因此强制发布仍保留旧 release 和 rollback 信息。
当前 `i18n run` 是离线工作流:刷新 TextUnit 队列,并可用 `--translation-file` 导出工作台;不会假装调用未接入的外部 provider 当前 `i18n run` 是离线工作流:刷新 TextUnit 队列,并可用 `--translation-file` 导出工作台;真实 provider 由单独的 worker 命令消费 `translation-tasks.sqlite`
provider worker 消费 handoff 后,可以用 `i18n task update` 回写当前 release 的任务状态 运行一次 mock provider worker
```bash
bat i18n worker run \
--output /tmp/bat-resources \
--provider mock \
--worker-concurrency 8
```
`--provider` 支持 `mock``crowdin``mock` 可通过 `--translation-fixture`
读取本地 JSON fixture`crowdin` 从环境变量 `CROWDIN_PROJECT_ID`
`CROWDIN_LANGUAGE_ID``CROWDIN_API_TOKEN` 读取配置。worker 默认并发为 8,
范围 `1..=256`;每个 worker 完成当前任务后立即从 SQLite 队列领取下一项,
不会等待当前一批 worker 全部结束后再重新分配。
可用参数包括 `--worker-max-attempts``--worker-lease-seconds`
`--worker-retry-backoff` / `--worker-retry-backoff-seconds`
`--worker-max-tasks``--worker-id`。worker 支持 `--run-count`
`--watch`,因此可以单次、限定次数或周期执行;`--run-count > 1` 时仍必须
显式指定 `--interval`
外部 provider 或人工流程也可以用 `i18n task update` 回写当前 release 的任务状态:
```bash ```bash
bat i18n task update \ bat i18n task update \
@@ -224,11 +245,24 @@ contract`id`、`group`、`action`、`args`、`next_run_unix_seconds`、
`enabled``schedule.list` 额外接受 `id``group``enabled` 过滤, `enabled``schedule.list` 额外接受 `id``group``enabled` 过滤,
`schedule.run` 额外接受 `group``force``max_runs` `schedule.run` 额外接受 `group``force``max_runs`
翻译任务状态可由已鉴权的 dashboard 通过 `GET /admin/translation/tasks`
查询,query 过滤项包括 `offset``limit``task_id``release_id`
`destination``path_pattern``archive_entry``status``worker_status`
`parse_status``format``has_reason``has_failure_reason`。完整 provider
run 交接视图通过 `GET /admin/translation/handoff` 查询。两个查询接口都只转发
Rust `translation.tasks` / `translation.handoff`,不在 Go 侧维护状态。
翻译任务状态也可由已鉴权的 dashboard 通过 翻译任务状态也可由已鉴权的 dashboard 通过
`POST /admin/control/translation-task-update` 回写,请求字段为 `POST /admin/control/translation-task-update` 回写,请求字段为
`task_id``status`,以及可选的 `failure_reason``provider_run_id` `task_id``status`,以及可选的 `failure_reason``provider_run_id`
该接口只转发 `translation.task.update` 该接口只转发 `translation.task.update`
`POST /admin/control/translation-worker-run` 会触发 Rust 侧
`translation.worker.run`,请求字段为 `provider``fixture_path`
`concurrency``max_attempts``lease_seconds``retry_backoff_seconds`
`max_tasks``worker_id`。bat-api 只做鉴权、JSON 解码和基础范围校验;
任务状态、lease、重试和 provider 结果仍由 Rust 持久化。
`POST /admin/control/translation-proofread` 会把当前汉化 workflow 标记为人工校对中; `POST /admin/control/translation-proofread` 会把当前汉化 workflow 标记为人工校对中;
该接口只转发 `translation.proofread`,不会改动已发布汉化 release 指针。 该接口只转发 `translation.proofread`,不会改动已发布汉化 release 指针。
+4
View File
@@ -258,6 +258,7 @@ cargo run -p bat-infrastructure --bin bat -- parse-errors --limit 50
cargo run -p bat-infrastructure --bin bat -- translation-tasks --task-status skipped_parse_failed --has-reason --limit 50 cargo run -p bat-infrastructure --bin bat -- translation-tasks --task-status skipped_parse_failed --has-reason --limit 50
cargo run -p bat-infrastructure --bin bat -- translation-tasks --worker-status failed --has-failure-reason --limit 50 cargo run -p bat-infrastructure --bin bat -- translation-tasks --worker-status failed --has-failure-reason --limit 50
cargo run -p bat-infrastructure --bin bat -- translation-handoff cargo run -p bat-infrastructure --bin bat -- translation-handoff
cargo run -p bat-infrastructure --bin bat -- i18n worker run --provider mock --worker-concurrency 8 --worker-max-tasks 10
cargo run -p bat-infrastructure --bin bat -- localized-status cargo run -p bat-infrastructure --bin bat -- localized-status
cargo run -p bat-infrastructure --bin bat -- resource-index --limit 50 cargo run -p bat-infrastructure --bin bat -- resource-index --limit 50
cargo run -p bat-infrastructure --bin bat -- resource-index --release-id <ID> --platform windows --archive-entry <PATH> --format json --limit 50 cargo run -p bat-infrastructure --bin bat -- resource-index --release-id <ID> --platform windows --archive-entry <PATH> --format json --limit 50
@@ -270,6 +271,9 @@ cargo run -p bat-infrastructure --bin bat -- resource-index --release-id <ID> --
worker 状态、parse status、TextUnit format、队列 reason 和 provider failure reason worker 状态、parse status、TextUnit format、队列 reason 和 provider failure reason
查询离线 TextUnit 翻译任务状态与跳过/失败原因;发布后的状态保存在当前 release 查询离线 TextUnit 翻译任务状态与跳过/失败原因;发布后的状态保存在当前 release
根目录的 `translation-tasks.sqlite`,旧 release 没有状态库时回退到 JSON 队列; 根目录的 `translation-tasks.sqlite`,旧 release 没有状态库时回退到 JSON 队列;
`i18n worker run` / `translation.worker.run` 会由 Rust provider worker 独立 claim
下一项任务并落库 lease、失败分类、重试计划和 TextUnit 级译文结果,默认并发为 8,
范围 `1..=256`
`translation-handoff` / `translation.handoff` 会动态合并版本化 `translation-handoff` / `translation.handoff` 会动态合并版本化
`translation-handoff.json` 与 SQLite 状态,返回 job、unit、provider run 的完整交接 `translation-handoff.json` 与 SQLite 状态,返回 job、unit、provider run 的完整交接
视图; 视图;
+38 -10
View File
@@ -150,11 +150,12 @@ SQLite `ResourceRepository`,索引不存在时返回 `ok=true` 且
本地文件未变且索引有效时复用,不重复解析。 本地文件未变且索引有效时复用,不重复解析。
- `official-textunit-tasks.json`:只由 added + modified 资源、parse cache 和 - `official-textunit-tasks.json`:只由 added + modified 资源、parse cache 和
TextUnit 明细索引派生,记录 TextUnit 任务、跳过原因和解析诊断。 TextUnit 明细索引派生,记录 TextUnit 任务、跳过原因和解析诊断。
- `crowdin-textunit-queue.json`:只包含已产生 TextUnit 的离线任务,预留给后续 - `crowdin-textunit-queue.json`:只包含已产生 TextUnit 的离线任务;官方同步阶段
Crowdin worker;当前不会发出网络请求。 不发出 provider 网络请求。
- `translation-tasks.sqlite`:当前 release 的可变 worker 状态库,记录 - `translation-tasks.sqlite`:当前 release 的可变 worker 状态库,记录
queued / running / failed / completed / skipped、attempt count、provider run queued / running / failed / completed / skipped、attempt count、provider run
ID 和 failure reasonschema 由 `schema_migrations` 版本表管理。 ID、provider、TextUnit 级译文结果、lease、失败分类、可重试标记和
next attemptschema 由 `schema_migrations` 版本表管理。
- `translation-handoff.json`:当前 release 的版本化 job/unit/provider run 交接 - `translation-handoff.json`:当前 release 的版本化 job/unit/provider run 交接
快照;worker 更新后的实时状态仍以 `translation-tasks.sqlite` 为准。 快照;worker 更新后的实时状态仍以 `translation-tasks.sqlite` 为准。
@@ -206,6 +207,7 @@ SQLite `ResourceRepository`,索引不存在时返回 `ok=true` 且
| `translation.tasks` | 已实现 | `{ "offset": 0, "limit": 100, "task_id": "...", "release_id": "...", "destination": "...", "archive_entry": "...", "status": "skipped_parse_failed", "parse_status": "failed", "format": "json", "has_reason": true }` | 当前官方 release 的离线 TextUnit 翻译任务状态分页。 | | `translation.tasks` | 已实现 | `{ "offset": 0, "limit": 100, "task_id": "...", "release_id": "...", "destination": "...", "archive_entry": "...", "status": "skipped_parse_failed", "parse_status": "failed", "format": "json", "has_reason": true }` | 当前官方 release 的离线 TextUnit 翻译任务状态分页。 |
| `translation.handoff` | 已实现 | `null` | 当前官方 release 的 job、unit、provider run 交接视图;动态合并队列和 SQLite worker 状态。 | | `translation.handoff` | 已实现 | `null` | 当前官方 release 的 job、unit、provider run 交接视图;动态合并队列和 SQLite worker 状态。 |
| `translation.task.update` | 已实现 | `{ "task_id": "...", "status": "failed", "failure_reason": "...", "provider_run_id": "..." }` | 写入当前 release 的 provider worker 状态,返回可回查任务记录。 | | `translation.task.update` | 已实现 | `{ "task_id": "...", "status": "failed", "failure_reason": "...", "provider_run_id": "..." }` | 写入当前 release 的 provider worker 状态,返回可回查任务记录。 |
| `translation.worker.run` | 已实现 | provider worker 参数 | 异步触发 Rust provider worker,返回 `{ "task_id": "...", "kind": "translation.worker.run", "worker": {...} }`。 |
| `translation.proofread` | 已实现 | `null` | 将当前汉化 workflow 标记为人工校对中,返回工作流状态报告。 | | `translation.proofread` | 已实现 | `null` | 将当前汉化 workflow 标记为人工校对中,返回工作流状态报告。 |
`parse.status` 是只读查询;没有当前 release 或没有解析缓存时返回 `parse.status` 是只读查询;没有当前 release 或没有解析缓存时返回
@@ -215,8 +217,8 @@ SQLite `ResourceRepository`,索引不存在时返回 `ok=true` 且
`textunit_index_summary`;存在 `official-textunit-tasks.json` 时,响应会包含 `textunit_index_summary`;存在 `official-textunit-tasks.json` 时,响应会包含
`textunit_queue_available=true``textunit_task_queue_path` `textunit_queue_available=true``textunit_task_queue_path`
`textunit_task_summary`。当 TextUnit 队列存在且有离线任务时, `textunit_task_summary`。当 TextUnit 队列存在且有离线任务时,
`translation_status_code=translation.queued_offline`真实 Crowdin worker `translation_status_code=translation.queued_offline`provider worker 完成任务后,
尚未接入时不会返回翻译完成状态 同一查询面会返回已落库的 worker 状态和 TextUnit 级译文结果
`parse.text_units` / `parse.errors` 是只读查询;没有当前 release 或没有 `parse.text_units` / `parse.errors` 是只读查询;没有当前 release 或没有
`official-textunit-index.json` 时返回 `ok=true``data.available=false` `official-textunit-index.json` 时返回 `ok=true``data.available=false`
@@ -260,6 +262,31 @@ bat-api 可通过 `translation.tasks` 查询单项任务,也可通过
不会触发下载或 provider 网络请求;没有当前 release 或任务队列时返回 不会触发下载或 provider 网络请求;没有当前 release 或任务队列时返回
`data.available=false` `data.available=false`
`translation.worker.run` 通过 daemon 任务队列异步启动 Rust provider worker。
provider worker 会先同步当前 release 的 TextUnit 队列到
`translation-tasks.sqlite`,回收过期 lease,然后由 `concurrency` 个独立 worker
循环 claim 下一项任务;任一 worker 完成当前任务后会立即领取下一项,不等待
其他 worker 完成本轮批次。默认并发为 8,范围 `1..=256`
provider worker 参数:
| 字段 | 类型 | 默认 | 说明 |
|---|---|---|---|
| `provider` | string | `mock` | `mock``crowdin`。 |
| `fixture_path` | string/null | `null` | mock provider fixture;别名为 `translation_fixture``provider_fixture``mock_fixture``fixture`。 |
| `concurrency` | uint | `8` | 独立 worker 数,范围 `1..=256`;别名为 `worker_concurrency``translation_concurrency`。 |
| `max_attempts` | uint | `3` | 单个任务最大 claim 次数,必须大于 0。 |
| `lease_seconds` | uint | `300` | claim lease 秒数,必须大于 0。 |
| `retry_backoff_seconds` | uint | `5` | 可重试 provider 失败的 next attempt 间隔,可为 0。 |
| `max_tasks` | uint/null | `null` | 本轮最多 claim 的任务数,设置时必须大于 0。 |
| `worker_id` | string | `bat-rpc-worker` | lease 诊断用 worker ID 前缀。 |
数字字段必须是 JSON number;字符串数字、负数和越界值会返回
`BAT-ERR-700002``mock` provider 在没有 fixture 时把 source text 写成可诊断的
mock 译文;`crowdin` provider 从 `CROWDIN_PROJECT_ID``CROWDIN_LANGUAGE_ID`
`CROWDIN_API_TOKEN` 读取配置,可选 `CROWDIN_API_BASE_URL``BAT_CURL`
token 不会进入报告、任务记录或调试输出。
### localized ### localized
| 方法 | 状态 | params | data | | 方法 | 状态 | params | data |
@@ -397,6 +424,7 @@ CLI 对应关系:
| `bat translation-handoff` | `translation.handoff` | | `bat translation-handoff` | `translation.handoff` |
| `bat i18n task list` / `bat i18n task status` | `translation.tasks` | | `bat i18n task list` / `bat i18n task status` | `translation.tasks` |
| `bat i18n task update` | `translation.task.update` | | `bat i18n task update` | `translation.task.update` |
| `bat i18n worker run` | `translation.worker.run` |
| `bat i18n proofread` | `translation.proofread` | | `bat i18n proofread` | `translation.proofread` |
| `bat localized-status` | `localized.status` | | `bat localized-status` | `localized.status` |
| `bat resource-index` | `resource.index` | | `bat resource-index` | `resource.index` |
@@ -424,10 +452,10 @@ CLI 对应关系:
- typed helper 已覆盖 daemon 已实现方法(`status/logs/stop/restart/reload/refresh/doctor`)、 - typed helper 已覆盖 daemon 已实现方法(`status/logs/stop/restart/reload/refresh/doctor`)、
`resource.state/sync/verify/repair/manifest/list``schedule.list/add/update/remove/run` `resource.state/sync/verify/repair/manifest/list``schedule.list/add/update/remove/run`
`catalog.*``parse.*` `catalog.*``parse.*`
`localized.status``translation.task.update``translation.proofread``task.*` 和三个 `localized.status``translation.tasks``translation.handoff`
`unityfs.patch_*` 方法。 `translation.task.update``translation.worker.run``translation.proofread`
- `resource.index``translation.tasks``translation.handoff` `task.*` 和三个 `unityfs.patch_*` 方法。
`patch.apply` 当前没有专用 typed helper;需要直接使用 `Call`,并仍须遵守 - `resource.index` `patch.apply` 当前没有专用 typed helper;需要直接使用 `Call`,并仍须遵守
本契约的参数和响应定义。 本契约的参数和响应定义。
`internal/api` 对 bat-api 生产路径进一步收窄接口: `internal/api` 对 bat-api 生产路径进一步收窄接口:
@@ -437,7 +465,7 @@ CLI 对应关系:
| `Backend` | `daemon.status``daemon.doctor``resource.state``catalog.status``resource.manifest` | 启动发现、周期刷新和资源分发 | | `Backend` | `daemon.status``daemon.doctor``resource.state``catalog.status``resource.manifest` | 启动发现、周期刷新和资源分发 |
| `ControlBackend` | `daemon.restart``daemon.reload``daemon.refresh``resource.sync``resource.verify``resource.repair``catalog.refresh` | 鉴权后的管理控制白名单 | | `ControlBackend` | `daemon.restart``daemon.reload``daemon.refresh``resource.sync``resource.verify``resource.repair``catalog.refresh` | 鉴权后的管理控制白名单 |
| `ScheduleBackend` | `schedule.list``schedule.add``schedule.update``schedule.remove``schedule.run` | 鉴权后的 dashboard 调度计划控制 | | `ScheduleBackend` | `schedule.list``schedule.add``schedule.update``schedule.remove``schedule.run` | 鉴权后的 dashboard 调度计划控制 |
| `TranslationBackend` | `translation.task.update``translation.proofread` | 鉴权后的 dashboard 翻译任务状态回写与人工校对标记 | | `TranslationBackend` | `translation.tasks``translation.handoff``translation.task.update``translation.worker.run``translation.proofread` | 鉴权后的 dashboard 翻译任务查询、交接视图、状态回写、provider worker 触发与人工校对标记 |
`daemon.stop``daemon.clean-stable` 和任意通用 RPC 不属于 bat-api 管理控制面。 `daemon.stop``daemon.clean-stable` 和任意通用 RPC 不属于 bat-api 管理控制面。
Rust dispatch、Go transport 和 bat-api 接口的权威实现位置分别是 Rust dispatch、Go transport 和 bat-api 接口的权威实现位置分别是
+4 -4
View File
@@ -232,8 +232,8 @@ issue 43 例外说明:新增的 `parse repack` 只编排已有 TextAsset、Typ
- `cmd/bat-api`:组织 Rust `bat` 已发布 release 的启动前资源入口,并只读分发官方 CDN host/path 形态资源。 - `cmd/bat-api`:组织 Rust `bat` 已发布 release 的启动前资源入口,并只读分发官方 CDN host/path 形态资源。
- **拉取归属 Rust `bat`**`bat-api` 不做下载器。 - **拉取归属 Rust `bat`**`bat-api` 不做下载器。
- 发现经 `bat.sock`:先 `daemon.status`,再 `daemon.doctor`,再 `catalog.status` / `resource.manifest` - 发现经 `bat.sock`:先 `daemon.status`,再 `daemon.doctor`,再 `catalog.status` / `resource.manifest`
- Rust RPC 的 `resource.state``catalog.status``parse.status``localized.status` 会返回短状态 `status` 与稳定生命周期状态码 `status_code``backendrpc` 已提供对应 client 能力,当前 `bat-api` 发现流程只消费 `resource.state``catalog.status``resource.manifest`parse/localized 不由 HTTP surface 暴露;错误原因仍以 `BAT-ERR-*` 为准。 - Rust RPC 的 `resource.state``catalog.status``parse.status``localized.status` 会返回短状态 `status` 与稳定生命周期状态码 `status_code``backendrpc` 已提供对应 client 能力,当前 `bat-api` 发现流程只消费 `resource.state``catalog.status``resource.manifest`translation task/handoff 通过受鉴权的 admin 查询端点暴露,parse/localized 不由 HTTP surface 暴露;错误原因仍以 `BAT-ERR-*` 为准。
- `bat-api` 可通过受限 Web 控制面转发 `reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`,这些控制动作均走 Rust live RPC`parse.*``localized.status` 和文件级 `unityfs.patch_*` 属于 `backendrpc` 能力,但当前不由 HTTP surface 暴露;`daemon.clean-stable` 等危险或离线生命周期命令不经 Web 转发。 - `bat-api` 可通过受限 Web 控制面转发 `reload` / `refresh` / `restart` / `sync` / `verify` / `repair` / `catalog-refresh`并通过 translation admin 端点查询/回写任务、触发 provider worker 和标记人工校对;这些动作均走 Rust live RPC`parse.*``localized.status` 和文件级 `unityfs.patch_*` 属于 `backendrpc` 能力,但当前不由 HTTP surface 暴露;`daemon.clean-stable` 等危险或离线生命周期命令不经 Web 转发。
- 生产与 Rust `bat` 同环境运行,资源根来自 RPC 返回的 `resource_root``--resource-root` 仅用于 fixture 或应急只读诊断。 - 生产与 Rust `bat` 同环境运行,资源根来自 RPC 返回的 `resource_root``--resource-root` 仅用于 fixture 或应急只读诊断。
- `.env` 配置端口 / public base / RPC socket / RPC 刷新周期;预留 database/redis。 - `.env` 配置端口 / public base / RPC socket / RPC 刷新周期;预留 database/redis。
- `/v1/bootstrap` 返回 RPC 健康、release 摘要、server-info URL、client-patch base 和改写后的 Addressables root。 - `/v1/bootstrap` 返回 RPC 健康、release 摘要、server-info URL、client-patch base 和改写后的 Addressables root。
@@ -294,9 +294,9 @@ issue 43 例外说明:新增的 `parse repack` 只编排已有 TextAsset、Typ
- 官方 release 发布后还会写入版本化 `translation-handoff.json``translation.handoff` RPC/CLI 动态合并该快照与 SQLite worker 状态,暴露 job、unit、provider run、attempt 和 failure reason。 - 官方 release 发布后还会写入版本化 `translation-handoff.json``translation.handoff` RPC/CLI 动态合并该快照与 SQLite worker 状态,暴露 job、unit、provider run、attempt 和 failure reason。
- `translation-tasks.sqlite``schema_migrations` 管理 durable task state、attempt count、provider run ID 和 failure reason;重复同步会保留已有 worker 状态。 - `translation-tasks.sqlite``schema_migrations` 管理 durable task state、attempt count、provider run ID 和 failure reason;重复同步会保留已有 worker 状态。
- `translation.tasks` RPC/CLI 优先查询 `translation-tasks.sqlite`,旧 release 没有状态库时回退到 `official-textunit-tasks.json`;返回队列 `status`、worker `task_status`、failure reason 和时间/尝试次数。 - `translation.tasks` RPC/CLI 优先查询 `translation-tasks.sqlite`,旧 release 没有状态库时回退到 `official-textunit-tasks.json`;返回队列 `status`、worker `task_status`、failure reason 和时间/尝试次数。
- `translation.task.update` 已提供 queued/running/failed/completed/skipped 状态回写契约,并已通过 `bat i18n task update`、Go typed helper 和 bat-api 鉴权控制面暴露;`translation.proofread` 已提供人工校对状态标记契约provider worker 可在消费 handoff 后按 task_id 更新并由同一查询接口反查 - `translation.tasks` / `translation.handoff` 已提供 Go typed helper 和 bat-api 鉴权查询端点;`translation.task.update` 已提供 queued/running/failed/completed/skipped 状态回写契约,并已通过 `bat i18n task update`、Go typed helper 和 bat-api 鉴权控制面暴露;`translation.worker.run` 已提供 Rust provider worker,支持 mock/Crowdin、独立 claim、lease、失败分类、重试和 TextUnit 级译文结果落库;`translation.proofread` 已提供人工校对状态标记契约。
- `i18n get/set/unset` 可查看、修改或清空单条工作台译文,`i18n validate` 可在发布前校验工作台 release、source text、重复 patch 目标,并区分可直接发布的 TextAsset 与必须进入 `parse repack` 的条目;`i18n proofread` 可把当前汉化 workflow 标记为人工校对中,且不会遮蔽已发布汉化 release;`parse clear-cache` 只删除可再生解析/队列 JSON,保留 `translation-tasks.sqlite` 的 worker 状态。 - `i18n get/set/unset` 可查看、修改或清空单条工作台译文,`i18n validate` 可在发布前校验工作台 release、source text、重复 patch 目标,并区分可直接发布的 TextAsset 与必须进入 `parse repack` 的条目;`i18n proofread` 可把当前汉化 workflow 标记为人工校对中,且不会遮蔽已发布汉化 release;`parse clear-cache` 只删除可再生解析/队列 JSON,保留 `translation-tasks.sqlite` 的 worker 状态。
- 真实 Crowdin 网络 worker、翻译记忆和完整 localized repack 仍属于后续翻译系统工作,不在当前 Rust 离线状态仓储范围内。 - 翻译记忆和完整 localized repack 仍属于后续翻译系统工作,不在当前 provider worker 状态闭环范围内。
验收: 验收:
+1 -1
View File
@@ -95,7 +95,7 @@
| 组件 | 路径 | 状态 | 说明 | | 组件 | 路径 | 状态 | 说明 |
|---|---|---|---| |---|---|---|---|
| Module | `go.mod``bat-api` | 已用 | 服务层模块名 | | Module | `go.mod``bat-api` | 已用 | 服务层模块名 |
| RPC client | `internal/backendrpc` | **完成** | Unix socket JSON-RPC transport + typed helpertyped helper 覆盖 daemon 已实现控制/查询、`resource.state/sync/verify/repair/manifest/list``catalog.*``parse.*``localized.status``task.*``translation.task.update``translation.proofread` 和文件级 UnityFS patch 调用;`resource.index`其余 `translation.*``patch.apply` 仍通过通用 `Call` 走同一 contractfake transport 单测,配合 `internal/api/testdata/contract/` 固化 Rust 输出 mirror | | RPC client | `internal/backendrpc` | **完成** | Unix socket JSON-RPC transport + typed helpertyped helper 覆盖 daemon 已实现控制/查询、`resource.state/sync/verify/repair/manifest/list``catalog.*``parse.*``localized.status``task.*``translation.tasks``translation.handoff``translation.task.update``translation.worker.run``translation.proofread` 和文件级 UnityFS patch 调用;`resource.index``patch.apply` 仍通过通用 `Call` 走同一 contractfake transport 单测,配合 `internal/api/testdata/contract/` 固化 Rust 输出 mirror |
| 资源 bootstrap/分发 | `cmd/bat-api` + `internal/api` | **MVP+生产控制面** | RPC 发现 + 周期刷新/诊断 + `/v1/bootstrap` + `/v1/launcher/bootstrap` + launcher 资源 metadata 兼容 + `/readyz` + CDN Range/缓存头 + 鉴权/限流/访问日志/反代适配 + OpenAPI + 管理控制白名单 + `.env` | | 资源 bootstrap/分发 | `cmd/bat-api` + `internal/api` | **MVP+生产控制面** | RPC 发现 + 周期刷新/诊断 + `/v1/bootstrap` + `/v1/launcher/bootstrap` + launcher 资源 metadata 兼容 + `/readyz` + CDN Range/缓存头 + 鉴权/限流/访问日志/反代适配 + OpenAPI + 管理控制白名单 + `.env` |
| 试验 CLI | `cmd/bat` | **试验** | doctor 固定 okmanifest/sync 走 FFI | | 试验 CLI | `cmd/bat` | **试验** | doctor 固定 okmanifest/sync 走 FFI |
| FFI | `internal/ffi` | **可选** | 需 `build-ffi` | | FFI | `internal/ffi` | **可选** | 需 `build-ffi` |
+541 -7
View File
@@ -23,14 +23,16 @@ use bat_infrastructure::{
OfficialUpdateService, OfficialUpdateSnapshot, OfficialUpdateStatus, OfficialUpdateService, OfficialUpdateSnapshot, OfficialUpdateStatus,
OfficialVerificationSummary, OfficialVersionRecord, OfficialVersionState, PatchApplyKind, OfficialVerificationSummary, OfficialVersionRecord, OfficialVersionState, PatchApplyKind,
PatchApplyParams, PatchApplyReport, ReleaseFlowStatusCode, RepackReport, PatchApplyParams, PatchApplyReport, ReleaseFlowStatusCode, RepackReport,
SqliteResourceRepository, SqliteTranslationTaskRepository, TranslationTaskStatus, SqliteResourceRepository, SqliteTranslationTaskRepository, TranslationProviderKind,
UnityFsFieldPatchParams, UnityFsPatchReport, UnityFsStringFieldPatchParams, TranslationTaskStatus, TranslationWorkerConfig, UnityFsFieldPatchParams, UnityFsPatchReport,
UnityFsTextAssetPatchParams, CROWDIN_TEXTUNIT_QUEUE_FILE, LOCALIZED_CURRENT_LINK, UnityFsStringFieldPatchParams, UnityFsTextAssetPatchParams, CROWDIN_TEXTUNIT_QUEUE_FILE,
DEFAULT_TRANSLATION_CONCURRENCY, DEFAULT_TRANSLATION_LEASE_SECONDS,
DEFAULT_TRANSLATION_MAX_ATTEMPTS, DEFAULT_TRANSLATION_RETRY_BACKOFF, LOCALIZED_CURRENT_LINK,
LOCALIZED_PATCH_MANIFEST_FILE, LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING, LOCALIZED_PATCH_MANIFEST_FILE, LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING,
LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING_LABEL, LOCALIZED_VERSIONS_DIR, LOCALIZED_TRANSLATION_STATUS_MANUAL_PROOFREADING_LABEL, LOCALIZED_VERSIONS_DIR,
LOCALIZED_VERSION_STATE_FILE, MAX_DOWNLOAD_CONCURRENCY, MIN_DOWNLOAD_CONCURRENCY, LOCALIZED_VERSION_STATE_FILE, MAX_DOWNLOAD_CONCURRENCY, MAX_TRANSLATION_CONCURRENCY,
OFFICIAL_PARSE_CACHE_FILE, OFFICIAL_TEXTUNIT_INDEX_FILE, OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE, MIN_DOWNLOAD_CONCURRENCY, MIN_TRANSLATION_CONCURRENCY, OFFICIAL_PARSE_CACHE_FILE,
PRIVATE_FILE_MODE, OFFICIAL_TEXTUNIT_INDEX_FILE, OFFICIAL_TEXTUNIT_TASK_QUEUE_FILE, PRIVATE_FILE_MODE,
}; };
use serde::de::DeserializeOwned; use serde::de::DeserializeOwned;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
@@ -85,6 +87,7 @@ use workflow_commands::{
run_parse_clear_cache, run_parse_once, run_publish_localized, run_repack, run_translate_once, run_parse_clear_cache, run_parse_once, run_publish_localized, run_repack, run_translate_once,
run_translation_get, run_translation_proofread, run_translation_set, run_translation_get, run_translation_proofread, run_translation_set,
run_translation_task_update, run_translation_unset, run_translation_validate, run_translation_task_update, run_translation_unset, run_translation_validate,
run_translation_worker,
}; };
const EXIT_ERROR: i32 = 1; const EXIT_ERROR: i32 = 1;
@@ -229,6 +232,10 @@ fn run() -> anyhow::Result<i32> {
run_translation_proofread(&options)?; run_translation_proofread(&options)?;
Ok(0) Ok(0)
} }
CliCommand::TranslationWorker => {
run_repeated_workflow(&options, "translation-worker", run_translation_worker)?;
Ok(0)
}
CliCommand::Repack => { CliCommand::Repack => {
run_repack(&options)?; run_repack(&options)?;
Ok(0) Ok(0)
@@ -378,6 +385,15 @@ struct CliOptions {
translation_text_file: Option<PathBuf>, translation_text_file: Option<PathBuf>,
translation_failure_reason: Option<String>, translation_failure_reason: Option<String>,
translation_provider_run_id: Option<String>, translation_provider_run_id: Option<String>,
translation_provider: Option<String>,
translation_fixture: Option<PathBuf>,
worker_concurrency: usize,
worker_max_attempts: u32,
worker_lease_seconds: u64,
worker_retry_backoff: Duration,
worker_max_tasks: Option<usize>,
worker_id: Option<String>,
translation_worker_option_explicit: bool,
localized_release_id: Option<String>, localized_release_id: Option<String>,
repack_spec: Option<PathBuf>, repack_spec: Option<PathBuf>,
schedule_group: Option<String>, schedule_group: Option<String>,
@@ -463,6 +479,15 @@ impl Default for CliOptions {
translation_text_file: None, translation_text_file: None,
translation_failure_reason: None, translation_failure_reason: None,
translation_provider_run_id: None, translation_provider_run_id: None,
translation_provider: None,
translation_fixture: None,
worker_concurrency: DEFAULT_TRANSLATION_CONCURRENCY,
worker_max_attempts: DEFAULT_TRANSLATION_MAX_ATTEMPTS,
worker_lease_seconds: DEFAULT_TRANSLATION_LEASE_SECONDS,
worker_retry_backoff: DEFAULT_TRANSLATION_RETRY_BACKOFF,
worker_max_tasks: None,
worker_id: None,
translation_worker_option_explicit: false,
localized_release_id: None, localized_release_id: None,
repack_spec: None, repack_spec: None,
schedule_group: None, schedule_group: None,
@@ -546,6 +571,7 @@ enum CliCommand {
TranslationGet, TranslationGet,
TranslationUnset, TranslationUnset,
TranslationTaskUpdate, TranslationTaskUpdate,
TranslationWorker,
TranslationProofread, TranslationProofread,
Repack, Repack,
PublishLocalized, PublishLocalized,
@@ -1025,6 +1051,7 @@ const RPC_METHOD_TRANSLATION_TASKS: &str = "translation.tasks";
const RPC_METHOD_TRANSLATION_HANDOFF: &str = "translation.handoff"; const RPC_METHOD_TRANSLATION_HANDOFF: &str = "translation.handoff";
const RPC_METHOD_TRANSLATION_TASK_UPDATE: &str = "translation.task.update"; const RPC_METHOD_TRANSLATION_TASK_UPDATE: &str = "translation.task.update";
const RPC_METHOD_TRANSLATION_PROOFREAD: &str = "translation.proofread"; const RPC_METHOD_TRANSLATION_PROOFREAD: &str = "translation.proofread";
const RPC_METHOD_TRANSLATION_WORKER_RUN: &str = "translation.worker.run";
const RPC_METHOD_LOCALIZED_STATUS: &str = "localized.status"; const RPC_METHOD_LOCALIZED_STATUS: &str = "localized.status";
const RPC_METHOD_CATALOG_STATUS: &str = "catalog.status"; const RPC_METHOD_CATALOG_STATUS: &str = "catalog.status";
const RPC_METHOD_CATALOG_VERSIONS: &str = "catalog.versions"; const RPC_METHOD_CATALOG_VERSIONS: &str = "catalog.versions";
@@ -2027,6 +2054,13 @@ fn dispatch_rpc_method(
mark_localized_manual_proofreading_report(state_dir, &tasks.base_config) mark_localized_manual_proofreading_report(state_dir, &tasks.base_config)
.and_then(|report| serde_json::to_value(report).map_err(anyhow::Error::from)), .and_then(|report| serde_json::to_value(report).map_err(anyhow::Error::from)),
), ),
RPC_METHOD_TRANSLATION_WORKER_RUN => {
let config = match rpc_translation_worker_config(request.params.as_ref()) {
Ok(config) => config,
Err(error) => return rpc_envelope_error(request_id, error),
};
enqueue_translation_worker_envelope(tasks, config, request_id)
}
RPC_METHOD_LOCALIZED_STATUS => rpc_envelope_from_result( RPC_METHOD_LOCALIZED_STATUS => rpc_envelope_from_result(
request_id, request_id,
"localized.status", "localized.status",
@@ -2211,7 +2245,9 @@ fn enqueue_task_envelope(
.unwrap_or_else(|| Arc::new(AtomicBool::new(false))); .unwrap_or_else(|| Arc::new(AtomicBool::new(false)));
let job = TaskJob { let job = TaskJob {
id: task_id.clone(), id: task_id.clone(),
kind,
config, config,
translation_worker_config: None,
cancel, cancel,
}; };
if tasks.queue.send(job).is_err() { if tasks.queue.send(job).is_err() {
@@ -2237,6 +2273,50 @@ fn enqueue_task_envelope(
) )
} }
fn enqueue_translation_worker_envelope(
tasks: &DaemonTaskContext,
worker_config: TranslationWorkerConfig,
request_id: String,
) -> RpcEnvelope {
let config = TaskKind::TranslationWorker.build_config(&tasks.base_config, false);
let task_id = tasks.registry.create(TaskKind::TranslationWorker);
let cancel = tasks
.registry
.cancel_flag(&task_id)
.unwrap_or_else(|| Arc::new(AtomicBool::new(false)));
let job = TaskJob {
id: task_id.clone(),
kind: TaskKind::TranslationWorker,
config,
translation_worker_config: Some(worker_config.clone()),
cancel,
};
if tasks.queue.send(job).is_err() {
tasks.registry.update(&task_id, |record| {
record.status = "failed";
record.finished_at = Some(unix_seconds_now());
record.error = Some(ApiError::new(
ErrorCode::INTERNAL,
"task.enqueue",
"任务执行器不可用",
));
});
return rpc_envelope_error(
request_id,
ApiError::new(ErrorCode::INTERNAL, "task.enqueue", "任务执行器不可用"),
);
}
rpc_envelope_ok(
request_id,
"accepted",
serde_json::json!({
"task_id": task_id,
"kind": RPC_METHOD_TRANSLATION_WORKER_RUN,
"worker": worker_config,
}),
)
}
fn rpc_ack_value( fn rpc_ack_value(
command: &'static str, command: &'static str,
message: &'static str, message: &'static str,
@@ -3070,6 +3150,207 @@ fn rpc_struct_params<T: DeserializeOwned>(
}) })
} }
fn rpc_translation_worker_config(
params: Option<&serde_json::Value>,
) -> Result<TranslationWorkerConfig, ApiError> {
let empty = serde_json::json!({});
let params = params.unwrap_or(&empty);
if !params.is_object() {
return Err(ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
"params 必须是 JSON object",
));
}
let provider = rpc_translation_worker_string_param(
params,
&["provider", "translation_provider"],
"provider",
)?
.unwrap_or_else(|| TranslationProviderKind::Mock.as_str().to_string());
let provider = TranslationProviderKind::parse(&provider).ok_or_else(|| {
ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
format!("不支持的 translation provider{provider}"),
)
})?;
let concurrency = rpc_translation_worker_usize_param(
params,
&[
"concurrency",
"worker_concurrency",
"translation_concurrency",
],
"concurrency",
)?
.unwrap_or(DEFAULT_TRANSLATION_CONCURRENCY);
let max_attempts = rpc_translation_worker_u32_param(
params,
&[
"max_attempts",
"worker_max_attempts",
"translation_max_attempts",
],
"max_attempts",
)?
.unwrap_or(DEFAULT_TRANSLATION_MAX_ATTEMPTS);
let lease_seconds = rpc_translation_worker_u64_param(
params,
&[
"lease_seconds",
"worker_lease_seconds",
"translation_lease_seconds",
],
"lease_seconds",
)?
.unwrap_or(DEFAULT_TRANSLATION_LEASE_SECONDS);
let retry_backoff_seconds = rpc_translation_worker_u64_param(
params,
&[
"retry_backoff_seconds",
"worker_retry_backoff_seconds",
"translation_retry_backoff_seconds",
],
"retry_backoff_seconds",
)?
.unwrap_or(DEFAULT_TRANSLATION_RETRY_BACKOFF.as_secs());
let max_tasks = rpc_translation_worker_usize_param(
params,
&["max_tasks", "worker_max_tasks", "translation_max_tasks"],
"max_tasks",
)?;
let config = TranslationWorkerConfig {
provider,
fixture_path: rpc_translation_worker_string_param(
params,
&[
"fixture_path",
"translation_fixture",
"provider_fixture",
"mock_fixture",
"fixture",
],
"fixture_path",
)?
.map(PathBuf::from),
concurrency,
max_attempts,
lease_seconds,
retry_backoff: Duration::from_secs(retry_backoff_seconds),
max_tasks,
worker_id: rpc_translation_worker_string_param(
params,
&["worker_id", "translation_worker_id"],
"worker_id",
)?
.unwrap_or_else(|| "bat-rpc-worker".to_string()),
};
config.validate().map_err(|error| {
ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
error.to_string(),
)
})?;
Ok(config)
}
fn rpc_translation_worker_string_param(
params: &serde_json::Value,
aliases: &[&str],
label: &str,
) -> Result<Option<String>, ApiError> {
for key in aliases {
let Some(value) = params.get(*key) else {
continue;
};
if value.is_null() {
return Ok(None);
}
let Some(text) = value.as_str() else {
return Err(ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
format!("{label} 必须是字符串"),
));
};
let text = text.trim();
if text.is_empty() {
return Ok(None);
}
return Ok(Some(text.to_string()));
}
Ok(None)
}
fn rpc_translation_worker_u64_param(
params: &serde_json::Value,
aliases: &[&str],
label: &str,
) -> Result<Option<u64>, ApiError> {
for key in aliases {
let Some(value) = params.get(*key) else {
continue;
};
if value.is_null() {
return Ok(None);
}
if value.as_i64().is_some_and(|number| number < 0) {
return Err(ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
format!("{label} 不能为负数"),
));
}
let Some(number) = value.as_u64() else {
return Err(ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
format!("{label} 必须是非负整数 JSON number"),
));
};
return Ok(Some(number));
}
Ok(None)
}
fn rpc_translation_worker_usize_param(
params: &serde_json::Value,
aliases: &[&str],
label: &str,
) -> Result<Option<usize>, ApiError> {
rpc_translation_worker_u64_param(params, aliases, label)?
.map(|number| {
usize::try_from(number).map_err(|error| {
ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
format!("{label} 无效:{error}"),
)
})
})
.transpose()
}
fn rpc_translation_worker_u32_param(
params: &serde_json::Value,
aliases: &[&str],
label: &str,
) -> Result<Option<u32>, ApiError> {
rpc_translation_worker_u64_param(params, aliases, label)?
.map(|number| {
u32::try_from(number).map_err(|error| {
ApiError::new(
ErrorCode::RPC_INVALID_PARAMS,
RPC_METHOD_TRANSLATION_WORKER_RUN,
format!("{label} 无效:{error}"),
)
})
})
.transpose()
}
/// 分页参数:`offset` 默认 0`limit` 默认 100,范围 1..=1000。 /// 分页参数:`offset` 默认 0`limit` 默认 100,范围 1..=1000。
fn rpc_page_params(params: Option<&serde_json::Value>) -> anyhow::Result<(usize, usize)> { fn rpc_page_params(params: Option<&serde_json::Value>) -> anyhow::Result<(usize, usize)> {
let offset = params let offset = params
@@ -4221,6 +4502,32 @@ impl HumanReport for bat_infrastructure::LocalizedTranslationWorkflowReport {
} }
} }
impl HumanReport for bat_infrastructure::TranslationWorkerReport {
fn print_human(&self) -> anyhow::Result<()> {
print_title("翻译 provider worker");
print_field("命令", self.command);
print_field("状态", self.status);
print_field("官方 release", &self.official_release_id);
print_field("provider", &self.provider);
print_field("回收 lease", self.recovered_lease_count);
print_field("领取任务", self.claimed_count);
print_field("完成任务", self.completed_count);
print_field("失败任务", self.failed_count);
print_field("已安排重试", self.retry_scheduled_count);
print_field("剩余任务", self.remaining_count);
for failure in &self.failures {
println!(
" - {} [{}] retryable={} {}",
failure.task_id,
failure.failure_class,
format_bool(failure.retryable),
failure.failure_reason
);
}
Ok(())
}
}
fn print_human_json_value(value: &serde_json::Value) -> anyhow::Result<()> { fn print_human_json_value(value: &serde_json::Value) -> anyhow::Result<()> {
if value.get("running").is_some() && value.get("state_dir").is_some() { if value.get("running").is_some() && value.get("state_dir").is_some() {
print_title("后台状态"); print_title("后台状态");
@@ -6321,6 +6628,17 @@ BAT_AUTO_DISCOVER=1
#BAT_DOWNLOAD_CONCURRENCY=8 #BAT_DOWNLOAD_CONCURRENCY=8
#BAT_UNZIP=unzip #BAT_UNZIP=unzip
# ---- provider worker ----
# providermock fixture crowdin CROWDIN_*
#BAT_TRANSLATION_PROVIDER=mock
#BAT_TRANSLATION_FIXTURE=
#BAT_TRANSLATION_CONCURRENCY=8
#BAT_TRANSLATION_MAX_ATTEMPTS=3
#BAT_TRANSLATION_LEASE_SECONDS=300
#BAT_TRANSLATION_RETRY_BACKOFF_SECONDS=5
#BAT_TRANSLATION_MAX_TASKS=
#BAT_TRANSLATION_WORKER_ID=
# ---- ---- # ---- ----
# 1 JSON # 1 JSON
#BAT_JSON=0 #BAT_JSON=0
@@ -6504,6 +6822,38 @@ fn apply_bat_env_overrides(
if let Some(v) = value("BAT_UNZIP") { if let Some(v) = value("BAT_UNZIP") {
options.config.unzip_command = PathBuf::from(v); options.config.unzip_command = PathBuf::from(v);
} }
if let Some(v) = value("BAT_TRANSLATION_PROVIDER") {
options.translation_provider = Some(v);
}
if let Some(v) = value("BAT_TRANSLATION_FIXTURE") {
options.translation_fixture = Some(PathBuf::from(v));
}
if let Some(v) = value("BAT_TRANSLATION_CONCURRENCY") {
options.worker_concurrency =
parse_translation_worker_concurrency(&v, "环境变量 BAT_TRANSLATION_CONCURRENCY")?;
}
if let Some(v) = value("BAT_TRANSLATION_MAX_ATTEMPTS") {
options.worker_max_attempts =
parse_positive_u32(&v, "环境变量 BAT_TRANSLATION_MAX_ATTEMPTS")?;
}
if let Some(v) = value("BAT_TRANSLATION_LEASE_SECONDS") {
options.worker_lease_seconds =
parse_positive_u64(&v, "环境变量 BAT_TRANSLATION_LEASE_SECONDS")?;
}
if let Some(v) = value("BAT_TRANSLATION_RETRY_BACKOFF_SECONDS") {
options.worker_retry_backoff = Duration::from_secs(v.parse::<u64>().map_err(|error| {
anyhow::anyhow!("环境变量 BAT_TRANSLATION_RETRY_BACKOFF_SECONDS 的秒数无效:{error}")
})?);
}
if let Some(v) = value("BAT_TRANSLATION_MAX_TASKS") {
options.worker_max_tasks = Some(parse_positive_usize(
&v,
"环境变量 BAT_TRANSLATION_MAX_TASKS",
)?);
}
if let Some(v) = value("BAT_TRANSLATION_WORKER_ID") {
options.worker_id = Some(v);
}
if let Some(v) = value("BAT_PROXY") { if let Some(v) = value("BAT_PROXY") {
options.config.curl_proxy = parse_proxy_config(&v)?; options.config.curl_proxy = parse_proxy_config(&v)?;
} }
@@ -6725,6 +7075,55 @@ fn parse_args_with_env(
"--provider-run-id" => { "--provider-run-id" => {
options.translation_provider_run_id = Some(next_option_value(&mut args, &flag)?); options.translation_provider_run_id = Some(next_option_value(&mut args, &flag)?);
} }
"--translation-provider" | "--provider" => {
options.translation_provider = Some(next_option_value(&mut args, &flag)?);
options.translation_worker_option_explicit = true;
}
"--translation-fixture" | "--provider-fixture" | "--mock-fixture" | "--fixture" => {
options.translation_fixture =
Some(PathBuf::from(next_option_value(&mut args, &flag)?));
options.translation_worker_option_explicit = true;
}
"--worker-concurrency" | "--translation-concurrency" => {
options.worker_concurrency = parse_translation_worker_concurrency(
&next_option_value(&mut args, &flag)?,
&flag,
)?;
options.translation_worker_option_explicit = true;
}
"--worker-max-attempts" | "--translation-max-attempts" => {
options.worker_max_attempts =
parse_positive_u32(&next_option_value(&mut args, &flag)?, &flag)?;
options.translation_worker_option_explicit = true;
}
"--worker-lease-seconds" | "--translation-lease-seconds" => {
options.worker_lease_seconds =
parse_positive_u64(&next_option_value(&mut args, &flag)?, &flag)?;
options.translation_worker_option_explicit = true;
}
"--worker-retry-backoff" | "--translation-retry-backoff" => {
options.worker_retry_backoff =
parse_duration(&next_option_value(&mut args, &flag)?)?;
options.translation_worker_option_explicit = true;
}
"--worker-retry-backoff-seconds" | "--translation-retry-backoff-seconds" => {
let seconds = next_option_value(&mut args, &flag)?
.parse::<u64>()
.map_err(|error| anyhow::anyhow!("{flag} 的秒数无效:{error}"))?;
options.worker_retry_backoff = Duration::from_secs(seconds);
options.translation_worker_option_explicit = true;
}
"--worker-max-tasks" | "--translation-max-tasks" => {
options.worker_max_tasks = Some(parse_positive_usize(
&next_option_value(&mut args, &flag)?,
&flag,
)?);
options.translation_worker_option_explicit = true;
}
"--worker-id" | "--translation-worker-id" => {
options.worker_id = Some(next_option_value(&mut args, &flag)?);
options.translation_worker_option_explicit = true;
}
"--localized-release-id" => { "--localized-release-id" => {
options.localized_release_id = Some(next_option_value(&mut args, &flag)?); options.localized_release_id = Some(next_option_value(&mut args, &flag)?);
} }
@@ -7185,6 +7584,12 @@ fn parse_args_with_env(
)); ));
} }
if options.command != CliCommand::TranslationWorker
&& options.translation_worker_option_explicit
{
return Err(anyhow::anyhow!("翻译 worker 参数只适用于 i18n worker run"));
}
match options.command { match options.command {
CliCommand::Status | CliCommand::Stop | CliCommand::Logs => { CliCommand::Status | CliCommand::Stop | CliCommand::Logs => {
if options.sync_option_explicit if options.sync_option_explicit
@@ -7406,6 +7811,56 @@ fn parse_args_with_env(
options.progress = false; options.progress = false;
options.banner = false; options.banner = false;
} }
CliCommand::TranslationWorker => {
if options.daemon || options.daemon_child {
return Err(anyhow::anyhow!(
"i18n worker run 使用 --watch 或 --run-count,不支持 daemon"
));
}
if options.config.force
|| options.config.dry_run
|| options.schedule_option_explicit
|| options.query_option_explicit
|| options.write_patch_option_explicit
|| options.translation_file.is_some()
|| options.translation_id.is_some()
|| options.translation_text.is_some()
|| options.translation_text_file.is_some()
|| options.translation_failure_reason.is_some()
|| options.translation_provider_run_id.is_some()
|| options.localized_release_id.is_some()
|| options.repack_spec.is_some()
|| options.proxy_option_explicit
|| tools_are_non_default(&options.config, &options.env_baseline_config)
{
return Err(anyhow::anyhow!(
"i18n worker run 只接受 --output、--resource-root、--state-dir、worker 参数和轮询参数"
));
}
let provider = options
.translation_provider
.as_deref()
.unwrap_or(TranslationProviderKind::Mock.as_str());
if TranslationProviderKind::parse(provider).is_none() {
return Err(anyhow::anyhow!(
"i18n worker run 的 provider 无效:{provider}"
));
}
TranslationWorkerConfig {
provider: TranslationProviderKind::parse(provider).expect("provider 已在上方校验"),
fixture_path: options.translation_fixture.clone(),
concurrency: options.worker_concurrency,
max_attempts: options.worker_max_attempts,
lease_seconds: options.worker_lease_seconds,
retry_backoff: options.worker_retry_backoff,
max_tasks: options.worker_max_tasks,
worker_id: options
.worker_id
.clone()
.unwrap_or_else(|| "bat-worker-validation".to_string()),
}
.validate()?;
}
CliCommand::TranslationProofread => { CliCommand::TranslationProofread => {
if options.watch || options.daemon || options.daemon_child { if options.watch || options.daemon || options.daemon_child {
return Err(anyhow::anyhow!("i18n proofread 只支持单次执行或 RPC 调用")); return Err(anyhow::anyhow!("i18n proofread 只支持单次执行或 RPC 调用"));
@@ -7506,6 +7961,7 @@ fn parse_args_with_env(
CliCommand::Pull CliCommand::Pull
| CliCommand::Parse | CliCommand::Parse
| CliCommand::Translate | CliCommand::Translate
| CliCommand::TranslationWorker
| CliCommand::PublishLocalized | CliCommand::PublishLocalized
) )
{ {
@@ -7515,7 +7971,11 @@ fn parse_args_with_env(
} }
if matches!( if matches!(
options.command, options.command,
CliCommand::Pull | CliCommand::Parse | CliCommand::Translate | CliCommand::PublishLocalized CliCommand::Pull
| CliCommand::Parse
| CliCommand::Translate
| CliCommand::PublishLocalized
| CliCommand::TranslationWorker
) { ) {
if options.watch && options.run_count.is_some() { if options.watch && options.run_count.is_some() {
return Err(anyhow::anyhow!( return Err(anyhow::anyhow!(
@@ -7632,6 +8092,9 @@ fn parse_translation_command(
if action == "workbench" || action == "wb" { if action == "workbench" || action == "wb" {
return parse_translation_workbench_command(args, options); return parse_translation_workbench_command(args, options);
} }
if action == "worker" {
return parse_translation_worker_command(args, options);
}
let command = match action.as_str() { let command = match action.as_str() {
"run" => CliCommand::Translate, "run" => CliCommand::Translate,
"export" => CliCommand::Translate, "export" => CliCommand::Translate,
@@ -7655,6 +8118,20 @@ fn parse_translation_command(
Ok(()) Ok(())
} }
fn parse_translation_worker_command(
args: &mut impl Iterator<Item = String>,
options: &mut CliOptions,
) -> anyhow::Result<()> {
let action = next_option_value(args, "translation worker")?;
let command = match action.as_str() {
"run" => CliCommand::TranslationWorker,
other => return Err(anyhow::anyhow!("未知 translation worker 二级命令:{other}")),
};
ensure_command_not_set(options.command, &format!("translation worker {action}"))?;
options.command = command;
Ok(())
}
fn parse_translation_workbench_command( fn parse_translation_workbench_command(
args: &mut impl Iterator<Item = String>, args: &mut impl Iterator<Item = String>,
options: &mut CliOptions, options: &mut CliOptions,
@@ -7738,6 +8215,48 @@ fn parse_download_concurrency(value: &str, source: &str) -> anyhow::Result<usize
Ok(parsed) Ok(parsed)
} }
fn parse_translation_worker_concurrency(value: &str, source: &str) -> anyhow::Result<usize> {
let parsed = value
.parse::<usize>()
.map_err(|error| anyhow::anyhow!("{source} 无效:{error}"))?;
if !(MIN_TRANSLATION_CONCURRENCY..=MAX_TRANSLATION_CONCURRENCY).contains(&parsed) {
return Err(anyhow::anyhow!(
"{source} 必须在 {MIN_TRANSLATION_CONCURRENCY}..={MAX_TRANSLATION_CONCURRENCY} 范围内"
));
}
Ok(parsed)
}
fn parse_positive_usize(value: &str, source: &str) -> anyhow::Result<usize> {
let parsed = value
.parse::<usize>()
.map_err(|error| anyhow::anyhow!("{source} 无效:{error}"))?;
if parsed == 0 {
return Err(anyhow::anyhow!("{source} 必须大于 0"));
}
Ok(parsed)
}
fn parse_positive_u32(value: &str, source: &str) -> anyhow::Result<u32> {
let parsed = value
.parse::<u32>()
.map_err(|error| anyhow::anyhow!("{source} 无效:{error}"))?;
if parsed == 0 {
return Err(anyhow::anyhow!("{source} 必须大于 0"));
}
Ok(parsed)
}
fn parse_positive_u64(value: &str, source: &str) -> anyhow::Result<u64> {
let parsed = value
.parse::<u64>()
.map_err(|error| anyhow::anyhow!("{source} 无效:{error}"))?;
if parsed == 0 {
return Err(anyhow::anyhow!("{source} 必须大于 0"));
}
Ok(parsed)
}
fn next_option_value( fn next_option_value(
args: &mut impl Iterator<Item = String>, args: &mut impl Iterator<Item = String>,
flag: &str, flag: &str,
@@ -7807,6 +8326,7 @@ fn print_usage(binary: &str) {
eprintln!(" i18n unset Clear one translated workbench entry"); eprintln!(" i18n unset Clear one translated workbench entry");
eprintln!(" i18n validate Validate workbench against the current official release"); eprintln!(" i18n validate Validate workbench against the current official release");
eprintln!(" i18n proofread Mark localized workflow as manual proofreading"); eprintln!(" i18n proofread Mark localized workflow as manual proofreading");
eprintln!(" i18n worker run Run translation provider worker once or repeatedly");
eprintln!( eprintln!(
" i18n tasks / i18n task list / i18n task status Query current offline TextUnit translation task status" " i18n tasks / i18n task list / i18n task status Query current offline TextUnit translation task status"
); );
@@ -7854,6 +8374,9 @@ fn print_usage(binary: &str) {
" {binary} i18n unset --translation-file /tmp/bat-workbench.json --translation-id unit-1" " {binary} i18n unset --translation-file /tmp/bat-workbench.json --translation-id unit-1"
); );
eprintln!(" {binary} i18n proofread --json"); eprintln!(" {binary} i18n proofread --json");
eprintln!(
" {binary} i18n worker run --provider mock --worker-concurrency 8 --run-count 2 --interval 30s"
);
eprintln!(" {binary} i18n tasks --json"); eprintln!(" {binary} i18n tasks --json");
eprintln!(" {binary} i18n handoff --json"); eprintln!(" {binary} i18n handoff --json");
eprintln!(" {binary} i18n status --json"); eprintln!(" {binary} i18n status --json");
@@ -7908,6 +8431,17 @@ fn print_usage(binary: &str) {
eprintln!(" --translated-file <PATH> UTF-8 translation file for i18n set"); eprintln!(" --translated-file <PATH> UTF-8 translation file for i18n set");
eprintln!(" --failure-reason <TEXT> Provider failure reason for i18n task update"); eprintln!(" --failure-reason <TEXT> Provider failure reason for i18n task update");
eprintln!(" --provider-run-id <ID> Provider run ID for i18n task update"); eprintln!(" --provider-run-id <ID> Provider run ID for i18n task update");
eprintln!(" --translation-provider <NAME> / --provider <NAME> Provider for i18n worker run (mock/crowdin)");
eprintln!(" --translation-fixture <PATH> Mock/provider fixture for i18n worker run");
eprintln!(
" --worker-concurrency <N> Translation worker concurrency (default: 8, range 1..=256)"
);
eprintln!(" --worker-max-attempts <N> Maximum claims per translation task");
eprintln!(" --worker-lease-seconds <N> Lease seconds for one claimed task");
eprintln!(" --worker-retry-backoff <DURATION> Retry backoff after retryable failure");
eprintln!(" --worker-retry-backoff-seconds <N> Retry backoff seconds");
eprintln!(" --worker-max-tasks <N> Maximum tasks claimed in one worker run");
eprintln!(" --worker-id <ID> Worker ID prefix for lease diagnostics");
eprintln!(" --localized-release-id <ID> Explicit localized publication ID"); eprintln!(" --localized-release-id <ID> Explicit localized publication ID");
eprintln!(" --repack-spec <PATH> UnityFS batch repack JSON spec"); eprintln!(" --repack-spec <PATH> UnityFS batch repack JSON spec");
eprintln!(); eprintln!();
+159
View File
@@ -140,6 +140,165 @@ fn env_invalid_values_error() {
assert!(parse(&["bat", "--download-concurrency", "257"]).is_err()); assert!(parse(&["bat", "--download-concurrency", "257"]).is_err());
} }
#[test]
fn translation_worker_command_options_are_validated() {
let options = parse(&[
"bat",
"i18n",
"worker",
"run",
"--provider",
"mock",
"--translation-fixture",
"/tmp/mock-translation.json",
"--worker-concurrency",
"8",
"--worker-max-attempts",
"4",
"--worker-lease-seconds",
"60",
"--worker-retry-backoff",
"0s",
"--worker-max-tasks",
"2",
"--worker-id",
"manual-run",
"--run-count",
"2",
"--interval",
"30s",
])
.unwrap();
assert_eq!(options.command, CliCommand::TranslationWorker);
assert_eq!(options.translation_provider.as_deref(), Some("mock"));
assert_eq!(
options.translation_fixture,
Some(PathBuf::from("/tmp/mock-translation.json"))
);
assert_eq!(options.worker_concurrency, 8);
assert_eq!(options.worker_max_attempts, 4);
assert_eq!(options.worker_lease_seconds, 60);
assert_eq!(options.worker_retry_backoff, Duration::ZERO);
assert_eq!(options.worker_max_tasks, Some(2));
assert_eq!(options.worker_id.as_deref(), Some("manual-run"));
assert_eq!(options.run_count, Some(2));
assert!(parse(&["bat", "i18n", "worker", "run"]).is_ok());
assert!(parse(&[
"bat",
"translation",
"worker",
"run",
"--translation-concurrency",
"256"
])
.is_ok());
assert!(parse(&["bat", "i18n", "worker", "run", "--worker-concurrency", "0"]).is_err());
assert!(parse(&[
"bat",
"i18n",
"worker",
"run",
"--worker-concurrency",
"257"
])
.is_err());
assert!(parse(&["bat", "i18n", "worker", "run", "--worker-max-attempts", "0"]).is_err());
assert!(parse(&[
"bat",
"i18n",
"worker",
"run",
"--worker-lease-seconds",
"0"
])
.is_err());
assert!(parse(&["bat", "i18n", "worker", "run", "--worker-max-tasks", "0"]).is_err());
assert!(parse(&["bat", "i18n", "run", "--worker-concurrency", "8"]).is_err());
assert!(parse(&["bat", "i18n", "worker", "run", "--provider", "unknown"]).is_err());
}
#[test]
fn translation_worker_env_defaults_apply() {
let options = parse_with_env(
&["bat", "i18n", "worker", "run"],
&[
("BAT_TRANSLATION_PROVIDER", "mock"),
("BAT_TRANSLATION_FIXTURE", "/tmp/fixture.json"),
("BAT_TRANSLATION_CONCURRENCY", "16"),
("BAT_TRANSLATION_MAX_ATTEMPTS", "5"),
("BAT_TRANSLATION_LEASE_SECONDS", "120"),
("BAT_TRANSLATION_RETRY_BACKOFF_SECONDS", "0"),
("BAT_TRANSLATION_MAX_TASKS", "7"),
("BAT_TRANSLATION_WORKER_ID", "env-worker"),
],
)
.unwrap();
assert_eq!(options.command, CliCommand::TranslationWorker);
assert_eq!(options.translation_provider.as_deref(), Some("mock"));
assert_eq!(
options.translation_fixture,
Some(PathBuf::from("/tmp/fixture.json"))
);
assert_eq!(options.worker_concurrency, 16);
assert_eq!(options.worker_max_attempts, 5);
assert_eq!(options.worker_lease_seconds, 120);
assert_eq!(options.worker_retry_backoff, Duration::ZERO);
assert_eq!(options.worker_max_tasks, Some(7));
assert_eq!(options.worker_id.as_deref(), Some("env-worker"));
assert!(parse_with_env(
&["bat", "i18n", "worker", "run"],
&[("BAT_TRANSLATION_CONCURRENCY", "257")]
)
.is_err());
assert!(parse_with_env(
&["bat", "i18n", "worker", "run"],
&[("BAT_TRANSLATION_MAX_ATTEMPTS", "0")]
)
.is_err());
}
#[test]
fn translation_worker_rpc_params_are_strict_and_accept_aliases() {
let default_config = rpc_translation_worker_config(None).unwrap();
assert_eq!(default_config.provider, TranslationProviderKind::Mock);
assert_eq!(default_config.concurrency, DEFAULT_TRANSLATION_CONCURRENCY);
let config = rpc_translation_worker_config(Some(&serde_json::json!({
"translation_provider": "mock",
"provider_fixture": "/tmp/mock-provider.json",
"worker_concurrency": 8,
"worker_max_attempts": 4,
"worker_lease_seconds": 60,
"worker_retry_backoff_seconds": 0,
"worker_max_tasks": 2,
"translation_worker_id": "rpc-worker"
})))
.unwrap();
assert_eq!(config.provider, TranslationProviderKind::Mock);
assert_eq!(
config.fixture_path,
Some(PathBuf::from("/tmp/mock-provider.json"))
);
assert_eq!(config.concurrency, 8);
assert_eq!(config.max_attempts, 4);
assert_eq!(config.lease_seconds, 60);
assert_eq!(config.retry_backoff, Duration::ZERO);
assert_eq!(config.max_tasks, Some(2));
assert_eq!(config.worker_id, "rpc-worker");
assert!(rpc_translation_worker_config(Some(&serde_json::json!([]))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"provider": "bad"}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": "8"}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": -1}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": 0}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"concurrency": 257}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"max_attempts": 0}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"lease_seconds": 0}))).is_err());
assert!(rpc_translation_worker_config(Some(&serde_json::json!({"max_tasks": 0}))).is_err());
}
#[test] #[test]
fn grouped_workflow_commands_use_short_top_level_aliases() { fn grouped_workflow_commands_use_short_top_level_aliases() {
let options = parse(&[ let options = parse(&[
+47 -5
View File
@@ -4,12 +4,14 @@ pub(super) const MAX_RETAINED_TASKS: usize = 64;
/// 每个任务保留的进度日志行数上限。 /// 每个任务保留的进度日志行数上限。
pub(super) const MAX_TASK_LOG_LINES: usize = 200; pub(super) const MAX_TASK_LOG_LINES: usize = 200;
/// 任务类型:目前覆盖官方同步、校验与 catalog 更新检查。 /// 任务类型:覆盖资源同步、校验、修复、翻译 worker 与 catalog 更新检查。
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum TaskKind { pub(super) enum TaskKind {
Sync, Sync,
Verify, Verify,
Repair, Repair,
/// 翻译 provider worker 轮次。
TranslationWorker,
/// catalog 更新检查:只做发现 + 拉取计划(dry-run),不下载不审计。 /// catalog 更新检查:只做发现 + 拉取计划(dry-run),不下载不审计。
Refresh, Refresh,
} }
@@ -20,6 +22,7 @@ impl TaskKind {
Self::Sync => RPC_METHOD_RESOURCE_SYNC, Self::Sync => RPC_METHOD_RESOURCE_SYNC,
Self::Verify => RPC_METHOD_RESOURCE_VERIFY, Self::Verify => RPC_METHOD_RESOURCE_VERIFY,
Self::Repair => RPC_METHOD_RESOURCE_REPAIR, Self::Repair => RPC_METHOD_RESOURCE_REPAIR,
Self::TranslationWorker => RPC_METHOD_TRANSLATION_WORKER_RUN,
Self::Refresh => RPC_METHOD_CATALOG_REFRESH, Self::Refresh => RPC_METHOD_CATALOG_REFRESH,
} }
} }
@@ -56,6 +59,10 @@ impl TaskKind {
config.repair = false; config.repair = false;
config.force = force; config.force = force;
} }
Self::TranslationWorker => {
config.dry_run = false;
config.force = false;
}
} }
config config
} }
@@ -155,6 +162,7 @@ fn task_kind_static(kind: &str) -> Option<&'static str> {
RPC_METHOD_RESOURCE_SYNC => Some(RPC_METHOD_RESOURCE_SYNC), RPC_METHOD_RESOURCE_SYNC => Some(RPC_METHOD_RESOURCE_SYNC),
RPC_METHOD_RESOURCE_VERIFY => Some(RPC_METHOD_RESOURCE_VERIFY), RPC_METHOD_RESOURCE_VERIFY => Some(RPC_METHOD_RESOURCE_VERIFY),
RPC_METHOD_RESOURCE_REPAIR => Some(RPC_METHOD_RESOURCE_REPAIR), RPC_METHOD_RESOURCE_REPAIR => Some(RPC_METHOD_RESOURCE_REPAIR),
RPC_METHOD_TRANSLATION_WORKER_RUN => Some(RPC_METHOD_TRANSLATION_WORKER_RUN),
RPC_METHOD_CATALOG_REFRESH => Some(RPC_METHOD_CATALOG_REFRESH), RPC_METHOD_CATALOG_REFRESH => Some(RPC_METHOD_CATALOG_REFRESH),
_ => None, _ => None,
} }
@@ -494,7 +502,9 @@ impl TaskStore {
/// 提交给任务 worker 的作业(配置已按任务类型派生完毕)。 /// 提交给任务 worker 的作业(配置已按任务类型派生完毕)。
pub(super) struct TaskJob { pub(super) struct TaskJob {
pub(super) id: String, pub(super) id: String,
pub(super) kind: TaskKind,
pub(super) config: OfficialUpdateConfig, pub(super) config: OfficialUpdateConfig,
pub(super) translation_worker_config: Option<TranslationWorkerConfig>,
/// 与任务记录共享的取消标志。 /// 与任务记录共享的取消标志。
pub(super) cancel: Arc<AtomicBool>, pub(super) cancel: Arc<AtomicBool>,
} }
@@ -528,6 +538,32 @@ pub(super) fn run_task_worker(
}); });
let cancel = Arc::clone(&job.cancel); let cancel = Arc::clone(&job.cancel);
let run_result = if job.kind == TaskKind::TranslationWorker {
let worker_config = job
.translation_worker_config
.as_ref()
.ok_or_else(|| anyhow::anyhow!("translation worker 任务缺少运行配置"));
worker_config.and_then(|worker_config| {
let _sync_guard = sync_lock
.lock()
.unwrap_or_else(|poison| poison.into_inner());
registry.append_log(&job.id, "[translation-worker] 开始执行".to_string());
registry.update(&job.id, |record| {
record.stage = Some("translation-worker".to_string());
record.message = Some("翻译 provider worker 正在执行".to_string());
});
let resource_root = active_official_resource_root(&job.config.output_root)?;
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()?;
runtime
.block_on(bat_infrastructure::run_translation_worker_at(
&resource_root,
worker_config,
))
.and_then(|report| serde_json::to_value(report).map_err(anyhow::Error::from))
})
} else {
let run_result = { let run_result = {
let _sync_guard = sync_lock let _sync_guard = sync_lock
.lock() .lock()
@@ -539,8 +575,10 @@ pub(super) fn run_task_worker(
service.run_with_progress_and_cancellation( service.run_with_progress_and_cancellation(
&job.config, &job.config,
|event| { |event| {
progress_registry progress_registry.append_log(
.append_log(&progress_id, format!("[{}] {}", event.stage, event.message)); &progress_id,
format!("[{}] {}", event.stage, event.message),
);
progress_registry.update(&progress_id, |record| { progress_registry.update(&progress_id, |record| {
record.stage = Some(event.stage.to_string()); record.stage = Some(event.stage.to_string());
record.message = Some(event.message.clone()); record.message = Some(event.message.clone());
@@ -552,12 +590,16 @@ pub(super) fn run_task_worker(
}, },
) )
}; };
run_result
.map(|report| serde_json::to_value(&report).map_err(anyhow::Error::from))
.and_then(|result| result)
};
match run_result { match run_result {
Ok(report) => registry.update(&job.id, |record| { Ok(report) => registry.update(&job.id, |record| {
record.status = "succeeded"; record.status = "succeeded";
record.finished_at = Some(unix_seconds_now()); record.finished_at = Some(unix_seconds_now());
record.result = serde_json::to_value(&report).ok(); record.result = Some(report);
}), }),
Err(error) => { Err(error) => {
let cancelled = cancel.load(Ordering::Relaxed); let cancelled = cancel.load(Ordering::Relaxed);
@@ -253,6 +253,43 @@ pub(super) fn run_translation_proofread(options: &CliOptions) -> anyhow::Result<
print_report(options.output_format, &report) print_report(options.output_format, &report)
} }
pub(super) fn run_translation_worker(options: &CliOptions) -> anyhow::Result<()> {
let resource_root = options
.resource_root
.clone()
.map(|path| lexical_absolute(&path).map_err(anyhow::Error::msg))
.transpose()?
.unwrap_or(active_official_resource_root(&options.config.output_root)?);
let provider = options
.translation_provider
.as_deref()
.unwrap_or(TranslationProviderKind::Mock.as_str());
let provider = TranslationProviderKind::parse(provider)
.ok_or_else(|| anyhow::anyhow!("i18n worker run 的 provider 无效:{provider}"))?;
let config = TranslationWorkerConfig {
provider,
fixture_path: options.translation_fixture.clone(),
concurrency: options.worker_concurrency,
max_attempts: options.worker_max_attempts,
lease_seconds: options.worker_lease_seconds,
retry_backoff: options.worker_retry_backoff,
max_tasks: options.worker_max_tasks,
worker_id: options
.worker_id
.clone()
.unwrap_or_else(|| format!("bat-worker-{}", std::process::id())),
};
config.validate()?;
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()?;
let report = runtime.block_on(bat_infrastructure::run_translation_worker_at(
&resource_root,
&config,
))?;
print_report(options.output_format, &report)
}
pub(super) fn run_repack(options: &CliOptions) -> anyhow::Result<()> { pub(super) fn run_repack(options: &CliOptions) -> anyhow::Result<()> {
let spec = options let spec = options
.repack_spec .repack_spec
+10 -2
View File
@@ -156,7 +156,7 @@ mod tests {
use super::*; use super::*;
use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::atomic::{AtomicUsize, Ordering};
use std::thread; use std::thread;
use std::time::Duration; use std::time::{Duration, Instant};
struct TestBackend { struct TestBackend {
active: AtomicUsize, active: AtomicUsize,
@@ -222,7 +222,15 @@ mod tests {
fn download(&self, task: usize) -> Result<Self::Output, Self::Error> { fn download(&self, task: usize) -> Result<Self::Output, Self::Error> {
if task == 0 { if task == 0 {
self.active.fetch_add(1, Ordering::SeqCst); self.active.fetch_add(1, Ordering::SeqCst);
thread::sleep(Duration::from_millis(20)); let deadline = Instant::now() + Duration::from_millis(500);
while self
.task_two_started_while_task_zero_active
.load(Ordering::SeqCst)
== 0
&& Instant::now() < deadline
{
thread::sleep(Duration::from_millis(1));
}
self.active.fetch_sub(1, Ordering::SeqCst); self.active.fetch_sub(1, Ordering::SeqCst);
} else { } else {
if task == 1 { if task == 1 {
+14 -3
View File
@@ -30,6 +30,7 @@ pub mod path_security;
pub mod release_flow; pub mod release_flow;
pub mod resources; pub mod resources;
pub mod translation_tasks; pub mod translation_tasks;
pub mod translation_worker;
pub mod translation_workflow; pub mod translation_workflow;
mod zip_validation; mod zip_validation;
@@ -138,11 +139,21 @@ pub use translation_tasks::{
build_translation_handoff, read_translation_handoff_at, sync_translation_task_repository_at, build_translation_handoff, read_translation_handoff_at, sync_translation_task_repository_at,
write_translation_handoff_at, PersistedTranslationTask, PersistedTranslationTaskState, write_translation_handoff_at, PersistedTranslationTask, PersistedTranslationTaskState,
ProviderRun, ProviderRunStatus, SqliteTranslationTaskRepository, TranslationHandoff, ProviderRun, ProviderRunStatus, SqliteTranslationTaskRepository, TranslationHandoff,
TranslationJob, TranslationJobStatus, TranslationTaskStatus, TranslationTaskSyncReport, TranslationJob, TranslationJobStatus, TranslationTaskFailure, TranslationTaskStatus,
TranslationUnit, TranslationUnitStatus, TRANSLATION_HANDOFF_FILE, TranslationTaskSyncReport, TranslationTaskUnitResult, TranslationUnit, TranslationUnitStatus,
TRANSLATION_HANDOFF_SCHEMA_VERSION, TRANSLATION_TASK_REPOSITORY_FILE, TRANSLATION_HANDOFF_FILE, TRANSLATION_HANDOFF_SCHEMA_VERSION, TRANSLATION_TASK_REPOSITORY_FILE,
TRANSLATION_TASK_SCHEMA_VERSION, TRANSLATION_TASK_SCHEMA_VERSION,
}; };
pub use translation_worker::{
run_translation_worker_at, run_translation_worker_with_provider, CrowdinProvider,
MockTranslationProvider, TranslationProvider, TranslationProviderFailureClass,
TranslationProviderKind, TranslationProviderRequest, TranslationProviderResponse,
TranslationProviderUnit, TranslationProviderUnitResult, TranslationWorkerConfig,
TranslationWorkerFailure, TranslationWorkerReport, DEFAULT_TRANSLATION_CONCURRENCY,
DEFAULT_TRANSLATION_LEASE_SECONDS, DEFAULT_TRANSLATION_MAX_ATTEMPTS,
DEFAULT_TRANSLATION_RETRY_BACKOFF, MAX_TRANSLATION_CONCURRENCY, MIN_TRANSLATION_CONCURRENCY,
MOCK_TRANSLATION_FIXTURE_VERSION,
};
pub use translation_workflow::{ pub use translation_workflow::{
export_translation_workbench, get_translation_entry, localized_text_asset_patches, export_translation_workbench, get_translation_entry, localized_text_asset_patches,
read_translation_workbench, repack_bundle, set_translation, unset_translation, read_translation_workbench, repack_bundle, set_translation, unset_translation,
+653 -14
View File
@@ -14,15 +14,15 @@ use crate::path_security::{
}; };
use bat_core::Result; use bat_core::Result;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use sqlx::sqlite::{SqliteConnectOptions, SqlitePoolOptions}; use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePoolOptions};
use sqlx::{QueryBuilder, Sqlite, SqlitePool}; use sqlx::{QueryBuilder, Sqlite, SqlitePool};
use std::collections::{BTreeMap, BTreeSet}; use std::collections::{BTreeMap, BTreeSet};
use std::path::Path; use std::path::Path;
use std::str::FromStr; use std::str::FromStr;
use std::time::{SystemTime, UNIX_EPOCH}; use std::time::{Duration, SystemTime, UNIX_EPOCH};
/// SQLite schema version for durable translation task state. /// SQLite schema version for durable translation task state.
pub const TRANSLATION_TASK_SCHEMA_VERSION: u32 = 1; pub const TRANSLATION_TASK_SCHEMA_VERSION: u32 = 2;
const TRANSLATION_TASK_SCHEMA_COMPONENT: &str = "translation_tasks"; const TRANSLATION_TASK_SCHEMA_COMPONENT: &str = "translation_tasks";
/// SQLite file name stored under a published official release root. /// SQLite file name stored under a published official release root.
pub const TRANSLATION_TASK_REPOSITORY_FILE: &str = "translation-tasks.sqlite"; pub const TRANSLATION_TASK_REPOSITORY_FILE: &str = "translation-tasks.sqlite";
@@ -147,6 +147,26 @@ pub struct TranslationUnit {
pub status: TranslationUnitStatus, pub status: TranslationUnitStatus,
/// Provider failure diagnostic, when present. /// Provider failure diagnostic, when present.
pub failure_reason: Option<String>, pub failure_reason: Option<String>,
/// TextUnit-level provider results associated with this resource task.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub translation_results: Vec<TranslationTaskUnitResult>,
}
/// One provider result linked to an immutable TextUnit.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct TranslationTaskUnitResult {
/// Stable TextUnit ID from `official-textunit-index.json`.
pub unit_id: String,
/// Source text submitted to the provider.
pub source_text: String,
/// Provider-produced or human-supplied translation.
pub translated_text: String,
/// Provider identifier.
pub provider: String,
/// Provider run that produced this result.
pub provider_run_id: String,
/// Result persistence time.
pub translated_unix_seconds: u64,
} }
/// One provider execution associated with one or more translation units. /// One provider execution associated with one or more translation units.
@@ -203,7 +223,16 @@ pub fn build_translation_handoff(
let state = persisted let state = persisted
.get(task.task_id.as_str()) .get(task.task_id.as_str())
.copied() .copied()
.map(|task| (task.task_status, task.failure_reason.clone(), task.attempt_count, task.provider_run_id.clone())) .map(|task| {
(
task.task_status,
task.failure_reason.clone(),
task.attempt_count,
task.provider_run_id.clone(),
task.provider.clone(),
task.translation_results.clone(),
)
})
.unwrap_or_else(|| { .unwrap_or_else(|| {
( (
initial_task_status(task), initial_task_status(task),
@@ -214,6 +243,8 @@ pub fn build_translation_handoff(
}, },
0, 0,
None, None,
None,
Vec::new(),
) )
}); });
let unit_status = match state.0 { let unit_status = match state.0 {
@@ -234,6 +265,7 @@ pub fn build_translation_handoff(
text_unit_formats: task.text_unit_formats.clone(), text_unit_formats: task.text_unit_formats.clone(),
status: unit_status, status: unit_status,
failure_reason: state.1.clone(), failure_reason: state.1.clone(),
translation_results: state.5.clone(),
}; };
if let Some(provider_run_id) = state.3 { if let Some(provider_run_id) = state.3 {
let run = provider_runs let run = provider_runs
@@ -241,7 +273,7 @@ pub fn build_translation_handoff(
.or_insert_with(|| ProviderRun { .or_insert_with(|| ProviderRun {
provider_run_id, provider_run_id,
job_id: job_id.clone(), job_id: job_id.clone(),
provider: "worker".to_string(), provider: state.4.unwrap_or_else(|| "worker".to_string()),
status: ProviderRunStatus::Queued, status: ProviderRunStatus::Queued,
unit_ids: Vec::new(), unit_ids: Vec::new(),
attempt_count: 0, attempt_count: 0,
@@ -403,6 +435,26 @@ pub struct PersistedTranslationTask {
/// Provider-side run identifier, if known. /// Provider-side run identifier, if known.
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
pub provider_run_id: Option<String>, pub provider_run_id: Option<String>,
/// Provider-produced results keyed by TextUnit ID.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub translation_results: Vec<TranslationTaskUnitResult>,
/// Provider identifier used by the latest run.
#[serde(skip_serializing_if = "Option::is_none")]
pub provider: Option<String>,
/// Worker currently holding the lease.
#[serde(skip_serializing_if = "Option::is_none")]
pub lease_owner: Option<String>,
/// Lease expiry as Unix seconds.
#[serde(skip_serializing_if = "Option::is_none")]
pub lease_expires_unix_seconds: Option<u64>,
/// Stable failure classification.
#[serde(skip_serializing_if = "Option::is_none")]
pub failure_class: Option<String>,
/// Whether the latest failure can be retried.
pub failure_retryable: bool,
/// Earliest Unix time at which a failed task may be claimed again.
#[serde(skip_serializing_if = "Option::is_none")]
pub next_attempt_unix_seconds: Option<u64>,
} }
/// Mutable persistence metadata associated with an official TextUnit task. /// Mutable persistence metadata associated with an official TextUnit task.
@@ -422,6 +474,39 @@ pub struct PersistedTranslationTaskState {
pub completed_unix_seconds: Option<u64>, pub completed_unix_seconds: Option<u64>,
/// Provider-side run identifier, if known. /// Provider-side run identifier, if known.
pub provider_run_id: Option<String>, pub provider_run_id: Option<String>,
/// Provider-produced results keyed by TextUnit ID.
pub translation_results: Vec<TranslationTaskUnitResult>,
/// Provider identifier used by the latest run.
pub provider: Option<String>,
/// Worker currently holding the lease.
pub lease_owner: Option<String>,
/// Lease expiry as Unix seconds.
pub lease_expires_unix_seconds: Option<u64>,
/// Stable failure classification.
pub failure_class: Option<String>,
/// Whether the latest failure can be retried.
pub failure_retryable: bool,
/// Earliest Unix time at which a failed task may be claimed again.
pub next_attempt_unix_seconds: Option<u64>,
}
/// Provider failure data used to atomically release a task lease.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TranslationTaskFailure {
/// Task being failed.
pub task_id: String,
/// Worker that owns the lease.
pub worker_id: String,
/// Provider run associated with the lease.
pub provider_run_id: String,
/// Stable failure classification.
pub failure_class: String,
/// Redacted diagnostic message.
pub failure_reason: String,
/// Whether a future attempt may retry the task.
pub retryable: bool,
/// Earliest retry time, when retryable.
pub next_attempt_unix_seconds: Option<u64>,
} }
/// Result of synchronizing an immutable release queue into SQLite. /// Result of synchronizing an immutable release queue into SQLite.
@@ -463,7 +548,9 @@ impl SqliteTranslationTaskRepository {
let options = SqliteConnectOptions::from_str(&format!("sqlite://{}", path.display())) let options = SqliteConnectOptions::from_str(&format!("sqlite://{}", path.display()))
.map_err(|error| bat_core::Error::Other(error.into()))? .map_err(|error| bat_core::Error::Other(error.into()))?
.create_if_missing(create_if_missing); .create_if_missing(create_if_missing)
.journal_mode(SqliteJournalMode::Wal)
.busy_timeout(Duration::from_secs(30));
let pool = SqlitePoolOptions::new() let pool = SqlitePoolOptions::new()
.max_connections(1) .max_connections(1)
.connect_with(options) .connect_with(options)
@@ -510,7 +597,14 @@ impl SqliteTranslationTaskRepository {
created_unix_seconds INTEGER NOT NULL, created_unix_seconds INTEGER NOT NULL,
updated_unix_seconds INTEGER NOT NULL, updated_unix_seconds INTEGER NOT NULL,
completed_unix_seconds INTEGER, completed_unix_seconds INTEGER,
provider_run_id TEXT provider_run_id TEXT,
translation_results_json TEXT NOT NULL DEFAULT '[]',
provider TEXT,
lease_owner TEXT,
lease_expires_unix_seconds INTEGER,
failure_class TEXT,
failure_retryable INTEGER NOT NULL DEFAULT 0,
next_attempt_unix_seconds INTEGER
) )
"#, "#,
) )
@@ -573,6 +667,37 @@ impl SqliteTranslationTaskRepository {
) )
.await?; .await?;
ensure_column(&self.pool, "translation_tasks", "provider_run_id", "TEXT").await?; ensure_column(&self.pool, "translation_tasks", "provider_run_id", "TEXT").await?;
ensure_column(
&self.pool,
"translation_tasks",
"translation_results_json",
"TEXT NOT NULL DEFAULT '[]'",
)
.await?;
ensure_column(&self.pool, "translation_tasks", "provider", "TEXT").await?;
ensure_column(&self.pool, "translation_tasks", "lease_owner", "TEXT").await?;
ensure_column(
&self.pool,
"translation_tasks",
"lease_expires_unix_seconds",
"INTEGER",
)
.await?;
ensure_column(&self.pool, "translation_tasks", "failure_class", "TEXT").await?;
ensure_column(
&self.pool,
"translation_tasks",
"failure_retryable",
"INTEGER NOT NULL DEFAULT 0",
)
.await?;
ensure_column(
&self.pool,
"translation_tasks",
"next_attempt_unix_seconds",
"INTEGER",
)
.await?;
let current: Option<i64> = let current: Option<i64> =
sqlx::query_scalar("SELECT version FROM schema_migrations WHERE component = ?1") sqlx::query_scalar("SELECT version FROM schema_migrations WHERE component = ?1")
@@ -615,7 +740,10 @@ impl SqliteTranslationTaskRepository {
let existing: Option<ExistingTaskRow> = sqlx::query_as( let existing: Option<ExistingTaskRow> = sqlx::query_as(
r#" r#"
SELECT task_json, worker_status, failure_reason, attempt_count, SELECT task_json, worker_status, failure_reason, attempt_count,
created_unix_seconds, completed_unix_seconds, provider_run_id created_unix_seconds, completed_unix_seconds, provider_run_id,
translation_results_json, provider, lease_owner,
lease_expires_unix_seconds, failure_class, failure_retryable,
next_attempt_unix_seconds
FROM translation_tasks FROM translation_tasks
WHERE task_id = ?1 WHERE task_id = ?1
"#, "#,
@@ -642,11 +770,31 @@ impl SqliteTranslationTaskRepository {
created, created,
completed, completed,
provider_run_id, provider_run_id,
translation_results_json,
provider,
lease_owner,
lease_expires,
failure_class,
failure_retryable,
next_attempt,
)) = existing )) = existing
{ {
let immutable_unchanged = previous_task_json == task_json; let immutable_unchanged = previous_task_json == task_json;
let (status, failure_reason, attempt_count, created, completed, provider_run_id) = let (
if immutable_unchanged { status,
failure_reason,
attempt_count,
created,
completed,
provider_run_id,
translation_results_json,
provider,
lease_owner,
lease_expires,
failure_class,
failure_retryable,
next_attempt,
) = if immutable_unchanged {
( (
worker_status, worker_status,
failure_reason, failure_reason,
@@ -654,6 +802,13 @@ impl SqliteTranslationTaskRepository {
created, created,
completed, completed,
provider_run_id, provider_run_id,
translation_results_json,
provider,
lease_owner,
lease_expires,
failure_class,
failure_retryable,
next_attempt,
) )
} else { } else {
report.refreshed_count += 1; report.refreshed_count += 1;
@@ -664,6 +819,13 @@ impl SqliteTranslationTaskRepository {
now, now,
None, None,
None, None,
"[]".to_string(),
None,
None,
None,
None,
0_i64,
None,
) )
}; };
if immutable_unchanged { if immutable_unchanged {
@@ -677,7 +839,11 @@ impl SqliteTranslationTaskRepository {
text_unit_formats_json = ?8, task_json = ?9, text_unit_formats_json = ?8, task_json = ?9,
worker_status = ?10, failure_reason = ?11, attempt_count = ?12, worker_status = ?10, failure_reason = ?11, attempt_count = ?12,
created_unix_seconds = ?13, updated_unix_seconds = ?14, created_unix_seconds = ?13, updated_unix_seconds = ?14,
completed_unix_seconds = ?15, provider_run_id = ?16 completed_unix_seconds = ?15, provider_run_id = ?16,
translation_results_json = ?17, provider = ?18,
lease_owner = ?19, lease_expires_unix_seconds = ?20,
failure_class = ?21, failure_retryable = ?22,
next_attempt_unix_seconds = ?23
WHERE task_id = ?1 WHERE task_id = ?1
"#, "#,
) )
@@ -697,6 +863,13 @@ impl SqliteTranslationTaskRepository {
.bind(now) .bind(now)
.bind(completed) .bind(completed)
.bind(provider_run_id) .bind(provider_run_id)
.bind(translation_results_json)
.bind(provider)
.bind(lease_owner)
.bind(lease_expires)
.bind(failure_class)
.bind(failure_retryable)
.bind(next_attempt)
.execute(&mut *transaction) .execute(&mut *transaction)
.await .await
.map_err(db_error)?; .map_err(db_error)?;
@@ -770,7 +943,9 @@ impl SqliteTranslationTaskRepository {
r#" r#"
SELECT task_json, worker_status, failure_reason, attempt_count, SELECT task_json, worker_status, failure_reason, attempt_count,
created_unix_seconds, updated_unix_seconds, created_unix_seconds, updated_unix_seconds,
completed_unix_seconds, provider_run_id completed_unix_seconds, provider_run_id, translation_results_json,
provider, lease_owner, lease_expires_unix_seconds, failure_class,
failure_retryable, next_attempt_unix_seconds
FROM translation_tasks FROM translation_tasks
ORDER BY task_id ORDER BY task_id
"#, "#,
@@ -794,6 +969,247 @@ impl SqliteTranslationTaskRepository {
Ok(self.list(query).await?.len() as u64) Ok(self.list(query).await?.len() as u64)
} }
/// Requeues tasks whose worker lease expired.
pub async fn recover_expired_leases(&self, now_unix_seconds: u64) -> Result<u64> {
let now = i64::try_from(now_unix_seconds).unwrap_or(i64::MAX);
let result = sqlx::query(
r#"
UPDATE translation_tasks
SET worker_status = CASE
WHEN attempt_count > 0 THEN 'failed'
ELSE 'queued'
END,
failure_reason = 'worker lease expired; task recovered',
failure_class = 'lease_expired',
failure_retryable = 1,
next_attempt_unix_seconds = ?1,
updated_unix_seconds = ?1,
lease_owner = NULL,
lease_expires_unix_seconds = NULL
WHERE worker_status = 'running'
AND lease_expires_unix_seconds IS NOT NULL
AND lease_expires_unix_seconds <= ?1
"#,
)
.bind(now)
.execute(&self.pool)
.await
.map_err(db_error)?;
Ok(result.rows_affected())
}
/// Claims one queued or retryable failed task and assigns an exclusive
/// lease to one worker.
pub async fn claim_next(
&self,
worker_id: &str,
provider: &str,
lease_seconds: u64,
max_attempts: u32,
) -> Result<Option<PersistedTranslationTask>> {
if worker_id.trim().is_empty() || provider.trim().is_empty() {
return Err(bat_core::Error::InvalidArgument(
"translation worker_id/provider 不能为空".to_string(),
));
}
if max_attempts == 0 {
return Err(bat_core::Error::InvalidArgument(
"translation worker max_attempts 必须大于 0".to_string(),
));
}
if lease_seconds == 0 {
return Err(bat_core::Error::InvalidArgument(
"translation worker lease_seconds 必须大于 0".to_string(),
));
}
let now = unix_seconds_now_i64();
let lease_expires = now.saturating_add(i64::try_from(lease_seconds).unwrap_or(i64::MAX));
let mut transaction = self.pool.begin().await.map_err(db_error)?;
let row: Option<TranslationTaskRow> = sqlx::query_as(
r#"
SELECT task_json, worker_status, failure_reason, attempt_count,
created_unix_seconds, updated_unix_seconds,
completed_unix_seconds, provider_run_id, translation_results_json,
provider, lease_owner, lease_expires_unix_seconds, failure_class,
failure_retryable, next_attempt_unix_seconds
FROM translation_tasks
WHERE queue_status = 'queued_offline'
AND (
worker_status = 'queued'
OR (
worker_status = 'failed'
AND failure_retryable = 1
AND attempt_count < ?1
AND (next_attempt_unix_seconds IS NULL
OR next_attempt_unix_seconds <= ?2)
)
)
AND (lease_expires_unix_seconds IS NULL OR lease_expires_unix_seconds <= ?2)
ORDER BY task_id
LIMIT 1
"#,
)
.bind(i64::from(max_attempts))
.bind(now)
.fetch_optional(&mut *transaction)
.await
.map_err(db_error)?;
let Some(row) = row else {
transaction.commit().await.map_err(db_error)?;
return Ok(None);
};
let current = PersistedTranslationTask::from_row(row)?;
let attempt_count = current.attempt_count.saturating_add(1);
let provider_run_id = format!(
"{provider}:{}:attempt-{attempt_count}",
current.task.task_id
);
let result = sqlx::query(
r#"
UPDATE translation_tasks
SET worker_status = 'running',
failure_reason = NULL,
attempt_count = ?2,
updated_unix_seconds = ?3,
completed_unix_seconds = NULL,
provider_run_id = ?4,
provider = ?5,
lease_owner = ?6,
lease_expires_unix_seconds = ?7,
failure_class = NULL,
failure_retryable = 0,
next_attempt_unix_seconds = NULL
WHERE task_id = ?1
AND (
worker_status = 'queued'
OR (
worker_status = 'failed'
AND failure_retryable = 1
AND attempt_count < ?8
AND (next_attempt_unix_seconds IS NULL
OR next_attempt_unix_seconds <= ?3)
)
)
AND queue_status = 'queued_offline'
AND (lease_expires_unix_seconds IS NULL OR lease_expires_unix_seconds <= ?3)
"#,
)
.bind(&current.task.task_id)
.bind(i64::from(attempt_count))
.bind(now)
.bind(&provider_run_id)
.bind(provider)
.bind(worker_id)
.bind(lease_expires)
.bind(i64::from(max_attempts))
.execute(&mut *transaction)
.await
.map_err(db_error)?;
if result.rows_affected() != 1 {
transaction.rollback().await.map_err(db_error)?;
return Ok(None);
}
transaction.commit().await.map_err(db_error)?;
self.find(&current.task.task_id).await.map(Some)
}
/// Stores an idempotent provider result while the worker still owns its lease.
pub async fn complete_claim(
&self,
task_id: &str,
worker_id: &str,
provider_run_id: &str,
provider: &str,
translation_results: &[TranslationTaskUnitResult],
) -> Result<PersistedTranslationTask> {
let results_json = serde_json::to_string(translation_results)
.map_err(|error| bat_core::Error::Serialization(error.to_string()))?;
let now = unix_seconds_now_i64();
let result = sqlx::query(
r#"
UPDATE translation_tasks
SET worker_status = 'completed',
failure_reason = NULL,
updated_unix_seconds = ?2,
completed_unix_seconds = ?2,
provider_run_id = ?3,
provider = ?4,
translation_results_json = ?5,
lease_owner = NULL,
lease_expires_unix_seconds = NULL,
failure_class = NULL,
failure_retryable = 0,
next_attempt_unix_seconds = NULL
WHERE task_id = ?1
AND worker_status = 'running'
AND lease_owner = ?6
AND provider_run_id = ?3
"#,
)
.bind(task_id)
.bind(now)
.bind(provider_run_id)
.bind(provider)
.bind(results_json)
.bind(worker_id)
.execute(&self.pool)
.await
.map_err(db_error)?;
if result.rows_affected() != 1 {
return Err(bat_core::Error::InvalidArgument(format!(
"翻译任务 {} 的 lease 已失效,拒绝写入 provider 结果",
task_id
)));
}
self.find(task_id).await
}
/// Records one provider failure and releases the worker lease.
pub async fn fail_claim(
&self,
failure: TranslationTaskFailure,
) -> Result<PersistedTranslationTask> {
let now = unix_seconds_now_i64();
let next_attempt = failure
.next_attempt_unix_seconds
.map(|value| i64::try_from(value).unwrap_or(i64::MAX));
let result = sqlx::query(
r#"
UPDATE translation_tasks
SET worker_status = 'failed',
failure_reason = ?2,
updated_unix_seconds = ?3,
lease_owner = NULL,
lease_expires_unix_seconds = NULL,
failure_class = ?4,
failure_retryable = ?5,
next_attempt_unix_seconds = ?6
WHERE task_id = ?1
AND worker_status = 'running'
AND lease_owner = ?7
AND provider_run_id = ?8
"#,
)
.bind(&failure.task_id)
.bind(&failure.failure_reason)
.bind(now)
.bind(&failure.failure_class)
.bind(if failure.retryable { 1_i64 } else { 0_i64 })
.bind(next_attempt)
.bind(&failure.worker_id)
.bind(&failure.provider_run_id)
.execute(&self.pool)
.await
.map_err(db_error)?;
if result.rows_affected() != 1 {
return Err(bat_core::Error::InvalidArgument(format!(
"翻译任务 {} 的 lease 已失效,拒绝写入 provider 失败状态",
failure.task_id
)));
}
self.find(&failure.task_id).await
}
/// Updates provider state and returns the durable task record. /// Updates provider state and returns the durable task record.
pub async fn update_status( pub async fn update_status(
&self, &self,
@@ -818,7 +1234,10 @@ impl SqliteTranslationTaskRepository {
UPDATE translation_tasks UPDATE translation_tasks
SET worker_status = ?2, failure_reason = ?3, attempt_count = ?4, SET worker_status = ?2, failure_reason = ?3, attempt_count = ?4,
updated_unix_seconds = ?5, completed_unix_seconds = ?6, updated_unix_seconds = ?5, completed_unix_seconds = ?6,
provider_run_id = COALESCE(?7, provider_run_id) provider_run_id = COALESCE(?7, provider_run_id),
lease_owner = NULL, lease_expires_unix_seconds = NULL,
failure_class = NULL, failure_retryable = 0,
next_attempt_unix_seconds = NULL
WHERE task_id = ?1 WHERE task_id = ?1
"#, "#,
) )
@@ -841,7 +1260,9 @@ impl SqliteTranslationTaskRepository {
r#" r#"
SELECT task_json, worker_status, failure_reason, attempt_count, SELECT task_json, worker_status, failure_reason, attempt_count,
created_unix_seconds, updated_unix_seconds, created_unix_seconds, updated_unix_seconds,
completed_unix_seconds, provider_run_id completed_unix_seconds, provider_run_id, translation_results_json,
provider, lease_owner, lease_expires_unix_seconds, failure_class,
failure_retryable, next_attempt_unix_seconds
FROM translation_tasks FROM translation_tasks
WHERE task_id = ?1 WHERE task_id = ?1
"#, "#,
@@ -864,6 +1285,13 @@ type ExistingTaskRow = (
i64, i64,
Option<i64>, Option<i64>,
Option<String>, Option<String>,
String,
Option<String>,
Option<String>,
Option<i64>,
Option<String>,
i64,
Option<i64>,
); );
type TranslationTaskRow = ( type TranslationTaskRow = (
@@ -875,6 +1303,13 @@ type TranslationTaskRow = (
i64, i64,
Option<i64>, Option<i64>,
Option<String>, Option<String>,
String,
Option<String>,
Option<String>,
Option<i64>,
Option<String>,
i64,
Option<i64>,
); );
impl PersistedTranslationTask { impl PersistedTranslationTask {
@@ -889,6 +1324,13 @@ impl PersistedTranslationTask {
updated_unix_seconds: state.updated_unix_seconds, updated_unix_seconds: state.updated_unix_seconds,
completed_unix_seconds: state.completed_unix_seconds, completed_unix_seconds: state.completed_unix_seconds,
provider_run_id: state.provider_run_id, provider_run_id: state.provider_run_id,
translation_results: state.translation_results,
provider: state.provider,
lease_owner: state.lease_owner,
lease_expires_unix_seconds: state.lease_expires_unix_seconds,
failure_class: state.failure_class,
failure_retryable: state.failure_retryable,
next_attempt_unix_seconds: state.next_attempt_unix_seconds,
} }
} }
@@ -910,6 +1352,13 @@ impl PersistedTranslationTask {
updated_unix_seconds: generated_unix_seconds, updated_unix_seconds: generated_unix_seconds,
completed_unix_seconds: None, completed_unix_seconds: None,
provider_run_id: None, provider_run_id: None,
translation_results: Vec::new(),
provider: None,
lease_owner: None,
lease_expires_unix_seconds: None,
failure_class: None,
failure_retryable: false,
next_attempt_unix_seconds: None,
}, },
) )
} }
@@ -924,12 +1373,21 @@ impl PersistedTranslationTask {
updated_unix_seconds, updated_unix_seconds,
completed_unix_seconds, completed_unix_seconds,
provider_run_id, provider_run_id,
translation_results_json,
provider,
lease_owner,
lease_expires_unix_seconds,
failure_class,
failure_retryable,
next_attempt_unix_seconds,
) = row; ) = row;
let task = serde_json::from_str(&task_json) let task = serde_json::from_str(&task_json)
.map_err(|error| bat_core::Error::Serialization(error.to_string()))?; .map_err(|error| bat_core::Error::Serialization(error.to_string()))?;
let task_status = TranslationTaskStatus::parse(&worker_status).ok_or_else(|| { let task_status = TranslationTaskStatus::parse(&worker_status).ok_or_else(|| {
bat_core::Error::Serialization(format!("未知翻译任务 worker 状态:{worker_status}")) bat_core::Error::Serialization(format!("未知翻译任务 worker 状态:{worker_status}"))
})?; })?;
let translation_results = serde_json::from_str(&translation_results_json)
.map_err(|error| bat_core::Error::Serialization(error.to_string()))?;
Ok(Self { Ok(Self {
task, task,
task_status, task_status,
@@ -951,6 +1409,25 @@ impl PersistedTranslationTask {
}) })
.transpose()?, .transpose()?,
provider_run_id, provider_run_id,
translation_results,
provider,
lease_owner,
lease_expires_unix_seconds: lease_expires_unix_seconds
.map(|value| {
u64::try_from(value).map_err(|_| {
bat_core::Error::Serialization("翻译任务 lease 时间无效".to_string())
})
})
.transpose()?,
failure_class,
failure_retryable: failure_retryable != 0,
next_attempt_unix_seconds: next_attempt_unix_seconds
.map(|value| {
u64::try_from(value).map_err(|_| {
bat_core::Error::Serialization("翻译任务 next_attempt 时间无效".to_string())
})
})
.transpose()?,
}) })
} }
} }
@@ -1219,6 +1696,140 @@ mod tests {
assert_eq!(retrievable[0].attempt_count, 1); assert_eq!(retrievable[0].attempt_count, 1);
} }
#[tokio::test]
async fn sqlite_translation_tasks_recover_expired_leases_for_retry() {
let temp = tempfile::TempDir::new().unwrap();
let repository =
SqliteTranslationTaskRepository::new(temp.path().join("translation-tasks.sqlite"))
.await
.unwrap();
let queue = queue(vec![task(
"task-a",
"Bundles/a.bundle",
OfficialTextUnitTaskStatus::QueuedOffline,
Some(OfficialParseStatus::Parsed),
None,
)]);
repository.sync_queue(&queue).await.unwrap();
let claimed = repository
.claim_next("worker-a", "mock", 1, 3)
.await
.unwrap()
.unwrap();
assert_eq!(claimed.task_status, TranslationTaskStatus::Running);
assert_eq!(claimed.lease_owner.as_deref(), Some("worker-a"));
force_expire_lease(&repository, "task-a").await;
let recovered = repository
.recover_expired_leases(unix_seconds_now())
.await
.unwrap();
assert_eq!(recovered, 1);
let recovered_task = repository.find("task-a").await.unwrap();
assert_eq!(recovered_task.task_status, TranslationTaskStatus::Failed);
assert_eq!(
recovered_task.failure_reason.as_deref(),
Some("worker lease expired; task recovered")
);
assert_eq!(
recovered_task.failure_class.as_deref(),
Some("lease_expired")
);
assert!(recovered_task.failure_retryable);
assert_eq!(recovered_task.lease_owner, None);
let reclaimed = repository
.claim_next("worker-b", "mock", 30, 3)
.await
.unwrap()
.unwrap();
assert_eq!(reclaimed.task_status, TranslationTaskStatus::Running);
assert_eq!(reclaimed.attempt_count, 2);
assert_eq!(reclaimed.lease_owner.as_deref(), Some("worker-b"));
}
#[tokio::test]
async fn sqlite_translation_tasks_reject_stale_worker_writes() {
let temp = tempfile::TempDir::new().unwrap();
let repository =
SqliteTranslationTaskRepository::new(temp.path().join("translation-tasks.sqlite"))
.await
.unwrap();
let queue = queue(vec![task(
"task-a",
"Bundles/a.bundle",
OfficialTextUnitTaskStatus::QueuedOffline,
Some(OfficialParseStatus::Parsed),
None,
)]);
repository.sync_queue(&queue).await.unwrap();
let first_claim = repository
.claim_next("worker-a", "mock", 1, 3)
.await
.unwrap()
.unwrap();
let first_run = first_claim.provider_run_id.clone().unwrap();
force_expire_lease(&repository, "task-a").await;
repository
.recover_expired_leases(unix_seconds_now())
.await
.unwrap();
let second_claim = repository
.claim_next("worker-b", "mock", 30, 3)
.await
.unwrap()
.unwrap();
let second_run = second_claim.provider_run_id.clone().unwrap();
let result = TranslationTaskUnitResult {
unit_id: "unit-a".to_string(),
source_text: "source".to_string(),
translated_text: "translated".to_string(),
provider: "mock".to_string(),
provider_run_id: second_run.clone(),
translated_unix_seconds: 1,
};
assert!(repository
.complete_claim(
"task-a",
"worker-a",
&first_run,
"mock",
std::slice::from_ref(&result)
)
.await
.is_err());
let completed = repository
.complete_claim(
"task-a",
"worker-b",
&second_run,
"mock",
std::slice::from_ref(&result),
)
.await
.unwrap();
assert_eq!(completed.task_status, TranslationTaskStatus::Completed);
assert_eq!(completed.translation_results, vec![result.clone()]);
assert!(repository
.fail_claim(TranslationTaskFailure {
task_id: "task-a".to_string(),
worker_id: "worker-a".to_string(),
provider_run_id: first_run,
failure_class: "network".to_string(),
failure_reason: "late failure".to_string(),
retryable: true,
next_attempt_unix_seconds: Some(2),
})
.await
.is_err());
let task = repository.find("task-a").await.unwrap();
assert_eq!(task.task_status, TranslationTaskStatus::Completed);
assert_eq!(task.translation_results, vec![result]);
}
#[test] #[test]
fn translation_handoff_maps_worker_state_and_preserves_provider_progress() { fn translation_handoff_maps_worker_state_and_preserves_provider_progress() {
let queue = queue(vec![ let queue = queue(vec![
@@ -1254,6 +1865,13 @@ mod tests {
updated_unix_seconds: 124, updated_unix_seconds: 124,
completed_unix_seconds: None, completed_unix_seconds: None,
provider_run_id: Some("provider-run-1".to_string()), provider_run_id: Some("provider-run-1".to_string()),
translation_results: Vec::new(),
provider: Some("fixture".to_string()),
lease_owner: None,
lease_expires_unix_seconds: None,
failure_class: None,
failure_retryable: false,
next_attempt_unix_seconds: None,
}, },
PersistedTranslationTask { PersistedTranslationTask {
task: queue.tasks[1].clone(), task: queue.tasks[1].clone(),
@@ -1264,6 +1882,13 @@ mod tests {
updated_unix_seconds: 125, updated_unix_seconds: 125,
completed_unix_seconds: Some(125), completed_unix_seconds: Some(125),
provider_run_id: Some("provider-run-1".to_string()), provider_run_id: Some("provider-run-1".to_string()),
translation_results: Vec::new(),
provider: Some("fixture".to_string()),
lease_owner: None,
lease_expires_unix_seconds: None,
failure_class: None,
failure_retryable: false,
next_attempt_unix_seconds: None,
}, },
]; ];
@@ -1286,6 +1911,20 @@ mod tests {
assert_eq!(handoff.provider_runs[0].attempt_count, 2); assert_eq!(handoff.provider_runs[0].attempt_count, 2);
} }
async fn force_expire_lease(repository: &SqliteTranslationTaskRepository, task_id: &str) {
sqlx::query(
r#"
UPDATE translation_tasks
SET lease_expires_unix_seconds = 0
WHERE task_id = ?1
"#,
)
.bind(task_id)
.execute(&repository.pool)
.await
.unwrap();
}
#[test] #[test]
fn translation_handoff_file_round_trips_with_version_check() { fn translation_handoff_file_round_trips_with_version_check() {
let temp = tempfile::TempDir::new().unwrap(); let temp = tempfile::TempDir::new().unwrap();
File diff suppressed because it is too large Load Diff
+167 -1
View File
@@ -6,6 +6,7 @@ import (
"errors" "errors"
"io" "io"
"net/http" "net/http"
"net/url"
"strconv" "strconv"
"strings" "strings"
@@ -36,6 +37,8 @@ func (s *Server) handleAdminIndex(w http.ResponseWriter, r *http.Request) {
"/v1/resources", "/v1/resources",
"/openapi.yaml", "/openapi.yaml",
"/admin/schedules", "/admin/schedules",
"/admin/translation/tasks",
"/admin/translation/handoff",
}, },
Controls: []string{ Controls: []string{
"/admin/control/reload", "/admin/control/reload",
@@ -50,6 +53,7 @@ func (s *Server) handleAdminIndex(w http.ResponseWriter, r *http.Request) {
"/admin/control/schedule-remove", "/admin/control/schedule-remove",
"/admin/control/schedule-run", "/admin/control/schedule-run",
"/admin/control/translation-task-update", "/admin/control/translation-task-update",
"/admin/control/translation-worker-run",
"/admin/control/translation-proofread", "/admin/control/translation-proofread",
}, },
} }
@@ -82,6 +86,10 @@ func (s *Server) handleAdminControl(w http.ResponseWriter, r *http.Request) {
s.handleAdminTranslationTaskUpdate(w, r) s.handleAdminTranslationTaskUpdate(w, r)
return return
} }
if action == "translation-worker-run" {
s.handleAdminTranslationWorkerRun(w, r)
return
}
if action == "translation-proofread" { if action == "translation-proofread" {
s.handleAdminTranslationProofread(w, r) s.handleAdminTranslationProofread(w, r)
return return
@@ -187,6 +195,34 @@ func (s *Server) handleAdminTranslationTaskUpdate(w http.ResponseWriter, r *http
}) })
} }
func (s *Server) handleAdminTranslationWorkerRun(w http.ResponseWriter, r *http.Request) {
backend, ok := s.backend.(TranslationBackend)
if !ok || backend == nil {
writeErrorJSON(w, http.StatusServiceUnavailable, "translation_backend_unavailable", "Rust bat translation backend is unavailable")
return
}
var params backendrpc.TranslationWorkerRunParams
if !decodeAdminTranslationJSON(w, r, &params) {
return
}
if err := validateTranslationWorkerRunParams(params); err != nil {
writeErrorJSON(w, http.StatusBadRequest, "invalid_translation_params", err.Error())
return
}
result, err := backend.TranslationWorkerRun(r.Context(), params)
if err != nil {
s.writeControlBackendError(w, "translation-worker-run", err)
return
}
writeNoStoreJSON(w, http.StatusAccepted, AdminControlResponse{
Service: "bat-api",
Action: "translation-worker-run",
RPCMethod: "translation.worker.run",
Status: "accepted",
Result: result,
})
}
func (s *Server) handleAdminTranslationProofread(w http.ResponseWriter, r *http.Request) { func (s *Server) handleAdminTranslationProofread(w http.ResponseWriter, r *http.Request) {
backend, ok := s.backend.(TranslationBackend) backend, ok := s.backend.(TranslationBackend)
if !ok || backend == nil { if !ok || backend == nil {
@@ -207,6 +243,63 @@ func (s *Server) handleAdminTranslationProofread(w http.ResponseWriter, r *http.
}) })
} }
func (s *Server) handleAdminTranslationTasks(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet && r.Method != http.MethodHead {
writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
return
}
if !s.requireAdminToken(w, r) {
return
}
backend, ok := s.backend.(TranslationBackend)
if !ok || backend == nil {
writeErrorJSON(w, http.StatusServiceUnavailable, "translation_backend_unavailable", "Rust bat translation backend is unavailable")
return
}
params, err := translationTaskListParams(r)
if err != nil {
writeErrorJSON(w, http.StatusBadRequest, "invalid_translation_query", err.Error())
return
}
result, err := backend.TranslationTasks(r.Context(), params)
if err != nil {
s.writeControlBackendError(w, "translation-tasks", err)
return
}
if r.Method == http.MethodHead {
w.Header().Set("Cache-Control", "no-store")
w.WriteHeader(http.StatusOK)
return
}
writeNoStoreJSON(w, http.StatusOK, result)
}
func (s *Server) handleAdminTranslationHandoff(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet && r.Method != http.MethodHead {
writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
return
}
if !s.requireAdminToken(w, r) {
return
}
backend, ok := s.backend.(TranslationBackend)
if !ok || backend == nil {
writeErrorJSON(w, http.StatusServiceUnavailable, "translation_backend_unavailable", "Rust bat translation backend is unavailable")
return
}
result, err := backend.TranslationHandoff(r.Context())
if err != nil {
s.writeControlBackendError(w, "translation-handoff", err)
return
}
if r.Method == http.MethodHead {
w.Header().Set("Cache-Control", "no-store")
w.WriteHeader(http.StatusOK)
return
}
writeNoStoreJSON(w, http.StatusOK, result)
}
func (s *Server) handleAdminSchedules(w http.ResponseWriter, r *http.Request) { func (s *Server) handleAdminSchedules(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet && r.Method != http.MethodHead { if r.Method != http.MethodGet && r.Method != http.MethodHead {
writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed") writeErrorJSON(w, http.StatusMethodNotAllowed, "method_not_allowed", "method not allowed")
@@ -254,6 +347,63 @@ func scheduleListParams(r *http.Request) (backendrpc.ScheduleListParams, error)
return params, nil return params, nil
} }
func translationTaskListParams(r *http.Request) (backendrpc.TranslationTaskListParams, error) {
query := r.URL.Query()
params := backendrpc.TranslationTaskListParams{
TaskID: strings.TrimSpace(query.Get("task_id")),
ReleaseID: firstTrimmedQuery(query, "release_id", "official_release_id"),
Destination: strings.TrimSpace(query.Get("destination")),
PathPattern: strings.TrimSpace(query.Get("path_pattern")),
ArchiveEntry: strings.TrimSpace(query.Get("archive_entry")),
Status: firstTrimmedQuery(query, "status", "task_status"),
WorkerStatus: strings.TrimSpace(query.Get("worker_status")),
ParseStatus: strings.TrimSpace(query.Get("parse_status")),
Format: firstTrimmedQuery(query, "format", "text_unit_format"),
}
if raw := strings.TrimSpace(query.Get("offset")); raw != "" {
offset, err := strconv.ParseUint(raw, 10, 64)
if err != nil {
return backendrpc.TranslationTaskListParams{}, errors.New("offset must be a non-negative integer")
}
params.Offset = &offset
}
if raw := strings.TrimSpace(query.Get("limit")); raw != "" {
limit, err := strconv.ParseUint(raw, 10, 64)
if err != nil || limit == 0 || limit > 1000 {
return backendrpc.TranslationTaskListParams{}, errors.New("limit must be in 1..=1000")
}
params.Limit = &limit
}
if raw := strings.TrimSpace(query.Get("has_reason")); raw != "" {
hasReason, err := strconv.ParseBool(raw)
if err != nil {
return backendrpc.TranslationTaskListParams{}, errors.New("has_reason must be a boolean")
}
params.HasReason = &hasReason
}
if raw := strings.TrimSpace(query.Get("has_failure_reason")); raw != "" {
hasFailureReason, err := strconv.ParseBool(raw)
if err != nil {
return backendrpc.TranslationTaskListParams{}, errors.New("has_failure_reason must be a boolean")
}
params.HasFailureReason = &hasFailureReason
}
return params, nil
}
func firstTrimmedQuery(query url.Values, keys ...string) string {
for _, key := range keys {
values := query[key]
if len(values) == 0 {
continue
}
if value := strings.TrimSpace(values[0]); value != "" {
return value
}
}
return ""
}
func (s *Server) handleAdminScheduleControl(w http.ResponseWriter, r *http.Request, action string) { func (s *Server) handleAdminScheduleControl(w http.ResponseWriter, r *http.Request, action string) {
backend, ok := s.backend.(ScheduleBackend) backend, ok := s.backend.(ScheduleBackend)
if !ok || backend == nil { if !ok || backend == nil {
@@ -325,7 +475,23 @@ func decodeAdminScheduleJSON(w http.ResponseWriter, r *http.Request, target any)
} }
func decodeAdminTranslationJSON(w http.ResponseWriter, r *http.Request, target any) bool { func decodeAdminTranslationJSON(w http.ResponseWriter, r *http.Request, target any) bool {
return decodeAdminJSON(w, r, target, "invalid_translation_params", "translation task request") return decodeAdminJSON(w, r, target, "invalid_translation_params", "translation request")
}
func validateTranslationWorkerRunParams(params backendrpc.TranslationWorkerRunParams) error {
if params.Concurrency != nil && (*params.Concurrency < 1 || *params.Concurrency > 256) {
return errors.New("translation worker concurrency must be in 1..=256")
}
if params.MaxAttempts != nil && *params.MaxAttempts == 0 {
return errors.New("translation worker max_attempts must be greater than 0")
}
if params.LeaseSeconds != nil && *params.LeaseSeconds == 0 {
return errors.New("translation worker lease_seconds must be greater than 0")
}
if params.MaxTasks != nil && *params.MaxTasks == 0 {
return errors.New("translation worker max_tasks must be greater than 0")
}
return nil
} }
func decodeAdminJSON(w http.ResponseWriter, r *http.Request, target any, errorCode string, subject string) bool { func decodeAdminJSON(w http.ResponseWriter, r *http.Request, target any, errorCode string, subject string) bool {
+122
View File
@@ -589,6 +589,7 @@ func (f *fakeBackend) ResourceManifest(ctx context.Context, offset int, limit in
type controlBackend struct { type controlBackend struct {
*fakeBackend *fakeBackend
calls []string calls []string
translationTaskListParams []backendrpc.TranslationTaskListParams
} }
func (b *controlBackend) DaemonReload(ctx context.Context) (*backendrpc.Ack, error) { func (b *controlBackend) DaemonReload(ctx context.Context) (*backendrpc.Ack, error) {
@@ -631,6 +632,38 @@ func (b *controlBackend) TranslationTaskUpdate(ctx context.Context, params backe
return json.RawMessage(`{"task_status":"` + params.Status + `"}`), nil return json.RawMessage(`{"task_status":"` + params.Status + `"}`), nil
} }
func (b *controlBackend) TranslationTasks(ctx context.Context, params backendrpc.TranslationTaskListParams) (json.RawMessage, error) {
b.calls = append(b.calls, "translation.tasks")
b.translationTaskListParams = append(b.translationTaskListParams, params)
return json.RawMessage(`{"tasks":[{"task_id":"textunit/v-current/Scenario","worker_status":"failed","failure_reason":"provider rejected payload"}]}`), nil
}
func (b *controlBackend) TranslationHandoff(ctx context.Context) (json.RawMessage, error) {
b.calls = append(b.calls, "translation.handoff")
return json.RawMessage(`{"jobs":[],"provider_runs":[]}`), nil
}
func (b *controlBackend) TranslationWorkerRun(ctx context.Context, params backendrpc.TranslationWorkerRunParams) (*backendrpc.TranslationWorkerRunResult, error) {
b.calls = append(b.calls, "translation.worker.run")
concurrency := uint64(8)
if params.Concurrency != nil {
concurrency = *params.Concurrency
}
return &backendrpc.TranslationWorkerRunResult{
TaskID: "task-translation-worker-1",
Kind: "translation.worker.run",
Worker: backendrpc.TranslationWorkerConfig{
Provider: params.Provider,
FixturePath: params.FixturePath,
Concurrency: concurrency,
MaxAttempts: 3,
LeaseSeconds: 300,
RetryBackoffSeconds: 5,
WorkerID: params.WorkerID,
},
}, nil
}
func (b *controlBackend) TranslationProofread(ctx context.Context) (json.RawMessage, error) { func (b *controlBackend) TranslationProofread(ctx context.Context) (json.RawMessage, error) {
b.calls = append(b.calls, "translation.proofread") b.calls = append(b.calls, "translation.proofread")
return json.RawMessage(`{"translation_workflow_status":"manual_proofreading"}`), nil return json.RawMessage(`{"translation_workflow_status":"manual_proofreading"}`), nil
@@ -743,6 +776,81 @@ func TestAdminScheduleEndpointsProxyAuthenticatedRequests(t *testing.T) {
} }
} }
func TestAdminTranslationQueryEndpointsProxyAuthenticatedRequests(t *testing.T) {
cfg := DefaultConfig()
cfg.AuthToken = "translation-token"
if err := cfg.Normalize(); err != nil {
t.Fatal(err)
}
backend := &controlBackend{fakeBackend: &fakeBackend{}}
s := NewServer(cfg, backend, nil)
request := httptest.NewRequest(
http.MethodGet,
"/admin/translation/tasks?offset=10&limit=25&task_id=textunit%2Fv-current%2FScenario&release_id=v-current&destination=MediaResources%2FGameData%2FScenario.zip&path_pattern=Scenario&archive_entry=ScenarioExcelTable.json&status=queued_offline&worker_status=failed&parse_status=ok&format=json&has_reason=true&has_failure_reason=false",
nil,
)
request.Header.Set("Authorization", "Bearer translation-token")
recorder := httptest.NewRecorder()
s.Handler().ServeHTTP(recorder, request)
if recorder.Code != http.StatusOK {
t.Fatalf("tasks status=%d body=%s", recorder.Code, recorder.Body.String())
}
if !json.Valid(recorder.Body.Bytes()) || !strings.Contains(recorder.Body.String(), "failure_reason") {
t.Fatalf("tasks body=%s", recorder.Body.String())
}
if len(backend.translationTaskListParams) != 1 {
t.Fatalf("translation task query params=%#v", backend.translationTaskListParams)
}
params := backend.translationTaskListParams[0]
if params.Offset == nil || *params.Offset != 10 ||
params.Limit == nil || *params.Limit != 25 ||
params.TaskID != "textunit/v-current/Scenario" ||
params.ReleaseID != "v-current" ||
params.Destination != "MediaResources/GameData/Scenario.zip" ||
params.PathPattern != "Scenario" ||
params.ArchiveEntry != "ScenarioExcelTable.json" ||
params.Status != "queued_offline" ||
params.WorkerStatus != "failed" ||
params.ParseStatus != "ok" ||
params.Format != "json" ||
params.HasReason == nil || !*params.HasReason ||
params.HasFailureReason == nil || *params.HasFailureReason {
t.Fatalf("params=%#v", params)
}
request = httptest.NewRequest(http.MethodGet, "/admin/translation/handoff", nil)
request.Header.Set("Authorization", "Bearer translation-token")
recorder = httptest.NewRecorder()
s.Handler().ServeHTTP(recorder, request)
if recorder.Code != http.StatusOK {
t.Fatalf("handoff status=%d body=%s", recorder.Code, recorder.Body.String())
}
if !strings.Contains(recorder.Body.String(), "provider_runs") {
t.Fatalf("handoff body=%s", recorder.Body.String())
}
if len(backend.calls) < 2 ||
backend.calls[len(backend.calls)-2] != "translation.tasks" ||
backend.calls[len(backend.calls)-1] != "translation.handoff" {
t.Fatalf("calls=%v", backend.calls)
}
request = httptest.NewRequest(http.MethodGet, "/admin/translation/tasks?limit=0", nil)
request.Header.Set("Authorization", "Bearer translation-token")
recorder = httptest.NewRecorder()
s.Handler().ServeHTTP(recorder, request)
if recorder.Code != http.StatusBadRequest {
t.Fatalf("invalid task query status=%d body=%s", recorder.Code, recorder.Body.String())
}
request = httptest.NewRequest(http.MethodGet, "/admin/translation/tasks", nil)
recorder = httptest.NewRecorder()
s.Handler().ServeHTTP(recorder, request)
if recorder.Code != http.StatusUnauthorized {
t.Fatalf("unauthenticated tasks status=%d body=%s", recorder.Code, recorder.Body.String())
}
}
func TestDiscoverCallsStatusBeforeDoctor(t *testing.T) { func TestDiscoverCallsStatusBeforeDoctor(t *testing.T) {
root := fixtureRoot(t) root := fixtureRoot(t)
bytes := uint64(20) bytes := uint64(20)
@@ -1281,6 +1389,11 @@ func TestOpenAPIAndAdminReservedEndpoints(t *testing.T) {
if len(admin.Controls) == 0 || admin.Controls[0] != "/admin/control/reload" { if len(admin.Controls) == 0 || admin.Controls[0] != "/admin/control/reload" {
t.Fatalf("admin controls=%v", admin.Controls) t.Fatalf("admin controls=%v", admin.Controls)
} }
links := strings.Join(admin.Links, "\n")
if !strings.Contains(links, "/admin/translation/tasks") ||
!strings.Contains(links, "/admin/translation/handoff") {
t.Fatalf("admin links=%v", admin.Links)
}
} }
func TestAdminControlForwardsAllowlistedActions(t *testing.T) { func TestAdminControlForwardsAllowlistedActions(t *testing.T) {
@@ -1305,6 +1418,7 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) {
{name: "repair", action: "repair", rpcMethod: "resource.repair", call: "resource.repair"}, {name: "repair", action: "repair", rpcMethod: "resource.repair", call: "resource.repair"},
{name: "catalog refresh", action: "catalog-refresh", rpcMethod: "catalog.refresh", call: "catalog.refresh"}, {name: "catalog refresh", action: "catalog-refresh", rpcMethod: "catalog.refresh", call: "catalog.refresh"},
{name: "translation task update", action: "translation-task-update", body: `{"task_id":"textunit/v-current/Scenario","status":"failed","failure_reason":"provider rejected payload","provider_run_id":"provider-run-1"}`, rpcMethod: "translation.task.update", call: "translation.task.update"}, {name: "translation task update", action: "translation-task-update", body: `{"task_id":"textunit/v-current/Scenario","status":"failed","failure_reason":"provider rejected payload","provider_run_id":"provider-run-1"}`, rpcMethod: "translation.task.update", call: "translation.task.update"},
{name: "translation worker run", action: "translation-worker-run", body: `{"provider":"mock","concurrency":8,"max_tasks":2,"retry_backoff_seconds":0,"worker_id":"dashboard-worker"}`, rpcMethod: "translation.worker.run", call: "translation.worker.run"},
{name: "translation proofread", action: "translation-proofread", rpcMethod: "translation.proofread", call: "translation.proofread"}, {name: "translation proofread", action: "translation-proofread", rpcMethod: "translation.proofread", call: "translation.proofread"},
} }
for _, tc := range tests { for _, tc := range tests {
@@ -1336,6 +1450,14 @@ func TestAdminControlForwardsAllowlistedActions(t *testing.T) {
if recorder.Code != http.StatusBadRequest { if recorder.Code != http.StatusBadRequest {
t.Fatalf("invalid translation update status=%d body=%s", recorder.Code, recorder.Body.String()) t.Fatalf("invalid translation update status=%d body=%s", recorder.Code, recorder.Body.String())
} }
request = httptest.NewRequest(http.MethodPost, "/admin/control/translation-worker-run", strings.NewReader(`{"concurrency":0}`))
request.Header.Set("Authorization", "Bearer control-token")
recorder = httptest.NewRecorder()
s.Handler().ServeHTTP(recorder, request)
if recorder.Code != http.StatusBadRequest {
t.Fatalf("invalid translation worker status=%d body=%s", recorder.Code, recorder.Body.String())
}
} }
func TestAdminControlRejectsUnauthenticatedDangerousAndUnsupportedActions(t *testing.T) { func TestAdminControlRejectsUnauthenticatedDangerousAndUnsupportedActions(t *testing.T) {
+103 -1
View File
@@ -139,6 +139,80 @@ paths:
description: Missing or invalid admin token. description: Missing or invalid admin token.
"503": "503":
description: Rust bat schedule backend is unavailable. description: Rust bat schedule backend is unavailable.
/admin/translation/tasks:
get:
summary: List Rust-owned translation task worker status
parameters:
- name: offset
in: query
schema:
type: integer
minimum: 0
- name: limit
in: query
schema:
type: integer
minimum: 1
maximum: 1000
- name: task_id
in: query
schema:
type: string
- name: release_id
in: query
schema:
type: string
- name: destination
in: query
schema:
type: string
- name: archive_entry
in: query
schema:
type: string
- name: status
in: query
schema:
type: string
- name: worker_status
in: query
schema:
type: string
- name: parse_status
in: query
schema:
type: string
- name: format
in: query
schema:
type: string
- name: has_reason
in: query
schema:
type: boolean
- name: has_failure_reason
in: query
schema:
type: boolean
responses:
"200":
description: Current translation task JSON report from Rust bat.
"400":
description: Invalid translation task query.
"401":
description: Missing or invalid admin token.
"503":
description: Rust bat translation backend is unavailable.
/admin/translation/handoff:
get:
summary: Read Rust-owned translation handoff state
responses:
"200":
description: Current translation handoff JSON report from Rust bat.
"401":
description: Missing or invalid admin token.
"503":
description: Rust bat translation backend is unavailable.
/admin/control/{action}: /admin/control/{action}:
post: post:
summary: Forward an allowlisted control or schedule action to Rust bat summary: Forward an allowlisted control or schedule action to Rust bat
@@ -148,7 +222,7 @@ paths:
required: true required: true
schema: schema:
type: string type: string
enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, translation-task-update, translation-proofread] enum: [reload, refresh, restart, sync, verify, repair, catalog-refresh, schedule-add, schedule-update, schedule-remove, schedule-run, translation-task-update, translation-worker-run, translation-proofread]
requestBody: requestBody:
required: false required: false
content: content:
@@ -199,6 +273,34 @@ paths:
type: string type: string
provider_run_id: provider_run_id:
type: string type: string
provider:
type: string
enum: [mock, crowdin]
fixture_path:
type: string
concurrency:
type: integer
format: int64
minimum: 1
maximum: 256
max_attempts:
type: integer
format: int64
minimum: 1
lease_seconds:
type: integer
format: int64
minimum: 1
retry_backoff_seconds:
type: integer
format: int64
minimum: 0
max_tasks:
type: integer
format: int64
minimum: 1
worker_id:
type: string
responses: responses:
"202": "202":
description: Rust bat accepted the control request. description: Rust bat accepted the control request.
+15
View File
@@ -54,6 +54,9 @@ type ScheduleBackend interface {
// by the dashboard. It does not create arbitrary translation jobs. // by the dashboard. It does not create arbitrary translation jobs.
type TranslationBackend interface { type TranslationBackend interface {
TranslationTaskUpdate(ctx context.Context, params backendrpc.TranslationTaskUpdateParams) (json.RawMessage, error) TranslationTaskUpdate(ctx context.Context, params backendrpc.TranslationTaskUpdateParams) (json.RawMessage, error)
TranslationTasks(ctx context.Context, params backendrpc.TranslationTaskListParams) (json.RawMessage, error)
TranslationHandoff(ctx context.Context) (json.RawMessage, error)
TranslationWorkerRun(ctx context.Context, params backendrpc.TranslationWorkerRunParams) (*backendrpc.TranslationWorkerRunResult, error)
TranslationProofread(ctx context.Context) (json.RawMessage, error) TranslationProofread(ctx context.Context) (json.RawMessage, error)
} }
@@ -117,6 +120,18 @@ func (r RPCClient) TranslationTaskUpdate(ctx context.Context, params backendrpc.
return r.Client.TranslationTaskUpdate(ctx, params) return r.Client.TranslationTaskUpdate(ctx, params)
} }
func (r RPCClient) TranslationTasks(ctx context.Context, params backendrpc.TranslationTaskListParams) (json.RawMessage, error) {
return r.Client.TranslationTasks(ctx, params)
}
func (r RPCClient) TranslationHandoff(ctx context.Context) (json.RawMessage, error) {
return r.Client.TranslationHandoff(ctx)
}
func (r RPCClient) TranslationWorkerRun(ctx context.Context, params backendrpc.TranslationWorkerRunParams) (*backendrpc.TranslationWorkerRunResult, error) {
return r.Client.TranslationWorkerRun(ctx, params)
}
func (r RPCClient) TranslationProofread(ctx context.Context) (json.RawMessage, error) { func (r RPCClient) TranslationProofread(ctx context.Context) (json.RawMessage, error) {
return r.Client.TranslationProofread(ctx) return r.Client.TranslationProofread(ctx)
} }
+4
View File
@@ -62,6 +62,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc(launcherHostPath("/api/launcher/resource/bootstrap.json"), s.handleLauncherBootstrap) mux.HandleFunc(launcherHostPath("/api/launcher/resource/bootstrap.json"), s.handleLauncherBootstrap)
mux.HandleFunc("/openapi.yaml", s.handleOpenAPI) mux.HandleFunc("/openapi.yaml", s.handleOpenAPI)
mux.HandleFunc("/admin/schedules", s.handleAdminSchedules) mux.HandleFunc("/admin/schedules", s.handleAdminSchedules)
mux.HandleFunc("/admin/translation/tasks", s.handleAdminTranslationTasks)
mux.HandleFunc("/admin/translation/handoff", s.handleAdminTranslationHandoff)
mux.HandleFunc("/admin/control/", s.handleAdminControl) mux.HandleFunc("/admin/control/", s.handleAdminControl)
mux.HandleFunc("/admin/", s.handleAdminIndex) mux.HandleFunc("/admin/", s.handleAdminIndex)
mux.HandleFunc("/"+ServerInfoHost+"/", s.handleServerInfoCDN) mux.HandleFunc("/"+ServerInfoHost+"/", s.handleServerInfoCDN)
@@ -127,6 +129,8 @@ func (s *Server) handleRoot(w http.ResponseWriter, r *http.Request) {
"/openapi.yaml", "/openapi.yaml",
"/admin/", "/admin/",
"/admin/schedules", "/admin/schedules",
"/admin/translation/tasks",
"/admin/translation/handoff",
"/admin/control/{action}", "/admin/control/{action}",
}, },
}) })
+66
View File
@@ -273,6 +273,58 @@ type TranslationTaskUpdateParams struct {
ProviderRunID string `json:"provider_run_id,omitempty"` ProviderRunID string `json:"provider_run_id,omitempty"`
} }
// TranslationTaskListParams filters the Rust-owned translation task queue.
// The result shape is intentionally left as Rust JSON so Go does not duplicate
// the translation task schema.
type TranslationTaskListParams struct {
Offset *uint64 `json:"offset,omitempty"`
Limit *uint64 `json:"limit,omitempty"`
TaskID string `json:"task_id,omitempty"`
ReleaseID string `json:"release_id,omitempty"`
Destination string `json:"destination,omitempty"`
PathPattern string `json:"path_pattern,omitempty"`
ArchiveEntry string `json:"archive_entry,omitempty"`
Status string `json:"status,omitempty"`
WorkerStatus string `json:"worker_status,omitempty"`
ParseStatus string `json:"parse_status,omitempty"`
Format string `json:"format,omitempty"`
HasReason *bool `json:"has_reason,omitempty"`
HasFailureReason *bool `json:"has_failure_reason,omitempty"`
}
// TranslationWorkerRunParams starts one Rust-owned provider worker task.
// Pointer numeric fields preserve explicit zeroes so Rust can reject invalid
// dashboard input instead of receiving omitted defaults.
type TranslationWorkerRunParams struct {
Provider string `json:"provider,omitempty"`
FixturePath string `json:"fixture_path,omitempty"`
Concurrency *uint64 `json:"concurrency,omitempty"`
MaxAttempts *uint64 `json:"max_attempts,omitempty"`
LeaseSeconds *uint64 `json:"lease_seconds,omitempty"`
RetryBackoffSeconds *uint64 `json:"retry_backoff_seconds,omitempty"`
MaxTasks *uint64 `json:"max_tasks,omitempty"`
WorkerID string `json:"worker_id,omitempty"`
}
// TranslationWorkerConfig mirrors the accepted worker config returned by Rust.
type TranslationWorkerConfig struct {
Provider string `json:"provider"`
FixturePath string `json:"fixture_path,omitempty"`
Concurrency uint64 `json:"concurrency"`
MaxAttempts uint64 `json:"max_attempts"`
LeaseSeconds uint64 `json:"lease_seconds"`
RetryBackoffSeconds uint64 `json:"retry_backoff_seconds"`
MaxTasks *uint64 `json:"max_tasks,omitempty"`
WorkerID string `json:"worker_id"`
}
// TranslationWorkerRunResult is returned when translation.worker.run is queued.
type TranslationWorkerRunResult struct {
TaskID string `json:"task_id"`
Kind string `json:"kind"`
Worker TranslationWorkerConfig `json:"worker"`
}
// TranslationProofread marks the current localized workflow as manual // TranslationProofread marks the current localized workflow as manual
// proofreading. Rust owns the persisted localized state. // proofreading. Rust owns the persisted localized state.
func (c *Client) TranslationProofread(ctx context.Context) (json.RawMessage, error) { func (c *Client) TranslationProofread(ctx context.Context) (json.RawMessage, error) {
@@ -502,6 +554,20 @@ func (c *Client) TranslationTaskUpdate(ctx context.Context, params TranslationTa
return c.rawData(ctx, "translation.task.update", params) return c.rawData(ctx, "translation.task.update", params)
} }
func (c *Client) TranslationTasks(ctx context.Context, params TranslationTaskListParams) (json.RawMessage, error) {
return c.rawData(ctx, "translation.tasks", params)
}
func (c *Client) TranslationHandoff(ctx context.Context) (json.RawMessage, error) {
return c.rawData(ctx, "translation.handoff", nil)
}
func (c *Client) TranslationWorkerRun(ctx context.Context, params TranslationWorkerRunParams) (*TranslationWorkerRunResult, error) {
var out TranslationWorkerRunResult
_, err := c.Call(ctx, "translation.worker.run", params, &out)
return &out, err
}
func (c *Client) CatalogStatus(ctx context.Context) (json.RawMessage, error) { func (c *Client) CatalogStatus(ctx context.Context) (json.RawMessage, error) {
return c.rawData(ctx, "catalog.status", nil) return c.rawData(ctx, "catalog.status", nil)
} }
+157
View File
@@ -336,6 +336,94 @@ func TestScheduleRunSendsScopeAndMaxRuns(t *testing.T) {
} }
} }
func TestTranslationTasksSendsQueryParams(t *testing.T) {
offset := uint64(10)
limit := uint64(25)
hasReason := true
hasFailureReason := false
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
if req.Method != "translation.tasks" {
t.Fatalf("method = %s", req.Method)
}
var params TranslationTaskListParams
if err := json.Unmarshal(req.Params, &params); err != nil {
t.Fatalf("decode params: %v", err)
}
if params.Offset == nil || *params.Offset != offset ||
params.Limit == nil || *params.Limit != limit ||
params.TaskID != "textunit/v-current/Scenario" ||
params.ReleaseID != "v-current" ||
params.Destination != "MediaResources/GameData/Scenario.zip" ||
params.PathPattern != "Scenario" ||
params.ArchiveEntry != "ScenarioExcelTable.json" ||
params.Status != "queued_offline" ||
params.WorkerStatus != "failed" ||
params.ParseStatus != "ok" ||
params.Format != "json" ||
params.HasReason == nil || *params.HasReason != hasReason ||
params.HasFailureReason == nil || *params.HasFailureReason != hasFailureReason {
t.Fatalf("params = %#v", params)
}
return testResponse{
Result: testEnvelope{
OK: true,
Status: "ok",
RequestID: "req-test-translation-tasks",
Data: map[string]any{"tasks": []any{}},
},
}
})
raw, err := client.TranslationTasks(context.Background(), TranslationTaskListParams{
Offset: &offset,
Limit: &limit,
TaskID: "textunit/v-current/Scenario",
ReleaseID: "v-current",
Destination: "MediaResources/GameData/Scenario.zip",
PathPattern: "Scenario",
ArchiveEntry: "ScenarioExcelTable.json",
Status: "queued_offline",
WorkerStatus: "failed",
ParseStatus: "ok",
Format: "json",
HasReason: &hasReason,
HasFailureReason: &hasFailureReason,
})
if err != nil {
t.Fatalf("TranslationTasks error: %v", err)
}
if !json.Valid(raw) {
t.Fatalf("invalid raw JSON: %s", string(raw))
}
}
func TestTranslationHandoffUsesRustMethod(t *testing.T) {
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
if req.Method != "translation.handoff" {
t.Fatalf("method = %s", req.Method)
}
if string(req.Params) != "null" {
t.Fatalf("params = %s, want null", req.Params)
}
return testResponse{
Result: testEnvelope{
OK: true,
Status: "ok",
RequestID: "req-test-translation-handoff",
Data: map[string]any{"jobs": []any{}},
},
}
})
raw, err := client.TranslationHandoff(context.Background())
if err != nil {
t.Fatalf("TranslationHandoff error: %v", err)
}
if !json.Valid(raw) {
t.Fatalf("invalid raw JSON: %s", string(raw))
}
}
func TestTranslationTaskUpdateSendsWorkerParams(t *testing.T) { func TestTranslationTaskUpdateSendsWorkerParams(t *testing.T) {
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
if req.Method != "translation.task.update" { if req.Method != "translation.task.update" {
@@ -375,6 +463,75 @@ func TestTranslationTaskUpdateSendsWorkerParams(t *testing.T) {
} }
} }
func TestTranslationWorkerRunSendsProviderConfig(t *testing.T) {
var concurrency uint64 = 8
var maxAttempts uint64 = 4
var leaseSeconds uint64 = 60
var retryBackoff uint64 = 0
var maxTasks uint64 = 2
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
if req.Method != "translation.worker.run" {
t.Fatalf("method = %s", req.Method)
}
var params TranslationWorkerRunParams
if err := json.Unmarshal(req.Params, &params); err != nil {
t.Fatalf("decode params: %v", err)
}
if params.Provider != "mock" ||
params.FixturePath != "/tmp/mock-provider.json" ||
params.Concurrency == nil || *params.Concurrency != concurrency ||
params.MaxAttempts == nil || *params.MaxAttempts != maxAttempts ||
params.LeaseSeconds == nil || *params.LeaseSeconds != leaseSeconds ||
params.RetryBackoffSeconds == nil || *params.RetryBackoffSeconds != retryBackoff ||
params.MaxTasks == nil || *params.MaxTasks != maxTasks ||
params.WorkerID != "dashboard-worker" {
t.Fatalf("params = %#v", params)
}
return testResponse{
Result: testEnvelope{
OK: true,
Status: "accepted",
RequestID: "req-test-translation-worker",
Data: map[string]any{
"task_id": "task-worker-1",
"kind": "translation.worker.run",
"worker": map[string]any{
"provider": "mock",
"fixture_path": "/tmp/mock-provider.json",
"concurrency": concurrency,
"max_attempts": maxAttempts,
"lease_seconds": leaseSeconds,
"retry_backoff_seconds": retryBackoff,
"max_tasks": maxTasks,
"worker_id": "dashboard-worker",
},
},
},
}
})
result, err := client.TranslationWorkerRun(context.Background(), TranslationWorkerRunParams{
Provider: "mock",
FixturePath: "/tmp/mock-provider.json",
Concurrency: &concurrency,
MaxAttempts: &maxAttempts,
LeaseSeconds: &leaseSeconds,
RetryBackoffSeconds: &retryBackoff,
MaxTasks: &maxTasks,
WorkerID: "dashboard-worker",
})
if err != nil {
t.Fatalf("TranslationWorkerRun error: %v", err)
}
if result.TaskID != "task-worker-1" ||
result.Kind != "translation.worker.run" ||
result.Worker.Concurrency != concurrency ||
result.Worker.RetryBackoffSeconds != retryBackoff ||
result.Worker.MaxTasks == nil || *result.Worker.MaxTasks != maxTasks {
t.Fatalf("unexpected result: %#v", result)
}
}
func TestTranslationProofreadUsesRustMethod(t *testing.T) { func TestTranslationProofreadUsesRustMethod(t *testing.T) {
client := newTestClient(t, func(t *testing.T, req testRequest) testResponse { client := newTestClient(t, func(t *testing.T, req testRequest) testResponse {
if req.Method != "translation.proofread" { if req.Method != "translation.proofread" {
+23 -13
View File
@@ -1,16 +1,26 @@
# bat-api Admin Panel # bat-api Admin Panel
This directory is reserved for the future player-facing `bat-api` management 本目录预留给后续 `bat-api` 管理面板。当前后端已提供只读入口
panel. The current backend exposes a non-mutating index at `GET /admin/` that `GET /admin/`,返回 health、ready、bootstrap、release、resources 和 OpenAPI
returns JSON links for health, readiness, bootstrap, release, resources, and 链接。
OpenAPI. `GET /admin/schedules` and mutating
`/admin/control/schedule-{add,update,remove,run}` routes are protected by the
configured admin token and forward the Rust-owned schedule contract.
`GET /admin/schedules` accepts optional `id`, `group`, and `enabled` filters.
`POST /admin/control/translation-task-update` uses the same token and forwards
the narrow `translation.task.update` provider-worker status contract.
The production panel must reuse the same HTTP authentication, rate limiting, 需要配置 `BAT_API_AUTH_TOKEN` 的接口:
access logging, reverse-proxy handling, and no-store dynamic response policy as
the resource API. Static frontend assets are not implemented yet. See - `GET /admin/schedules`:转发 Rust 持有的 schedule 查询,支持 `id``group`
`docs/reports/GO_STATUS.md` for the current component boundary and status. `enabled` 过滤。
- `POST /admin/control/schedule-{add,update,remove,run}`:转发
`schedule.*` 计划控制。
- `POST /admin/control/translation-task-update`:转发
`translation.task.update`,供外部 provider 流程回写任务状态。
- `POST /admin/control/translation-worker-run`:转发
`translation.worker.run`,触发 Rust provider worker。
- `POST /admin/control/translation-proofread`:转发
`translation.proofread`,把汉化 workflow 标记为人工校对中。
- `GET /admin/translation/tasks`:转发 `translation.tasks`,按分页、release、
TextUnit、状态和失败原因过滤查询 worker 状态。
- `GET /admin/translation/handoff`:转发 `translation.handoff`,读取当前
release 的完整翻译交接视图。
正式前端必须复用资源 API 的 HTTP 鉴权、限流、访问日志、反代处理和动态响应
`Cache-Control: no-store` 策略。静态前端资产仍为 not implemented,尚未作为
本目录的稳定交付物。当前组件边界见 `docs/reports/GO_STATUS.md`